Image processing device, image processing system, image processing method, and program

Abstract

This enables more efficient registration of subject features that are effective in improving matching accuracy. [Solution] The image processing device detects a predetermined subject from an image captured by an imaging device and extracts feature quantities from the image of the detected subject. The image processing device also registers a first feature quantity, which is the feature quantity of the subject to be recognized, and a second feature quantity, which is the feature quantity of a subject other than the subject to be recognized. Based on the comparison result between the extracted feature quantity and the first feature quantity, and the comparison result between the extracted feature quantity and the second feature quantity, the device verifies whether the detected subject is the subject to be recognized. Furthermore, the image processing device obtains feature quantities of other subjects that are the subject to be recognized by another imaging device different from the imaging device that captured the image, via data acquired through data communication, and registers those feature quantities as second feature quantities.

Description

Technical Field

[0001] The present invention relates to image processing technology for processing captured images.

Background Art

[0002] In recent years, image processing technology has been developed to recognize (collate) whether a subject to be compared is the same subject by learning (optimizing) parameters of a multi-layer NN (Neural Network) using a large amount of image data. As the subject, the face of a specific person, etc. is assumed, and if learning is sufficiently performed, a specific person can be recognized with high accuracy. This image processing technology is also used in cases where the features of the face of a specific person to be photographed are registered in advance in a camera, and when the features of the face of a person within the shooting range match the features of the face of the person registered in advance, the focus is preferentially adjusted to that face. On the other hand, when applying face recognition to focus assistance during shooting, for example, in scenes such as sports scenes or school sports meets, the race, gender, age, etc. of the subject person are often similar. Also, the opportunities for similarity in expressions and face orientations / angles that are easily affected by the content of competitions such as sports scenes and sports meets tend to increase. For this reason, the similarity between the person for whom focus is to be adjusted and other persons is likely to be high, and it is likely that the focus will be adjusted to other persons. In contrast, Patent Document 1 discloses a technique for suppressing mis-collation by explicitly providing a face of another person similar to (that is, easily mistaken for) the face to be collated.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technology described in Patent Document 1, for example, in order to register other subjects similar to the subject to be matched, the user must visually check the matching results or use more precise biometric recognition results to sequentially confirm that the other subjects are not the subject to be matched. However, when photographing people participating in an event, for example, spending time and effort on sequential confirmation may cause the user to miss the crucial photo opportunity or even miss the event itself. Therefore, it is desirable to be able to register the characteristics of subjects other than the subject to be matched more efficiently, which is effective in improving matching accuracy.

[0005] Therefore, the present invention aims to enable more efficient registration of subject characteristics that are effective in improving matching accuracy. [Means for solving the problem]

[0006] The present invention provides an image processing apparatus comprising: detection means for detecting a predetermined subject from an image captured by an imaging device; extraction means for extracting feature quantities from an image of the detected predetermined subject; registration means for registering a first feature quantity, which is a feature quantity of a subject to be recognized, and a second feature quantity, which is a feature quantity of a subject other than the subject to be recognized; matching means for determining whether the detected subject is the subject to be recognized based on a comparison result between the extracted feature quantity and the first feature quantity and a comparison result between the extracted feature quantity and the second feature quantity; and communication means for performing data communication. The present invention is characterized in that, via the data acquired through the data communication, feature quantities of other subjects that are the subject to be recognized by another imaging device different from the imaging device that captured the image are acquired, and the acquired feature quantities of other subjects are registered as the second feature quantities by the registration means. [Effects of the Invention]

[0007] According to the present invention, it becomes possible to register subject features that are effective in improving matching accuracy more efficiently. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the external appearance of an imaging device to which the image processing apparatus of this embodiment is applied. [Figure 2] This diagram shows an example of the internal configuration of an imaging device. [Figure 3] This diagram shows an example of the internal configuration of the imaging unit. [Figure 4] This figure shows an example of the basic configuration of the image processing unit of this embodiment. [Figure 5] This diagram shows an example of connecting the image processing unit to the image processing unit of another imaging device. [Figure 6] This figure shows examples of scenes captured using multiple imaging devices. [Figure 7] This figure shows an example of a face detection result obtained from an captured image. [Figure 8] This diagram shows an overview of the registration information for each imaging device. [Figure 9] This diagram shows an overview of the registration information, including the characteristics of the target person and the characteristics of other people. [Figure 10] This is a flowchart for the process of acquiring and registering the features of other people's objects. [Figure 11] This is a flowchart for person matching and focus processing using feature vectors. [Figure 12] This figure shows an example of connecting each imaging device via a server. [Figure 13] This is a flowchart for the process of registering external object features via a server. [Figure 14] This diagram shows an example configuration where model management and event management are performed on a server. [Modes for carrying out the invention]

[0009] Hereinafter, embodiments of the present invention will be described while referring to the drawings. Each of the following embodiments does not limit the present invention, and not all combinations of the features described in this embodiment are essential for the solution means of the present invention. The configuration of the embodiment can be appropriately modified or changed according to the specifications of the device to which the present invention is applied and various conditions (usage conditions, usage environment, etc.). Also, in each of the following embodiments, the same or similar configurations and processing steps are denoted by the same reference numerals, and duplicate descriptions are omitted.

[0010] <First Embodiment> FIG. 1 is a diagram showing an example of the schematic external configuration of the imaging device 100 according to this embodiment. FIG. 1(a) shows the front side where the lens of the imaging unit 101 is arranged, and FIG. 1(b) shows the schematic external configuration of the back side where the display unit 103 is arranged. The imaging device 100 includes an imaging unit 101, a shooting button 102, a display unit 103, a cross key 104, and the like.

[0011] The imaging unit 101 includes an optical system including a lens and an aperture, an imaging sensor, an A / D conversion circuit, and the like. The details of the configuration of the imaging unit 101 will be described later using FIG. 3. The imaging unit 101 captures an object image formed by the optical system with the imaging sensor in response to the pressing of the shooting button 102 by the user who is the user of the imaging device 100. The imaging signal acquired by the imaging sensor is analog / digitally converted by the A / D conversion circuit and output from the imaging unit 101 as digital image data. In the following description, for the sake of simplicity of description, the digital image data handled in the imaging device 100 is appropriately described as an image.

[0012] The display unit 103 displays the captured image acquired by the imaging unit 101 or the captured images stored in a memory card or the like, or superimposes predetermined related information on those images. Also, a menu screen for various settings of the imaging device 100 and the like is displayed on the display unit 103. The display unit 103 is, for example, a liquid crystal display and may have a touch screen function.

[0013] The cross key 104 is provided to obtain various user operations, such as switching the content displayed on the display unit 103 or selecting a menu, according to the up, down, left, and right operation inputs by the user. Note that the positions, types, and numbers of the buttons, display unit, and imaging unit mounted on the imaging device 100 are not limited to the example in FIG. 1. The imaging device 100 also has, for example, a power button, an infrared sensor, a mode dial, a strobe light-emitting unit, a wireless communication unit, and a cable interface unit, which are provided in a general digital camera.

[0014] FIG. 2 is a diagram showing an example of the internal configuration of the imaging device 100 of the present embodiment. The CPU 201 is a central processing unit that comprehensively controls the following components provided in the imaging device 100 of the present embodiment. The RAM (Random Access Memory) 202 is used as the main memory, work area, etc. of the CPU 201. The ROM (Read Only Memory) 203 stores control programs executed by the CPU 201, image processing programs executed by the image processing unit 212 described later, and other setting information.

[0015] The bus 204 is a transfer path for various data in the imaging device 100. For example, an image (digital data) acquired by the imaging unit 101 is appropriately sent to each information processing unit such as the signal processing unit 208, the encoder unit 209, and the image processing unit 212 via this bus 204.

[0016] The operation unit 205 is provided to obtain instructions from the user in the imaging device 100. In the example of FIG. 1, the shooting button 102, the cross key 104, etc. are included in the operation unit 205. When the display unit 103 has a touch screen function as described above, the operation unit 205 also obtains user instructions for the touch screen. The display control unit 206 performs display control of the captured image, items in the menu, characters, etc. displayed on the display unit 103 illustrated in FIG. 1.

[0017] Based on instructions from the CPU 201, the imaging control unit 207 performs control over the imaging system, such as controlling the drive of the focus lens included in the optical system of the imaging unit 101, controlling the opening and closing of a shutter (not shown) provided in the imaging device 100, and adjusting the aperture. The signal processing unit 208 performs various digital signal processing on the captured image received via the bus 204, including white balance processing, gamma processing, and noise reduction processing. The encoder unit 209 performs a process to convert the image data received via the bus 204 into image data in a predetermined file format such as JPEG or MPEG.

[0018] The memory control unit 210 is an interface for connecting to various media (e.g., hard disk, memory card, CF card, SD card, USB memory). The communication control unit 211 is an interface for sending and receiving data by connecting to external devices (such as personal computers or smartphones) via wired, wireless LAN, or short-range wireless communication. If these external devices can provide input / output functions to the user, they may be used as input sources to the operation unit 205 or as output destinations from the display control unit 206.

[0019] The image processing unit 212 consists of, for example, a CPU dedicated to image processing, separate from the CPU 201, and executes the image processing program according to this embodiment to realize the image processing described later. In this embodiment, the image processing unit 212 detects a predetermined subject area from the image acquired by the imaging unit 101 or the image output from the signal processing unit 208, and extracts feature quantities from the detected subject area. The image processing unit 212 also registers a first feature quantity, which is the feature quantity of the subject to be recognized, and a second feature quantity, which is the feature quantity of a subject other than the subject to be recognized. In the following description, the first feature quantity of the subject to be recognized will be called the target person feature quantity, and the second object feature quantity of a subject other than the subject to be recognized will be called the other person object feature quantity. The image processing unit 212 then performs a matching process to determine whether the person in the detected subject area is the person to be recognized, using the feature quantities extracted from the detected subject area and the registered target person feature quantity and other person object feature quantity. In this embodiment, a person (especially a face) is used as an example of the subject, and the subject area detected from the image is assumed to be the image area of ​​the person's face (hereinafter referred to as the face image). Details of the face image detection process, feature extraction process from face images, and matching process using the extracted features from the detected face images and the registered first and second features, performed by the image processing unit 212, will be described later. Note that these processes performed by the image processing unit 212 are not limited to those implemented by the execution of the image processing program according to this embodiment, but may also be implemented by hardware configurations such as circuit configurations. The imaging device 100 of this embodiment also includes various configurations that are common to general imaging devices, in addition to the configurations shown in Figure 2, but their illustration and description are omitted.

[0020] Figure 3 shows an example of the internal configuration of the imaging unit 101. The imaging unit 101 comprises a lens group 301, an aperture 302, a shutter 303, a filter group 304, an image sensor 305, and an A / D conversion unit 306. The lens group 301 includes, for example, a zoom lens, a focus lens, and an image stabilization lens, and forms an image of the subject on the imaging surface of the image sensor 305. The shutter 303 is opened and closed by the image control unit 207 when the aforementioned shooting button 102 is pressed. The filter group 304 consists of, for example, a low-pass filter, an infrared cut filter (iR cut filter), and a color filter. The image sensor 305 is, for example, an image sensor such as a CMOS or CCD. The A / D conversion unit 306 converts the imaging signal output from the image sensor 305 into digital image data and outputs the digital image data to the aforementioned bus 204.

[0021] Figure 4 shows an example of the basic configuration of the image processing unit 212 according to this embodiment, which is mounted on the imaging device. The imaging device of this embodiment stores feature quantities extracted in advance from the face image of the person to be recognized, and has a focus support function that preferentially focuses on the face of the person if there is a person whose face matches the feature quantities in the captured image. In this embodiment, the person to be recognized is a specific person who is photographed at a sports competition or athletic meet. That is, the imaging device of this embodiment has a focus support function that detects a person's face image from the captured image, identifies a specific person based on the detected face image, and focuses on the face of the identified specific person. In this embodiment, the feature quantities extracted in advance from the face image of the person to be recognized are the target person feature quantities (first feature quantities).

[0022] Furthermore, the imaging device according to the first embodiment has a configuration that enables the transmission and reception of various types of data, including feature quantities, with other imaging devices. Figure 5 shows two imaging devices 501 and 502 as an example. It is assumed that imaging devices 501 and 502 each have an image processing unit 212 with the same configuration as shown in Figure 4. Figure 5 shows a configuration in which the image processing units 212 of these two imaging devices 501 and 502 are connected in a communicative manner. It is also assumed that the image processing units 212 of these imaging devices 501 and 502 can extract the same feature quantities from, for example, the same subject (face image of the same person). Furthermore, it is assumed that the target person feature quantities stored in advance by imaging devices 501 and 502 are feature quantities extracted from face images of different target persons for recognition, each for each imaging device. In other words, the person in the face image from which the target person feature quantities stored by imaging device 501 are extracted is a different person from the person in the face image from which the target person feature quantities stored by imaging device 502 are extracted. Although Figure 5 shows an example of communication between two imaging devices, the imaging device in this embodiment can also communicate with multiple other imaging devices.

[0023] Figure 6 shows an example where two imaging devices, imaging device 501 and imaging device 502, are capturing the same scene. An example of imaging devices 501 and 502 capturing the same scene is a scene in which multiple subjects are being captured simultaneously, such as in a sports day competition. Note that in Figure 6, for the sake of simplicity, only the faces of the multiple subjects are depicted. Here, it is assumed that imaging device 501 stores target person features that have been previously extracted from the face image of the face 603 of the person to be recognized. On the other hand, it is assumed that imaging device 502 stores target person features that have been previously extracted from the face image of the face 604 of the person to be recognized as target person features. In this embodiment, imaging device 501 acquires the target person features that imaging device 502 has acquired and stored from the face 604 of the person to be recognized as other object features, and registers the acquired other object features in association with the target person features that it has previously stored. In other words, in the imaging device 501, the other person feature quantities registered in association with the target person feature quantities are feature quantities extracted from the face images of other people other than the target of recognition. Similarly, in the imaging device 502, the target person feature quantities that the imaging device 501 has acquired and stored from the face 603 of the target person of recognition may be acquired as other person feature quantities, and these acquired other person feature quantities may be registered in association with the target person feature quantities that have been stored in advance.

[0024] The following describes the processing performed by the image processing unit 212 shown in Figures 4 and 5. As shown in Figures 4 and 5, the image processing unit 212 of the imaging device 501 and the imaging device 502 have the same configuration. Therefore, in the following description, unless it is necessary to explicitly state whether the image processing unit 212 of the imaging device 501 or the imaging device 502 is being referred to, the imaging devices 501 and 502 will not be distinguished.

[0025] The acquisition unit 401 acquires the image captured by the imaging unit 101 shown in Figure 2. Figure 7 shows an example of an image captured of the four subjects illustrated in Figure 6. The detection unit 402 detects a face image as a specific subject area from the captured image acquired by the acquisition unit 401 from the imaging unit 101. The detection unit 402 detects a person's face image present in the captured image and identifies the position of that face image within the captured image.

[0026] In this embodiment, a model consisting of a multilayer neural network that estimates the location of a face in an image is pre-trained, and the detection unit 402 uses this model to detect a face image from the captured image. This model can be any known network that is a multilayered Convolutional Neural Network (CNN), such as VGG or ResNet. However, it is not limited to CNNs; any architecture that can estimate face position using an image as input is acceptable.

[0027] In this embodiment, the model used in the detection unit 402 is assumed to be a model trained to estimate a map in which the pixel at the center of the face position is close to 1 and the other pixels are close to 0. During training, each training image and a ground truth map in which the pixel at the center of the face position is set to 1 and the others to 0 are prepared in advance, and training is repeated so that the error between the estimated map and the ground truth map becomes small. As the loss function to evaluate this error, a generally known cross-entropy loss can be used. Note that it is not necessary to estimate so that values ​​appear only at the center of the face position; the ground truth map may be a region in which the values ​​are higher as the coordinates of the center of the face position increase (for example, a region blurred to follow a Gaussian distribution). Such a training method is a common method for training detection tasks using CNNs, and this embodiment also adopts a configuration that follows this method. In addition, since each face image is required during training, the size of the face is also estimated. For example, the model may be trained to estimate the height and width of the face simultaneously. Multitask learning, which involves learning multiple tasks simultaneously, can be expected to improve accuracy through a complementary effect by combining highly correlated information. For example, the multi-layer network described above only outputs a map that estimates the face position, but it would be better to output maps where the estimated height and width values ​​are located near the same pixel position as the face center. In this embodiment, such a model is pre-trained, and the detection unit 402 uses this model to identify the face position and size in the captured image, and then extracts a face image from the captured image based on that face position and size.

[0028] In the example of the captured image shown in Figure 7, the detection unit 402 obtains rectangles 701 to 704 as estimation results, which estimate the center of each of the four faces and the size of each face, and extracts each region of these rectangles 701 to 704 from the captured image as a face image.

[0029] The extraction unit 403 extracts features from the face images detected by the detection unit 402 that can be used to identify each individual person. Details of the feature extraction process will be described later. In the example shown in Figure 7, the extraction unit 403 extracts features from the face images within each rectangle 701 to 704 that the detection unit 402 has cut out from the captured image. In this embodiment, a model consisting of a multilayer neural network that extracts features from face images to perform the task of personal identification is pre-trained, and the extraction unit 403 uses this model to extract features from face images and perform the personal identification task. In the training for the personal identification task, a large number of face images are assigned an ID (identification information) to identify an individual as the correct answer, and the training is performed to estimate the ID from the input face image. Here, the personal identification task can be considered as a type of classification task that estimates which class a face image belongs to. For the classification task, a method is known in which the output layer is trained to output the likelihood distribution of each class. This method, which uses cross-entropy loss as the loss function, is a common method for multi-class classification using CNNs, and this embodiment also adopts a configuration that follows this method. However, it is not limited to CNNs, and as long as it is possible to estimate an ID to identify an individual by inputting a face image, the architecture is not particularly limited. In the case of a model that has been sufficiently trained and can estimate the ID for personal identification with high accuracy, it is known that the output vectors of the intermediate layer before the output layer will be similar for the same person. Therefore, by utilizing the distance between these vectors, it is possible to perform individual identification even for unknown faces (people) that have not been trained. In this embodiment as well, the output vectors of the intermediate layers of such a model are used as features.

[0030] In this embodiment, the models used in the detection unit 402 and the extraction unit 403 are assumed to be models that have been trained separately, but this is not limited to this. For example, a model may be used in which the detection network and the personal identification network are connected end-to-end to train both the detection and personal identification tasks simultaneously. Furthermore, in the image processing unit 212 of this embodiment, the extraction unit 403 is common to the imaging devices (imaging device 501 and imaging device 502) that exchange feature quantities, in order to compare the feature quantities registered in each imaging device in the matching unit 405, which will be described later.

[0031] The registration unit 404 registers the feature quantities to be matched in the matching unit 405, which will be described later. In this embodiment, the registration unit 404 registers two types of feature quantities: the target person feature quantity for the person to be recognized, and the other person feature quantity, which is the feature quantity for other people other than the person to be recognized. The registration unit 404 associates the target person feature quantity and the other person feature quantity and records and stores them on a recording medium via the memory control unit 210. In this embodiment, the information registered by the registration unit 404 is, as an example, the registration information shown in Figure 8.

[0032] Figure 8(a) is a diagram showing an example of registration information stored in the imaging device 501 shown in Figures 5 and 6. The registration information of the imaging device 501 includes a face image 801 of the person to be recognized by the imaging device 501 (the person to be recognized), a target person feature quantity 803 extracted from the face image 801, and the device ID 805, which is the identification information of the imaging device 501. Note that the face image 801 does not necessarily have to be registered as long as information that can identify the person to be recognized (for example, the target person feature quantity 803) is registered. However, the face image 801 of the person to be recognized may be registered, for example, so that it can be used when a user of the imaging device 501 can confirm the face image 801 from which the target person feature quantity 803 was extracted. In this embodiment, the target person feature quantity 803 is assumed to have been extracted and registered in advance from the face image of the person to be recognized, and is used when the person to be recognized is matched by the matching unit 405, which will be described later. Furthermore, the target person feature quantity 803 registered in the imaging device 501 may be used as a feature quantity for another person in another imaging device (imaging device 502 in the examples of Figures 5 and 6). For this reason, the device ID 805 is registered in association with the target person feature quantity 803 so that the imaging device 501 that has registered the target person feature quantity 803 can be identified by the other imaging device (imaging device 502). Note that the device ID 805 is device-specific identification information for the imaging device 501, and a general-purpose device identifier such as a UUID can be used.

[0033] Figure 8(b) shows an example of registration information stored in the imaging device 502, as shown in Figures 5 and 6. The registration information of the imaging device 502 includes a face image 802 of the person that the imaging device 502 is intended to recognize (the person to be recognized), a target person feature quantity 804 extracted from the face image 802, and the device ID 806, which is the identification information of the imaging device 502. In the case of the registration information of the imaging device 502, as described above, the face image 802 does not necessarily have to be registered as long as information that can identify the person to be recognized (such as the target person feature quantity 804) is registered. However, since the user of the imaging device 502 may check the face image 802, for example, the face image 802 may be registered. The target person feature quantity 804 is assumed to have been extracted and registered in advance from the face image 802 of the person to be recognized, and is used when the matching unit 405 performs matching of the person to be recognized. Furthermore, the target person feature quantity 804 registered in the imaging device 502 may be used as a feature quantity for another person in another imaging device (imaging device 501 in this example). For this reason, the device ID 806 is registered in association with the target person feature quantity 804 so that other imaging devices (imaging device 501) can identify it.

[0034] As mentioned above, for example, imaging device 501 can acquire the target person feature quantity 804 registered in another imaging device 502 and use that target person feature quantity 804 as another person feature quantity. When imaging device 501 acquires the target person feature quantity 804 registered in another imaging device 502 as another person feature quantity, it adds that target person feature quantity 804 and the device ID 806 of imaging device 502 to the registration information. Similarly, imaging device 502 can acquire the target person feature quantity 803 registered in another imaging device 501 and use that target person feature quantity 803 as another person feature quantity. When imaging device 502 acquires the target person feature quantity 803 registered in another imaging device 501 as another person feature quantity, it adds that target person feature quantity 803 and the device ID 805 of imaging device 501 to the registration information.

[0035] Figure 9 shows the registration information after, for example, the imaging device 501 has acquired the target person feature quantity 804 and device ID 806, as exemplified in Figure 8(b), from another imaging device 502 and added them as the other person's feature quantity 901 and device ID 902. In other words, the registration unit 404 of the imaging device 501 registers the other person's feature quantity 901 (target person feature quantity 804) and device ID 902 (device ID 806) acquired from the other imaging device 502, associating them with the registration information of the face image 801, the target person feature quantity 803, and the device ID 805. Although the example in Figure 9 shows the registration information of the imaging device 501, the imaging device 502 can similarly acquire the target person feature quantity 803 and device ID 805 from the other imaging device 501 and add them as the other person's feature quantity and device ID.

[0036] Furthermore, if the imaging device 501 has registered the target person feature quantity 804 of the other imaging device 502 as another person's feature quantity 901, then, from the standpoint of protecting privacy, it is not necessarily required to acquire and register the face image 802 registered in the imaging device 502. Similarly, if the imaging device 502 has registered the target person feature quantity 803 of the other imaging device 501 as another person's feature quantity, then it is not necessarily required to register the face image 801 registered in the imaging device 501. However, this is not the only option, and if the imaging device 501 and the imaging device 502 have strictly confirmed mutual agreement on the policy regarding the use of personal information, they may mutually acquire and retain face images. In this way, if mutual permission to use face images is granted, the respective extraction unit 403 can acquire feature quantities from the face image, and the registration unit 404 can register those feature quantities as another person's feature quantity. In this case, the extraction unit 403 of the imaging device 501 and the imaging device 502 may be different.

[0037] Furthermore, Figure 9 illustrates an example where two imaging devices, 501 and 502, capture the same scene as in Figure 6, and imaging device 501 acquires the target person feature quantity 804 and device ID 806 registered in imaging device 502 as the other person's feature quantity 901 and device ID 902. In addition to this example, there may be multiple other imaging devices capturing the same scene as imaging device 501, and each of these other imaging devices may have different target person feature quantities registered. In such a case, imaging device 501 may acquire each target person feature quantity and device ID registered in each of these multiple other imaging devices and register them as the other person's feature quantity and device ID, respectively. This allows imaging device 501 to avoid duplicate acquisition and registration of feature quantities related to the same person as other person's feature quantities by referring to the device IDs of these other imaging devices. In this case, it also becomes possible to use other person's feature quantities limited to other imaging devices that are close to imaging device 501, for example.

[0038] The matching unit 405 calculates a matching score based on the comparison result between the target person features registered in the registration unit 404 and the features of each person detected by the detection unit 402 from the image captured by the imaging unit 101 and extracted by the extraction unit 403 (referred to as matching person features). In this embodiment, the matching unit 405 calculates the matching score as the vector distance (dot product) between the target person features and the matching person features. If the calculated matching score exceeds a predetermined matching threshold, the matching unit 405 determines (matches) that the person from whom the matching person features were extracted is a person to be recognized. In this embodiment, if the aforementioned other person features are already registered, the matching unit 405 also calculates a matching score based on the comparison result between the matching person features and the other person features. If the matching score calculated using the other person features is higher than the matching score calculated using the target person features, the matching unit 405 determines that the person from whom the matching person features were extracted is not a person to be recognized.

[0039] The matching threshold used for person matching, in the case of person recognition based on facial images as in this embodiment, is generally determined based on the False Acceptance Rate (FAR), which is the rate at which other people are misidentified as the target person. For example, if there is a sufficient number of evaluation image sets, the matching score of each face in the evaluation image set can be aggregated and a threshold adjusted to, for example, FAR = 1e-6 can be used. An evaluation image set is a set of registered faces of the target person to be recognized and images in the evaluation images that clearly show the faces of the target person.

[0040] It should be noted that the resources of the imaging device may not be sufficient. In this case, the matching unit 405 may perform matching using the features of other objects only when the matching score calculated using the features of the target person is below a predetermined sufficiently high score threshold, rather than performing matching using the features of other objects for each image captured. The score threshold may be sufficiently higher than the matching threshold mentioned above. However, this is not the only option, as the matching score of the features of other objects may increase due to factors such as face orientation, facial expression, and brightness. Therefore, the threshold for whether or not to compare with the matching score using the features of other objects may be dynamically changed depending on the imaging environment and scene. The imaging environment can be determined from various sensor information at the time of imaging. The scene can be obtained by referring to the scene when the user of the imaging device provides instructions for setting the scene. That is, for example, if the environment at the time of imaging is dark, or if the subject moves quickly or the face orientation changes dynamically (e.g., a sports scene), the threshold for whether or not to perform matching with the features of other objects may be adjusted to be lower.

[0041] The search unit 406 searches for other imaging devices that meet predetermined conditions via the communication unit 407. For example, the search unit 406 may set a predetermined condition that the imaging device is operating within a predetermined range of a certain distance from the imaging device, and then search for other imaging devices that meet that predetermined condition. Alternatively, the search unit 406 may set a predetermined condition that the imaging device operates in a similar manner with a configuration equivalent to that of Figures 4 and 5, and then search for other imaging devices that meet that predetermined condition. For example, the configuration equivalent to that of Figures 4 and 5, and the operation equivalent to that of Figures 4 and 5, could be described as the extraction of feature quantities using an extraction process common to the extraction unit 403. Alternatively, the search unit 406 may set a predetermined condition that imaging has been performed by another imaging device within a certain period of time since imaging by the imaging unit 101, and then search for other imaging devices that meet that predetermined condition. Furthermore, the search unit 406 may set a predetermined condition that the other imaging device has agreed to a policy regarding the use of feature quantities of the person to be recognized, and then search for other imaging devices that meet that predetermined condition. The search unit 406 may use at least one or more combinations of these predetermined conditions to search for other imaging devices.

[0042] In this embodiment, for example, in Figure 5, if the imaging device 501 is the source imaging device, an example is given in which other imaging devices 502, which have an image processing unit 212 with a similar configuration and are located within range of proximity communication with imaging device 501, are searched for. Furthermore, in this embodiment, an example is given in which the facial feature quantities of the person to be recognized are mutually used between imaging devices, so during the search, it is checked whether consent has been obtained to the policy regarding the use of personal information such as faces, and imaging devices that have consented to the policy are searched for. Thus, the search unit 406 in this embodiment has predetermined conditions for mutually searching for other imaging devices that are suitable as communication destinations, and searches for other imaging devices that meet those conditions.

[0043] The communication unit 407 performs data communication with an external device via the communication control unit 211. In the first embodiment, the communication unit 407 connects to another imaging device and sends and receives feature quantities, device IDs, etc., as data communication. In the first embodiment, the communication unit 407 performs proximity communication with the communication unit 407 of the other imaging device, and obtains the target person feature quantities and device ID registered in the other imaging device to which it is connected, or conversely, transmits the registered target person feature quantities and device ID to the connected device. The communication standard used by the communication unit 407 according to the first embodiment is not limited and may be any general-purpose short-range communication standard such as Bluetooth®.

[0044] Figure 10 is a flowchart showing the process flow in the image processing unit 212 described in Figures 4 and 5, which acquires the feature quantities of a target person registered in another imaging device and registers them as feature quantities of another person. In the explanation of Figure 10, as shown in Figure 6, it is assumed that imaging devices 501 and 502 are capturing images of the same scene, such as a sports day, and that imaging devices 501 and 502 are each capturing images of different people participating in the competition simultaneously as the target people for recognition. The person targeted for recognition by imaging device 501 is the person with face 603, and as shown in Figure 8(a), the feature quantities of face 603 are pre-registered in imaging device 501 as the target person feature quantities 803. On the other hand, the person targeted for recognition by imaging device 502 is the person with face 604, and as shown in Figure 8(b), the feature quantities of face 604 are pre-registered in imaging device 502 as the target person feature quantities 804. Furthermore, it is assumed that imaging devices 501 and 502 have pre-established agreements regarding policies for the mutual use of the registered target person feature quantities with other imaging devices (permission for mutual use of feature quantities). Note that while the flowchart in Figure 10 uses imaging device 501 as an example to explain the process performed by imaging device 501, the same process may be performed in parallel by imaging device 502. The explanation of the device ID, which is acquired and registered along with the target person features registered in other imaging devices, is omitted in the explanation of the flowchart in Figure 10.

[0045] First, in step S1001, the search unit 406 of the imaging device 501 searches for and connects to another imaging device that meets predetermined conditions via the communication unit 407. If there are multiple other imaging devices capable of nearby communication, the search unit 406 connects to one of them via the communication unit 407. In the example shown in Figure 4 of this embodiment, the search unit 406 of the imaging device 501 connects to the imaging device 502 via the communication unit 407.

[0046] Next, in step S1002, the search unit 406 obtains information from the imaging device 502, which was searched for and connected in step S1001, regarding whether or not it has agreed to the policy on the mutual use of the target person features 804 registered in the imaging device 502. It is assumed that imaging devices 501 and 502 have pre-configured agreements to the policy on the mutual use of pre-registered target person features with other imaging devices (permission for mutual use of features).

[0047] Next, in step S1003, the search unit 406 determines, based on the policy information obtained from the imaging device 502 in step S1002, whether the target person feature quantities 804 pre-registered in the imaging device 502 are set to be mutually shareable. If it is determined in step S1003 that the settings are mutually shareable, the search unit 406 proceeds to step S1004. On the other hand, if it is not determined in step S1002 that the settings are mutually shareable, that is, if it is not possible to obtain information from the imaging device 502 that the policy has been agreed to, or if information that the policy has not been agreed to has been obtained, the search unit 406 returns to step S1001. When the process returns from step S1003 to step S1001, the search unit 406 further searches for other imaging devices that can communicate in proximity.

[0048] If the process proceeds from step S1003 to step S1004, the search unit 406 obtains the target person feature quantities 804 that have been pre-registered in the imaging device 502 from the imaging device 502 connected via the communication unit 407. Next, in step S1005, the registration unit 404 adds the target person feature quantity 804 acquired from the imaging device 502 to the registration information as another person's feature quantity 901, as shown in Figure 9. The search unit 406 also acquires the device ID 806 along with the target person feature quantity 804 from other imaging devices 502, and the registration unit 404 registers this as the device ID 902 together with the other person's feature quantity 901. This makes it possible to distinguish each other person's feature quantity even if there are other imaging devices that can be connected besides the imaging device 502 and other person's feature quantities are acquired from those imaging devices as well.

[0049] As mentioned above, in this embodiment, as shown in Figure 6, imaging devices 501 and 502 capture the same scene, such as a sports day, and imaging devices 501 and 502 each capture different people participating in the competition simultaneously as the target people for recognition. Here, as in the example in Figure 6, when two adjacent imaging devices 501 and 502 capture the same scene, it is highly likely that imaging devices 501 and 502 are each capturing different people within the same scene as the target people for recognition. For example, the person targeted for recognition by imaging device 502 (the person with face 604) is highly likely not the person targeted for recognition by imaging device 501 (the person with face 603). Therefore, in this embodiment, imaging device 501 acquires and registers the target person feature quantity 803 of the face 603 of the person targeted for recognition, as well as the target person feature quantity 804 of the face 604 of the person targeted for recognition by the other imaging device 502, as a third-party object feature quantity 901. Furthermore, since the feature quantities of other people not in the same scene captured by imaging devices 501 and 502 do not contribute to matching with the target person, there is no need to register those feature quantities. Thus, imaging device 501 registers the target person feature quantity 804 registered in the nearby imaging device 502, which is highly likely to be capturing the same scene, as the other person feature quantity 901. This allows imaging device 501 to register other person feature quantities that are limited to other people who are highly likely to be captured simultaneously with the target person in the same scene.

[0050] Figure 11 is a flowchart showing the process of focusing on the target person during imaging by the imaging unit 101, using the pre-registered target person features and the other person features registered in the process of Figure 10, in the image processing unit 212 of Figures 4 and 5. When imaging is performed by the imaging device 501, it is assumed that the user performs a series of operations, from pointing the imaging unit 101 towards the subject person to pressing the capture button 102. At this time, it is assumed that the image captured by the imaging unit 101 is sent to the acquisition unit 401 at a predetermined frame rate, and the acquisition unit 401 acquires the image at that predetermined frame rate. This is an example when a still image is captured, but it goes without saying that if the imaging device 501 is capturing a video, the image may be acquired during a series of operations by the user from the start to the stop of video recording. When acquiring a video in this way, the target person may be matched using a series of movements such as gait or running style.

[0051] First, as part of step S1101, the matching unit 405 obtains the registered target person features from the registration unit 404. That is, the matching unit 405 obtains the target person features 803 from the registration information shown in Figure 9. Next, in step S1102, the matching unit 405 retrieves the registered other-person feature quantity from the registration unit 404. That is, the matching unit 405 retrieves the other-person feature quantity 901 from the registration information in Figure 9.

[0052] Next, in step S1103, the acquisition unit 401 acquires the image captured by the imaging unit 101, as described above. Next, in step S1104, the detection unit 402 detects each face in the image acquired by the acquisition unit 401 in S1103, and extracts the region of the detected face from the image to obtain a face image (referred to as the detected face image in the explanation of Figure 11). Next, in step S1105, the extraction unit 403 extracts feature quantities from the detected face image obtained by the detection unit 402 in S1104. If there is no waiting time for the user to take the image during imaging, the extraction unit 403 may perform feature quantity extraction for each detected face image by batch processing using multiple detected face images. Next, in step S1106, the matching unit 405 calculates a matching score using the target person features obtained from the registration information in S1101 and the features extracted from the detected face image by the extraction unit 403 in S1105 as input.

[0053] Then, in the next step S1107, the matching unit 405 determines whether the matching score calculated in S1106 is equal to or greater than a predetermined matching threshold. If it is equal to or greater than the matching threshold, the person in the detected face image may be the person to be recognized, so the matching unit 405 proceeds to step S1108. On the other hand, if it is less than the matching threshold, the matching unit 405 determines that the person in the detected face image is not the person to be recognized, and returns to step S1105. As a result, in step S1105, the extraction unit 403 extracts the feature quantities of the next detected face image in the image, and in the next step S1106, the matching unit 405 compares the feature quantities of the detected face image with the feature quantities of the target person.

[0054] In this embodiment, since the focusing process during image acquisition is assumed, the processing of this flowchart for the currently processed image is interrupted and terminated when the next image acquisition unit 401 acquires the next image. In addition, the matching unit 405 may separately define a confirmation threshold for confirming the person to be recognized, which is a threshold sufficiently higher than the matching threshold, and when the score exceeds the confirmation threshold, it may be confirmed that the person is a person to be recognized and proceed to step S1110. For example, if the maximum value of the matching score is 1.0 and the matching threshold is 0.3, the confirmation threshold may be set to 0.8, which is sufficiently higher than the matching threshold and close to the maximum value, and if the matching score is equal to or greater than the confirmation threshold, it may be confirmed that the person is a person to be recognized.

[0055] When the process moves to step SS1108, the matching unit 405 calculates a matching score between the feature quantities obtained from the detected face image and the registered feature quantities of other people obtained in S1102. Next, in step S1109, the matching unit 405 compares the matching score with the target person feature calculated in S1106 with the matching score with the other person's feature calculated in S1108. If the matching score with the target person feature is higher, the matching unit 405 considers the detected face image to be the face image of the person to be recognized and proceeds to step S1110. On the other hand, if the matching score with the other person's feature is higher, the matching unit 405 determines that it is not the person to be recognized and returns to step S1105. As a result, in step S1105, the extraction unit 403 extracts the feature of the next detected face image in the image, and in the next step S1106, the matching unit 405 performs a comparison between the feature of the detected face image and the target person feature. In this embodiment, the matching is repeated for each detected face in the image until the acquisition unit 401 acquires the next image. This method helps to suppress the occurrence of misidentification, where a person who is not the target of recognition is mistakenly identified as the target person, while also making it easier to identify the target person.

[0056] The processing in step S1110 is performed by the imaging control unit 207 and the display control unit 206. The imaging control unit 207 prepares to focus on the detected face which has been identified as the person to be recognized, and the display control unit 206 informs the user of the imaging device 501 via the display unit 103 that the focus target has been identified and which face in the image was targeted. Here, since the focus target has been determined to be the person to be recognized, the display control unit 206 may also indicate that the person in the focus target is the person to be recognized by adding an icon or the like. If the user then operates the capture button 102, the imaging control unit 207 controls the camera to focus on the focus target and executes the image capture.

[0057] As described above, the image processing device according to the first embodiment acquires the feature quantities of a target person registered in other nearby imaging devices during imaging and registers them as feature quantities of other objects, and then performs person matching based on these target person feature quantities and other object feature quantities. At this time, the image processing device recognizes that, for example, if the matching score of the target person feature quantity is above the matching threshold but does not exceed the matching score of the other object feature quantity, the person of the other object feature quantity has been incorrectly matched as the target person to be recognized. As a result, the imaging device having the image processing device of this embodiment can suppress the loss of shooting opportunities due to mismatching. In other words, with the imaging device of this embodiment, by efficiently acquiring the feature quantities of other objects that may be captured simultaneously with the person to be focused and using them during matching, it becomes easier to suppress mismatching of the target person to be recognized. As a result, with the imaging device of this embodiment, it is possible to prevent the loss of shooting opportunities due to mismatching, and it is expected that the user of the imaging device will be able to capture the scene they intended.

[0058] In the embodiments described above, an example was given in which features extracted from a facial image were used when matching the person to be recognized. However, the source image for extracting features is not limited to a facial image, as long as features that can be used to match the person to be recognized can be obtained. For example, if the person to be recognized can be matched based on features of clothing, hats, or belongings, then those features of clothing, hats, or belongings may be used. Also, in the embodiments described above, an example was given in which a person was the subject of recognition. However, the subject of recognition is not limited to a person; for example, it could be a pet animal.

[0059] <Second Embodiment> In the first embodiment, as shown in Figure 5, an example was given in which the imaging device 501 directly communicates with another nearby imaging device 502. However, as long as the imaging device 501 can acquire the target person features registered in the other imaging device 502, the acquisition route is not limited to the example in Figure 5.

[0060] Figure 12 shows an example configuration of the second embodiment. In the second embodiment, an image processing system is given as an example in which imaging devices 501 and 502 are connected to a server 1204 via, for example, a smart device that can connect to a wide-area wireless network, and data communication is performed between the imaging devices via the server 1204. In the example configuration of Figure 12, the communication units 407 of imaging devices 501 and 502 include smart devices that can connect to a wide-area wireless network. Imaging devices 501 and 502 are each connected to a WAN (Wide Area Network) 1203 via the communication unit 407, and communicate information such as target person features and device IDs to each other via the server 1204 connected to the WAN 1203. In the example of Figure 12, two imaging devices are given as an example, but the imaging devices connected to the server 1204 via the WAN 1203 are not limited to two, and there may be three or more imaging devices. The internal configurations of the imaging devices 501 and 502 illustrated in Figure 12 are the same as those in the example shown in Figure 5, so a detailed explanation of them will be omitted.

[0061] The server 1204 according to the second embodiment includes at least an imaging device management unit 1205 and a feature quantity management unit 1206. The imaging device management unit 1205 and the feature quantity management unit 1206 are realized, for example, by the CPU of a computer constituting the server 1204 executing a program for the server according to the second embodiment. The server 1204 may also be a service such as a web service provided over a WAN.

[0062] The imaging device management unit 1205 manages information indicating at least the current device status of each imaging device (imaging devices 501, 502) connected via the WAN 1203, such as startup status, the most recent imaging time, and the imaging location. The feature management unit 1206 acquires at least the target person features and device ID registered in each imaging device (imaging devices 501, 502) connected via the WAN 1203, and manages the target person features and device ID for each imaging device. In addition, the feature management unit 1206 may also acquire and manage the facial images of the recognized target person as described above, along with the target person features and device ID. However, server 1204 manages such information only if it has obtained consent from each user of each imaging device connected via WAN 1203 to the policy regarding the sharing of information such as subject person characteristics and device status. Thus, in the second embodiment, the imaging device 501 and the imaging device 502 only need to communicate with the server 1204.

[0063] Figure 13 is a flowchart showing the process in the second embodiment in which an imaging device acquires target person features registered in another imaging device via a server and registers them as features of another person. Here, in the configuration example shown in Figure 12, for example, the case in which imaging device 501 acquires registered target person features from another imaging device 502 via server 1204 and registers them as features of another person will be explained as an example.

[0064] First, as part of step S1301, the imaging device management unit 1205 of the server 1204 registers imaging devices 501 and 502 connected via the WAN 1203, and then acquires and registers target person features from imaging devices 501 and 502, respectively. In this embodiment, each user of an imaging device is required to register the device ID of their imaging device with the server 1204 in advance. When the imaging devices are pre-registered, the server 1204 prompts each user of an imaging device to agree to a policy regarding the sharing of target person features registered in their respective imaging devices. Once the user agrees to the policy, the server 1204 acquires the information on target person features registered in each imaging device and manages it in association with the device ID of that imaging device.

[0065] Next, as part of step S1302, for example, the search unit 406 of the imaging device 501 transmits information indicating the current device status of the imaging device 501 to the server 1204. Furthermore, the search unit 406 of the imaging device 501 requests the server 1204 to transmit target person features that are managed in association with other nearby imaging devices, from among the target person features that the server 1204 manages in association with each imaging device. In other words, the search unit 406 of the imaging device 501 requests the transmission of features that the server 1204 manages in association with each imaging device, which will be registered as other person features in the registration unit 404 of the imaging device 501.

[0066] Next, in step S1303, the server 1204 identifies other imaging devices and target person features in the vicinity of imaging device 501 based on the information indicating the current device status of the imaging device received from imaging device 501. First, the imaging device management unit 1205 updates the current device status of imaging device 501, which requested transmission in step S1302, and identifies imaging devices in the vicinity of imaging device 501 based on that device status and the device status of other imaging devices. For example, the imaging device management unit 1205 identifies other imaging devices as nearby imaging devices that performed imaging within a certain time period from the imaging time of imaging device 501 and took images at an imaging position within a certain distance range from the imaging position of imaging device 501. Note that the certain time period and the certain distance range are predetermined. The feature management unit 1206 then identifies the target person features, which are managed in association with other imaging devices identified as being in the vicinity of the imaging device 501, as features to be transmitted to the imaging device 501, that is, features to be registered as other person features in the imaging device 501.

[0067] Next, in step S1304, the server 1204 transmits the target person features associated with the other imaging device identified in step S1303 to the imaging device 501. In the example in Figure 12, imaging device 502 is identified as another imaging device in the vicinity of imaging device 501, and the target person features managed in association with imaging device 502 are transmitted to imaging device 501. Note that when imaging device 501 requests in step S1302 to transmit target person features associated with other nearby imaging devices, the server 1204 may control whether or not to transmit the target person features in step S1304 depending on whether or not it has agreed to the aforementioned policy.

[0068] Next, in step S1305, when the search unit 406 of the imaging device 501 obtains the target person feature quantities linked to other nearby imaging devices 502 from the server 1204, the registration unit 404 registers those target person feature quantities as other person feature quantities. After registering the other person feature quantities, the matching process using these other person feature quantities is equivalent to the process in the flowchart shown in Figure 11 above, so its explanation is omitted here. According to the second embodiment, even in situations where imaging devices cannot communicate directly with each other when imaging is performed, for example, feature quantities can be shared via the server 1204, thereby improving the accuracy of matching the person to be recognized.

[0069] <Third Embodiment> In the second embodiment, an example was given in which the server 1204 configuration includes an imaging device management unit 1205 and a feature quantity management unit 1206, but it is not limited to this. For example, if the imaging device can acquire other person feature quantities that can be used when matching the person to be recognized, the server 1204 may also include a configuration for managing other information.

[0070] Figure 14 is a diagram showing an example configuration of an image processing system according to the third embodiment. In the third embodiment, the imaging device 501 and imaging device 502 are connected to the server 1400 via a smart device that can connect to a wide-area wireless network, and data communication is performed between the imaging devices via the server 1400. The internal configurations of the imaging devices 501 and 502 illustrated in Figure 14 are the same as those in the example in Figure 5 described above, so a detailed explanation of them is omitted. Also, in Figure 14, the imaging device management unit 1205 and the feature quantity management unit 1206 of the server 1400 are the same as in the example of the second embodiment described above, and the communication between imaging devices 501 and 502 via the WAN 1203 is also the same as described above, so a explanation of them is also omitted. In the example in Figure 14, two imaging devices are given as an example, but the imaging devices connected to the server 1400 via the WAN 1203 are not limited to two, and there may be three or more imaging devices.

[0071] The server 1400 of the third embodiment includes, in addition to the imaging device management unit 1205 and feature quantity management unit 1206 described above, a model management unit 1401 and an event management unit 1402. The server 1400 may have both the model management unit 1401 and the event management unit 1402, or it may have only one of them.

[0072] The model management unit 1401 manages the feature extraction models deployed in each imaging device managed by the imaging device management unit 1205. In other words, in this third embodiment, it is assumed that the server 1400 provides a function to customize the feature extraction models used in each imaging device. For example, it is assumed that users of imaging devices can tune the model as needed with individual data (such as family photos or photos of their favorite athletes), or that multiple people can contribute data to share a more accurate and robust model. In such cases, different feature extraction models are deployed in each imaging device, and the features used for matching the target person may not be compatible. Therefore, in the server 1400 of the third embodiment, the model management unit 1401 manages the feature extraction models deployed in each imaging device. The model management unit 1401 extracts features from the images of subjects detected by the imaging devices using the mutual models, converts them into features usable for matching, and provides them to each imaging device, even if the policy setting allows each imaging device to share only features with each other. Alternatively, the model management unit 1401 may manage the models deployed on each imaging device, thereby limiting the search to imaging devices that are using the same model. This makes it possible to limit the search range to only imaging devices that can mutually utilize the same features.

[0073] The event management unit 1402 manages event information such as participants and schedules related to events when the imaging devices are used, and based on this event information, provides each imaging device with the target person features registered in other imaging devices as other-person features. For example, if we consider an event such as a sports day, the event management unit 1402 may have information such as the participants and competition times of the competition pre-configured as event information. In this case, the event management unit 1402 can identify the imaging devices that recognize each person participating in the competition based on this event information. Then, when an acquisition request for other-person features is sent from each imaging device that recognizes a participant in the competition, the event management unit 1402 sends the features of other participants who are scheduled to participate in the competition at the same time as other-person features to each imaging device that made the acquisition request. In this way, each of these imaging devices can register other-person features with one another. In this case, although prior registration of event information is required, each imaging device can acquire feature quantities of other objects that may be captured within the same field of view as the target person, regardless of the device's status at the time of the event (for example, if it happens to be unable to capture images). This allows each imaging device to improve the accuracy of target person matching.

[0074] <Fourth Embodiment> In the embodiments described above, we described an example in which, when each imaging device captures the same scene, the target person feature quantities registered in each imaging device can be used mutually as other person feature quantities. In the fourth embodiment, an example is described in which an expiration date is set for the use of other-person feature quantities that are made mutually available in each imaging device. In the fourth embodiment, for example, the registration unit 404 includes an expiration date setting function for setting the expiration date of the use of other-person feature quantities, and for example, when the expiration date is reached, the registration information or the use of other-person feature quantities may be deleted. Alternatively, the matching unit 405 may also include an expiration date setting function for setting the expiration date of the use of other-person feature quantities, and when the expiration date of the use of other-person feature quantities has passed, the use of other-person feature quantities is disabled when matching the person to be recognized.

[0075] Examples of setting expiration dates for the use of other-person feature quantities include the following: For example, the expiration date for other-person feature quantities that are made mutually available to each imaging device may be the period until imaging of the same scene is completed. Also, for example, in the case where nearby imaging devices mutually acquire target person feature quantities as other-person feature quantities, as in the first embodiment, the expiration date for the other-person feature quantities may be limited to the period during which proximity communication continues between the acquiring imaging device and the acquiring imaging device. In other words, when proximity communication with the acquiring imaging device becomes impossible, the use of mutually acquired other-person feature quantities may be made impossible. Also, for example, in the case of the third embodiment, where each imaging device is connected to a server, the server manages event information, and a request to acquire other-person feature quantities is made during the holding of that event, the expiration date for the other-person feature quantities may be limited to the duration of that event. In this way, by enabling the use of other-person feature quantities only during periods when it is effective to use them, it is possible to reduce the consumption of unnecessary resources and processing costs, and to suppress the retention of personally identifiable information on other imaging devices.

[0076] <Fifth Embodiment> In the embodiments described above, several examples were given as predetermined conditions for the search performed by the search unit 406, but the predetermined conditions are not limited to the examples described above. In the fifth embodiment, for example, in order to acquire other person feature quantities that can be effectively used during person matching, the condition may be whether the most recent time the person to be recognized was matched in the imaging device is within a certain period. That is, in the fifth embodiment, the search unit 406 searches for other imaging devices that meet the condition that the most recent time the person to be recognized was matched is within a certain period. In this way, by referring to the time of the most recent match of the person to be recognized as a condition, it is possible to search for other imaging devices that are highly likely to be currently imaging a person corresponding to the other person feature quantity. According to the fifth embodiment, it becomes possible to acquire the feature quantities of other objects that are likely to be captured simultaneously during imaging.

[0077] <Sixth Embodiment> In the embodiment described above, an example was given in which the feature quantities of a target person registered in another imaging device that meets predetermined conditions are acquired as the feature quantities of another person. Here, for example, it is possible that the same person may be registered as the target person in multiple imaging devices. In this case, the feature quantities of the same person registered in each of those multiple imaging devices will be acquired by each imaging device as the feature quantities of another person. Acquiring such duplicate feature quantities of another person will unnecessarily consume resources and increase processing costs.

[0078] Therefore, in the sixth embodiment, imaging devices may be grouped for each person to be recognized, and other imaging devices may be searched based on whether or not they belong to that group. The setting to group imaging devices for each person to be recognized may be performed by the user of each imaging device, or, for example, by the server if multiple imaging devices are connected via a server. For example, when the user of each imaging device sets up the grouping, unique information such as the email address of each imaging device user can be used. For example, if each imaging device user is recognizing the same person, they will inform each other of their email addresses. The search unit 406 groups the imaging devices used by the users of those email addresses, and when searching for other imaging devices, it excludes imaging devices belonging to that group from the search target. This eliminates the acquisition of duplicate features from other objects, reducing wasted resources and processing costs.

[0079] <Seventh Embodiment> In the third embodiment described above, an example was given in which the search unit 406 of the imaging device requests the server for the features of other objects. In the seventh embodiment, if the server is managing the device status and events of the imaging device, it may appropriately transmit the feature quantities of valid people (for example, people likely to be captured at the same time) to the imaging device that requested the acquisition of other person feature quantities, depending on the progress of the event. According to the seventh embodiment, by the server transmitting valid other person feature quantities to the imaging device at an appropriate time, the imaging device can utilize other person feature quantities at an appropriate time, and an improvement in the accuracy of matching the person to be recognized can be expected.

[0080] The present invention can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, it can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions. The above-described embodiments are merely examples of concrete implementations of the present invention, and the technical scope of the invention should not be limited by them. That is, the present invention can be implemented in various forms without departing from its technical concept or its main features.

[0081] Each embodiment of the disclosure includes the following configurations, methods, and programs. (Composition 1) A detection means for detecting a predetermined subject from an image captured by an imaging device, An extraction means for extracting feature quantities from the image of the predetermined subject detected, A registration means for registering a first feature quantity, which is a feature quantity of the subject to be recognized, and a second feature quantity, which is a feature quantity of a subject other than the subject to be recognized. A matching means that determines whether the detected subject is the subject of the recognition target based on the comparison result between the extracted feature quantity and the first feature quantity, and the comparison result between the extracted feature quantity and the second feature quantity. A means of communication for data transmission, It has, Through the data acquired by the aforementioned data communication, the characteristic quantities of other subjects that are being recognized by another imaging device different from the imaging device that captured the aforementioned image are acquired, and the acquired characteristic quantities of the other subjects are registered as second characteristic quantities by the registration means. An image processing apparatus characterized by the following: (Configuration 2) The system further includes a search means for searching for the aforementioned other imaging device, The image processing apparatus according to configuration 1, characterized in that the communication means communicates with the other imaging device that was searched and obtains the feature quantities of the other subject that the other imaging device is targeting for recognition. (Composition 3) The image processing apparatus according to configuration 2, characterized in that the search means searches for other imaging devices that meet predetermined conditions. (Composition 4) The image processing apparatus according to configuration 3, characterized in that the search means performs the search of the other imaging device, provided that the search means agrees to the policy regarding the use of feature quantities of the other subject that the other imaging device is targeting for recognition, as a predetermined condition. (Composition 5) The image processing apparatus according to configuration 3 or 4, characterized in that the search means performs the search for the other imaging device, with the predetermined condition that it is within a certain distance range from the imaging device. (Composition 6) The image processing apparatus according to any one of configurations 3 to 5, wherein the search means performs the search for the other imaging device, with the predetermined condition that imaging was performed by the other imaging device within a certain period of time after imaging by the imaging device. (Composition 7) The image processing apparatus according to any one of configurations 3 to 6, characterized in that the search means performs the search of the other imaging device, with the predetermined condition that feature extraction is performed by a process common to the extraction means of the imaging device. (Composition 8) The image processing apparatus according to any one of configurations 3 to 7, wherein the search means performs the search of the other imaging device, with the predetermined condition that the other subject to be recognized has been matched with the other imaging device within a certain period of time. (Composition 9) The image processing apparatus according to any one of configurations 2 to 8, characterized in that the search means excludes other imaging devices belonging to groups that recognize the same subject as the recognition target from the search. (Composition 10) The image processing apparatus according to any one of configurations 2 to 9, characterized in that the search means excludes other imaging devices that recognize the same subject as the subject to be recognized from the search. (Composition 11) The communication means acquires an image of the other subject that the other imaging device, which has been searched by the search means, is recognizing through data communication. The extraction means extracts feature quantities from the image of the other subject obtained by the communication, The image processing apparatus according to any one of configurations 2 to 10, characterized in that the registration means registers the feature quantities extracted from the image of the other subject as the second feature quantities. (Composition 12) The image processing apparatus according to any one of configurations 1 to 11, characterized in that the communication means communicates with the other imaging device by predetermined short-range communication. (Composition 13) The image processing apparatus according to any one of configurations 1 to 12, further comprising setting means for setting an expiration date for the second feature quantity. (Composition 14) The image processing apparatus according to any one of configurations 1 to 13, characterized in that the matching means determines that the detected subject is the subject to be recognized if the value of the comparison result between the extracted feature and the first feature is greater than or equal to a predetermined threshold, and the value of the comparison result between the extracted feature and the first feature is greater than the value of the comparison result between the extracted feature and the second feature. (Composition 15) The detection means detects an image of a person's face as the image of the predetermined subject from the image captured by the imaging device. The image processing apparatus according to any one of configurations 1 to 14, characterized in that the extraction means extracts the feature quantities from the image of the person's face. (Composition 16) A predetermined server manages the feature quantities of the subject that each of the multiple imaging devices recognizes, and information about the multiple imaging devices. The communication means is an image processing device according to any one of configurations 1 to 15 that can communicate with the server, It has, The server identifies an imaging device that meets the predetermined conditions among the multiple imaging devices it manages. The image processing device is characterized in that it identifies the imaging device identified by the server as the other imaging device, and registers the feature quantities of other subjects that the other imaging device is recognizing as the second feature quantities. (Composition 17) The aforementioned server, The system also manages the status of each of the multiple imaging devices and information indicating the events being captured by the multiple imaging devices. To each imaging device that recognizes a subject simultaneously captured in the aforementioned event, the feature quantities of the subject to be recognized are transmitted to each other in accordance with the progress of the event. The image processing system according to configuration 16, characterized in that the image processing device registers the feature quantities of the subject that each of the imaging devices is targeting for recognition, which are transmitted from the server, as the second feature quantities. (Composition 18) The aforementioned server, The information of the extraction means that extracts feature quantities from the image of the subject in each of the multiple imaging devices is also managed. The image processing system according to configuration 16 or 17, characterized in that, if the feature extraction models used in the extraction means of the image processing devices having the plurality of imaging devices are different, feature quantities that can be used for matching by the matching means are extracted from the image of the detected subject and provided to each image processing device. (Method 1) A detection step for detecting a predetermined subject from an image captured by an imaging device, An extraction step of extracting feature quantities from the image of the predetermined subject detected, A registration step of registering a first feature quantity, which is a feature quantity of the subject to be recognized, and a second feature quantity, which is a feature quantity of a subject other than the subject to be recognized. A matching step in which the detected subject is determined to be the subject of the recognition target based on the comparison result between the extracted feature quantity and the first feature quantity, and the comparison result between the extracted feature quantity and the second feature quantity. The communication process that performs data communication, It has, Through the data acquired by the aforementioned data communication, feature quantities of other subjects that are being recognized by another imaging device different from the imaging device that captured the aforementioned image are acquired, and the acquired feature quantities of the other subjects are registered as second feature quantities in the registration step. An image processing method characterized by the following: (program) A program that causes a computer to function as an image processing device described in any one of configurations 1 to 18. [Explanation of symbols]

[0082] 401: Acquisition unit, 402: Detection unit, 403: Extraction unit, 404: Registration unit, 405: Verification unit, 406: Search unit, 407: Communication unit

Claims

1. A detection means for detecting a predetermined subject from an image captured by an imaging device, An extraction means for extracting feature quantities from the image of the predetermined subject detected, A registration means for registering a first feature quantity, which is a feature quantity of the subject to be recognized, and a second feature quantity, which is a feature quantity of a subject other than the subject to be recognized. A matching means that determines whether the detected subject is the subject of the recognition target based on the comparison result between the extracted feature quantity and the first feature quantity, and the comparison result between the extracted feature quantity and the second feature quantity. A means of communication for data transmission, It has, Through the data acquired by the aforementioned data communication, the characteristic quantities of other subjects that are being recognized by another imaging device different from the imaging device that captured the aforementioned image are acquired, and the acquired characteristic quantities of the other subjects are registered as second characteristic quantities by the registration means. An image processing apparatus characterized by the following:

2. The system further includes a search means for searching for the aforementioned other imaging device, The image processing apparatus according to claim 1, wherein the communication means communicates with the other imaging device that was searched and obtains the feature quantities of the other subject that the other imaging device is recognizing.

3. The image processing apparatus according to claim 2, characterized in that the search means searches for other imaging devices that meet predetermined conditions.

4. The image processing apparatus according to claim 3, characterized in that the search means performs the search of the other imaging device on the predetermined condition that it agrees to the policy regarding the use of feature quantities of the other subject that the other imaging device is targeting for recognition.

5. The image processing apparatus according to claim 3, characterized in that the search means performs the search for the other imaging device, with the predetermined condition that it is within a certain distance range from the imaging device.

6. The image processing apparatus according to claim 3, wherein the search means performs the search for the other imaging device, with the predetermined condition that imaging was performed by the other imaging device within a certain period of time after imaging by the imaging device.

7. The image processing apparatus according to claim 3, characterized in that the search means performs the search of the other imaging apparatus, provided that the search means performs feature extraction using a process common to the extraction means of the imaging apparatus, as a predetermined condition.

8. The image processing apparatus according to claim 3, wherein the search means performs the search of the other imaging device, with the predetermined condition that the other subject of the recognition target has been matched with the other imaging device within a certain period of time.

9. The image processing apparatus according to claim 2, characterized in that the search means excludes other imaging devices belonging to groups that recognize the same subject as the recognition target from the search.

10. The image processing apparatus according to claim 2, characterized in that the search means excludes other imaging devices that recognize the same subject as the subject to be recognized from the search.

11. The communication means acquires an image of the other subject that the other imaging device, which has been searched by the search means, is recognizing through data communication. The extraction means extracts feature quantities from the image of the other subject obtained by the communication, The image processing apparatus according to claim 2, characterized in that the registration means registers the feature quantities extracted from the image of the other subject as the second feature quantities.

12. The image processing apparatus according to claim 1, characterized in that the communication means communicates with the other imaging device by predetermined short-range communication.

13. The image processing apparatus according to claim 1, further comprising setting means for setting an expiration date for the second feature quantity.

14. The image processing apparatus according to claim 1, characterized in that the matching means determines that the detected subject is the subject to be recognized if the value of the comparison result between the extracted feature and the first feature is greater than or equal to a predetermined threshold, and the value of the comparison result between the extracted feature and the first feature is greater than the value of the comparison result between the extracted feature and the second feature.

15. The detection means detects an image of a person's face as the image of the predetermined subject from the image captured by the imaging device. The image processing apparatus according to claim 1, wherein the extraction means extracts the feature quantities from the image of the person's face.

16. A predetermined server manages the feature quantities of the subject that each of the multiple imaging devices recognizes, and information about the multiple imaging devices. The image processing apparatus according to any one of claims 1 to 15, wherein the communication means is capable of communicating with the server, It has, The server identifies an imaging device that meets the predetermined conditions among the multiple imaging devices it manages. The image processing device is characterized in that it identifies the imaging device identified by the server as the other imaging device, and registers the feature quantities of other subjects that the other imaging device is recognizing as the second feature quantities.

17. The aforementioned server, The system also manages the status of each of the multiple imaging devices and information indicating the events being captured by the multiple imaging devices. To each imaging device that recognizes a subject simultaneously captured in the aforementioned event, the feature quantities of the subject to be recognized are transmitted to each other in accordance with the progress of the event. The image processing system according to claim 16, characterized in that the image processing device registers the feature quantities of the subject that each of the imaging devices is targeting for recognition, which have been transmitted from the server, as the second feature quantities.

18. The aforementioned server, The information of the extraction means that extracts feature quantities from the image of the subject in each of the multiple imaging devices is also managed. The image processing system according to claim 16, characterized in that, if the feature extraction models used in the extraction means of the image processing devices having the plurality of imaging devices are different, feature quantities that can be used for matching by the matching means are extracted from the image of the detected subject and provided to each image processing device.

19. A detection step for detecting a predetermined subject from an image captured by an imaging device, An extraction step of extracting feature quantities from the image of the predetermined subject detected, A registration step of registering a first feature quantity, which is a feature quantity of the subject to be recognized, and a second feature quantity, which is a feature quantity of a subject other than the subject to be recognized. A matching step in which the detected subject is determined to be the subject of the recognition target based on the comparison result of the extracted feature quantity and the first feature quantity, and the comparison result of the extracted feature quantity and the second feature quantity. The communication process that performs data communication, It has, Through the data acquired by the aforementioned data communication, feature quantities of other subjects that are being recognized by another imaging device different from the imaging device that captured the aforementioned image are acquired, and the acquired feature quantities of the other subjects are registered as second feature quantities in the registration step. An image processing method characterized by the following:

20. Computers, A detection means for detecting a predetermined subject from an image captured by an imaging device, An extraction means for extracting feature quantities from the image of the predetermined subject detected, A registration means for registering a first feature quantity, which is a feature quantity of the subject to be recognized, and a second feature quantity, which is a feature quantity of a subject other than the subject to be recognized. A matching means that determines whether the detected subject is the subject of the recognition target based on the comparison result between the extracted feature quantity and the first feature quantity, and the comparison result between the extracted feature quantity and the second feature quantity. A means of communication for data transmission, It has, A program that functions as an image processing device that, via the data acquired through the aforementioned data communication, acquires feature quantities of other subjects that are being recognized by another imaging device different from the imaging device that captured the aforementioned image, and registers the acquired feature quantities of the other subjects as second feature quantities using the registration means.

Images