Information processing apparatus, method, program, and storage medium

Abstract

To improve the probability of successful face recognition by a face recognition system while suppressing the load on the face recognition system.SOLUTION: An information processing apparatus comprises: image acquisition means for acquiring a face image; analysis means for calculating a face likelihood and a face feature amount of the face image acquired by the image acquisition means; recording means for recording a registration image used for face recognition; calculation means for calculating a similarity by comparing the feature amounts between at least one registration image recorded in the recording means and at least one face image having a face likelihood calculated by the analysis means that is equal to or greater than a predetermined face likelihood and having a face feature amount calculated by the analysis means that is equal to or greater than a predetermined face feature amount; and selection means for selecting at least one face image as a registration candidate image in order from the face image with the lowest similarity calculated by the calculation means.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an information processing device, a method, a program, and a storage medium. [Background technology]

[0002] Facial recognition may fail if the photographed face is blurred, the face is not photographed from the front, makeup or hairstyle has changed over time, etc. Measures to avoid such situations include adding facial images to the facial recognition system or changing the facial images registered in the facial recognition system.

[0003] For example, Patent Document 1 discloses a technology that provides a means for comparing a registered image with a facial image input from a camera or the like, and if it is determined to be an impersonation and authentication of the person fails, making the facial image determined to be an impersonation a candidate image to be added to the facial recognition system. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-12452 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the technology disclosed in Patent Document 1, it is difficult to select a face image from a plurality of face images that will improve the probability of successful face recognition. Also, if too many face images are added to the face image system, the number of face images to be matched increases, which may increase the load on the face image system.

[0006] Therefore, an object of the present invention is to provide an information processing device, method, program, and storage medium that can improve the probability of successful face recognition by a face recognition system while reducing the load on the face recognition system. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the information processing device of the present invention is characterized in that it comprises an image acquisition means for acquiring a facial image, an analysis means for calculating a facial likelihood and facial feature amounts of the facial image acquired by the image acquisition means, a recording means in which a registration image used for facial authentication is recorded, a calculation means for calculating a similarity by comparing feature amounts of at least one registration image recorded in the recording means with at least one facial image whose facial likelihood calculated by the analysis means is equal to or greater than a predetermined facial likelihood and whose facial feature amounts calculated by the analysis means are equal to or greater than a predetermined facial feature amount, and a selection means for selecting at least one facial image in order from the facial image with the lowest similarity calculated by the calculation means as a registration candidate image. Effect of the Invention

[0008] According to the present invention, it is possible to improve the probability that face recognition by the face recognition system will be successful while suppressing the load on the face recognition system. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a block diagram illustrating an example of a hardware configuration of an information processing device. [Diagram 2] FIG. 1 is a diagram illustrating an example of a schematic configuration of an image analysis system including an information processing device. [Diagram 3] 1 is a block diagram illustrating an example of a functional configuration of an information processing device according to a first embodiment. [Figure 4] FIG. 2 is a block diagram showing an example of a detailed functional configuration of a block according to the first embodiment. [Diagram 5] 10 is a flowchart showing a procedure of a process for selecting a candidate image according to the first embodiment. [Figure 6] FIG. 4 is a diagram of a user interface for displaying candidate images according to the first embodiment. [Figure 7] FIG. 11 is a diagram illustrating an effect of the first embodiment. [Figure 8]13 is a flowchart showing a procedure of a process for selecting a candidate image when there are a plurality of registered images according to a modification of the first embodiment. [Figure 9] 13 is a flowchart showing the procedure of a process for excluding registered images when a limit is set on the number of face images that can be registered in the face authentication system according to the second embodiment. [Figure 10] 13A and 13B are diagrams for explaining a case in which the rate of erroneous recognition as a person other than the intended person increases according to the third embodiment. [Figure 11] 13 is a flowchart showing a procedure of a process for selecting, as a candidate image, an image that can be correctly authenticated as the person in question, including preventing erroneous recognition as a person other than the person in the third embodiment. [Figure 12] 13A and 13B are diagrams for explaining a process for selecting, as a candidate image, an image that can be correctly authenticated as the person in question, including preventing erroneous recognition as a person other than the person in the third embodiment. [Figure 13] A flowchart showing the procedure of a process for displaying auxiliary information on a registered image in order to set an image that is correctly authenticated as the person in question as a candidate image, including preventing erroneous recognition as a person other than the person in a modified example of the third embodiment. [Figure 14] FIG. 13 is a diagram showing a user interface for displaying auxiliary information on a registered image according to a modified example of the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] An information processing device according to an embodiment includes an image acquisition means, an analysis means, a recording means, a calculation means, and a selection means. The image acquisition means acquires a face image generated by photographing the face of a person for whom the accuracy of face recognition is to be improved. The analysis means calculates a face likelihood and a face feature amount of the face image acquired by the image acquisition means. The recording means records a registration image used for face recognition. The calculation means calculates a similarity by comparing the feature amounts of at least one registration image recorded in the recording means with at least one face image for which the face likelihood calculated by the analysis means is equal to or greater than a predetermined face likelihood and the face feature amount calculated by the analysis means is equal to or greater than a predetermined face feature amount. The selection means selects at least one face image as a registration candidate image in order from the face image with the lowest similarity calculated by the calculation means.

[0011] Also, a limit may be set on the number of registered images that can be recorded in the recording means based on the performance of the face authentication system. In this case, if the number of registered images exceeds the limit when a face image is recorded in the recording means as a registered image, the calculation means calculates the sum of similarities for the face image acquired by the image acquisition means and the registered images already recorded in the recording means. Then, the selection means selects the face image acquired by the image acquisition means as a candidate image for registration, and replaces the registered image with the highest calculated sum of similarities.

[0012] The recording means may record face images of persons other than the person. In this case, the selection means selects, as the registration candidate image, a face image having the lowest similarity calculated by the calculation means, when all of the similarities between the face images of all persons other than the person are less than a predetermined similarity.

[0013] The recording means may record a face image of a person different from the person and a result of past matching of face images with each other.The information processing device according to the embodiment may further include a display means.When the face image with the lowest similarity calculated by the calculation means is matched with each face image of all people different from the person, and a face image determined to be the same person as the face image with the lowest similarity calculated by the calculation means is present, the display means executes the following process.That is, in such a case, the display means displays auxiliary information based on the face image and the face image with the lowest similarity calculated by the calculation means.

[0014] The auxiliary information may also indicate the latest date and time when matching was performed on a face image determined to include the same person as the face image with the lowest similarity calculated by the calculation means. Alternatively, the auxiliary information may indicate the ratio of the face image matched with the face image to all face images recorded in the recording unit. Alternatively, the auxiliary information may indicate a face image determined to include the same person as the face image with the lowest similarity calculated by the calculation means. Alternatively, the auxiliary information may indicate the date and time when matching was performed on each registered image and the ratio of matching on each registered image.

[0015] The recording means may record a face image of a person different from the person. The information processing device according to the embodiment may further include a display means. When a person different from the face image with the lowest similarity calculated by the calculation means is photographed and there is a face image determined by the calculation means to have a similarity to the person equal to or higher than a predetermined similarity, the display means executes the following process. That is, in such a case, the display means displays the face image and a result of past matching of the face image. Note that, before this process, the face image with the lowest similarity calculated by the calculation means is matched with each face image of all people different from the person.

[0016] The recording means may record a face image of a person different from the person. The information processing device according to the embodiment may further include a display means. The display means executes the following process when a person different from the face image with the lowest similarity calculated by the calculation means is photographed and there is a face image determined by the calculation means to have a similarity to the person equal to or higher than a predetermined similarity. That is, in such a case, the display means displays a user interface for adjusting a threshold value so that the similarity between the face image of the person and the face image of the person different from the person is less than the predetermined similarity. Note that, before this process, the face image with the lowest similarity calculated by the calculation means is compared with each face image of all people different from the person.

[0017] The method according to the embodiment acquires a face image, and calculates a face likelihood and a face feature amount of the face image acquired by the image acquisition means. Next, the method according to the embodiment calculates a similarity by comparing the feature amount of at least one registered image used for face authentication and recorded in a recording means with at least one face image. The at least one face image is a face image in which the face likelihood calculated by the analysis means is equal to or greater than a predetermined face likelihood, and the face feature amount calculated by the analysis means is equal to or greater than a predetermined face feature amount. Then, the method according to the embodiment selects at least one face image as a candidate image for registration in order from the face image with the lowest similarity calculated by the calculation means.

[0018] The program according to the embodiment acquires a face image, and calculates a face likelihood and a face feature amount of the face image acquired by the image acquisition means. Next, the program according to the embodiment calculates a similarity by comparing the feature amount of at least one registered image used for face authentication and recorded in a recording means with at least one face image. The at least one face image is a face image in which the face likelihood calculated by the analysis means is equal to or greater than a predetermined face likelihood, and the face feature amount calculated by the analysis means is equal to or greater than a predetermined face feature amount. Then, the program according to the embodiment selects at least one face image as a candidate image for registration in order from the face image with the lowest similarity calculated by the calculation means.

[0019] A storage medium according to an embodiment stores a program. The program acquires a face image and calculates a face likelihood and a face feature amount of the face image acquired by the image acquisition means. Next, the program calculates a similarity by comparing the feature amount of at least one registered image used for face authentication and recorded in a recording means with at least one face image. The at least one face image is a face image in which the face likelihood calculated by the analysis means is equal to or greater than a predetermined face likelihood and the face feature amount calculated by the analysis means is equal to or greater than a predetermined face feature amount. Then, the program selects at least one face image as a registration candidate image in order from the face image with the lowest similarity calculated by the calculation means.

[0020] Hereinafter, the embodiments for carrying out the present invention will be described in detail with reference to the attached drawings. Note that the embodiments described below are examples of means for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiments.

[0021] First Embodiment 1 is a block diagram showing an example of a hardware configuration of an information processing device. An information processing device 100 according to a first embodiment includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a storage 104, and a communication I / F 105. The ROM 102, the RAM 103, the storage 104, and the communication I / F 105 are connected to an internal bus 106.

[0022] The CPU 101 is a central processing unit that performs overall control of operations in the information processing device 100. The ROM 102 is a non-volatile memory that stores programs and various initial setting data required for the CPU 101 to execute processing. The RAM 103 is a volatile memory that functions as the main memory, work area, etc. of the CPU 101, and temporarily stores programs, image data, etc. When executing processing, the CPU 101 loads the necessary programs, etc. from the ROM 102 into the RAM 103 and executes the programs, etc. to realize various functional operations.

[0023] The storage 104 is a recording device having a larger capacity than the RAM 103, and is, for example, a hard disk drive (HDD) or a solid state drive (SSD). The storage 104 stores an operating system (OS), which is basic software executed by the CPU 101, face likelihood according to the first embodiment, a face feature amount analysis program, a similarity calculation program, and various other data. The storage 104 can also record image data acquired via a network and registered images for face authentication.

[0024] When the information processing device 100 is started by turning on the power or the like, the CPU 101 executes a startup program stored in the ROM 102. This startup program is for reading the OS stored in the storage 104 and loading it in the RAM 103. After the OS is started, for example, a user issues an instruction via the operation unit 107 or the like shown in FIG. 2 to start a process of analyzing a face likelihood, a process of analyzing a face feature amount, and a process of calculating a similarity. In this case, the CPU 101 reads a face likelihood analysis program, a face feature amount analysis program, and a similarity calculation program from the storage 104 and loads them in the RAM 103. This enables the CPU 101 to execute the face likelihood analysis process, the face feature amount analysis process, and the similarity calculation process. The CPU 101 also stores various data used in the operation of the face likelihood analysis program, the face feature amount analysis program, and the similarity calculation program in the RAM 103 and reads and writes them.

[0025] The communication I / F 105 is, for example, an interface for a local area network (LAN), and communicates with a network camera, other computers, and the like via the network. Note that the image handled in the analysis of the face likelihood and face feature amount in this embodiment, the analysis program for executing the analysis of the face likelihood and face feature amount and the calculation of the similarity, and various data used in these analysis processes may be acquired via the network. The information processing device 100 may be configured by one device having the configuration shown in FIG. 1, such as a personal computer (PC), tablet PC, or the like. However, the configuration shown in FIG. 1 may be configured by separate hardware. That is, the information processing device 100 may be configured by a plurality of devices. In addition, the information processing device 100 may process the arithmetic processing associated with the analysis of the face likelihood and face feature amount and the calculation of the similarity using a GPU (not shown).

[0026] FIG. 2 is a diagram showing an example of a schematic configuration of an image analysis system including an information processing device. The image analysis system 200 includes an information processing device 100, a storage device 109, and a camera 110. The information processing device 100, the storage device 109, and the camera 110 are connected to each other via a network 111 in a manner that allows them to communicate with each other. The network 111 can be, for example, a LAN. The network 111 may adopt any communication standard, scale, and configuration as long as it is configured to allow communication between the information processing device 100, the camera 110, and the storage device 109. In addition, the physical connection to the network 111 may be wired or wireless. Furthermore, the connection between the information processing device 100, the storage device 109, and the camera 110 is not limited to the network 111, and may be, for example, a USB (Universal Serial Bus) or the like.

[0027] 1, the information processing device 100 may include an operation unit 107 and a display unit 108 as a hardware configuration. Here, the operation unit 107 includes a pointing device such as a keyboard and a mouse. The display unit 108 is, for example, a monitor such as a liquid crystal display (LCD), and is a display device for viewing when the user operates the information processing device 100.

[0028] The camera 110 is, for example, a network camera, and is an imaging device having a function of capturing an image within an imaging range and transmitting the captured image to the information processing device 100 via the network 111. The camera 110 may be a pan tilt zoom (PTZ) camera configured to be able to change the imaging area. In this case, the camera 110 may have a function of transmitting imaging parameters including information on the imaging direction and imaging angle of view, such as a pan angle, a tilt angle, and a zoom magnification, to the information processing device 100 via the network 111.

[0029] In the first embodiment, a case will be described in which the camera 110 is a network camera operated as a surveillance camera, but the present invention is not limited to this. For example, the camera 110 may be a digital still camera, a digital video camera, a smartphone equipped with a camera function, a tablet terminal, an industrial camera, an in-vehicle camera, a wearable camera, or the like. The information processing device 100 receives an image captured by the camera 110 as an input image from the camera 110 via the network 111, and performs processing on the received input image, which will be described later.

[0030] The camera 110 may transmit the captured image to the storage device 109, a storage device of another device, or the like via the network 111. In this case, the information processing device 100 may receive the captured image stored in the storage device 109, a storage device of another device, or the like as an input image via the network 111. Here, the input image is not limited to the captured image captured by the camera 110. For example, the input image may be a partial image that is a part of the captured image. In addition, the configuration of the image analysis system 200 shown in FIG. 2 is an example, and each device may be integrated like a tablet PC or the like. That is, the information processing device 100 may have the function of the camera 110.

[0031] The information processing device 100 according to the first embodiment acquires an image including a person who adds a face image to the face recognition device or replaces a face image registered in the face recognition device via a camera 110 or the like. Here, the image including the subject may be acquired from the storage device 109 via a network 111. Alternatively, the image including the subject may be an image stored in the storage 104.

[0032] In the first embodiment, after an image is acquired via the camera 110, a group of images stored in the storage 104 are stored in the ROM 102. Then, according to the face likelihood analysis program and the face feature amount analysis program expanded in the RAM 103, an image of a person whose face likelihood and face feature reliability exceed a threshold value is selected.

[0033] The facial features of the selected face image are compared with those of the registered image of the target person, and the image with the lowest similarity is selected as the candidate image for registration. The candidate images are displayed on the screen user interface (UI), and the user can decide whether or not to register them.

[0034] FIG. 3 is a block diagram showing an example of a functional configuration of an information processing device according to a first embodiment. As shown in FIG. 3, the information processing device 100 includes an image acquisition unit 301, a recording unit 302, an analysis unit 303, and a calculation unit 304. The image acquisition unit 301 acquires an image of a target person. The recording unit 302 records a registered image of the target person. The analysis unit 303 analyzes face likelihood and face feature reliability, that is, the likelihood of face organ features. The calculation unit 304 compares the selected face image with the registered image of the target person to calculate a similarity. Then, the calculation unit 304 selects an image with the lowest similarity as a candidate image for registration.

[0035] 4 is a block diagram showing an example of a detailed functional configuration of the blocks according to the first embodiment. The analysis unit 303 includes a face detection unit 401 and a face organ feature acquisition unit 402. The face detection unit 401 executes face detection to detect the facial portion of a target person from an image of the person. The face detection detects a portion that is likely to be a human face, that is, a portion that exceeds a threshold value of the face likelihood, by applying a classifier that has learned human faces to an input image. Then, the face detection unit 401 executes a threshold determination on the face likelihood obtained by the face detection.

[0036] The facial organ feature acquisition unit 402 acquires features of facial organs of a face detected by face detection. The facial organs are, for example, the eyes, nose, and mouth. By applying a classifier that has learned the facial organs, parts that seem to be each organ of the face, that is, parts that exceed the threshold of the likelihood of the facial organ, are detected. Then, the facial organ feature acquisition unit 402 performs threshold judgment on the obtained likelihood of the facial organs. The calculation unit 304 includes a feature amount matching unit 403, a similarity calculation unit 404, and a candidate image selection unit 405.

[0037] The feature amount matching unit 403 matches the facial features of the image of the person acquired by the image acquisition unit 301 with the registered image recorded in the recording unit 302. The facial features are expressions of the features of each face organ in a multidimensional space. The facial feature information is plotted in the multidimensional space.

[0038] The similarity calculation unit 404 calculates the similarity of the features matched by the feature matching unit 403. The similarity is expressed as the distance between the facial features plotted in the multidimensional space. The closer the distance in the multidimensional space, the more similar the facial features are, and the farther the distance in the multidimensional space, the more dissimilar the facial features are.

[0039] A candidate image selection unit 405 selects an image with the lowest similarity calculated by the similarity calculation unit 404 as a candidate image.

[0040] At least some of the functions of each element of the information processing device 100 shown in Figures 3 and 4 can be realized by the CPU 101 executing a program. However, at least some of the elements of the information processing device 100 shown in Figure 3 may be configured to operate as dedicated hardware. In this case, the dedicated hardware operates under the control of the CPU 101.

[0041] Hereinafter, the functions of the information processing device 100 shown in FIGS. 3 and 4 will be described in detail with reference to FIGS.

[0042] Fig. 5 is a flowchart showing the procedure of a process for selecting candidate images according to the first embodiment. Fig. 6 is a diagram of a user interface for displaying candidate images according to the first embodiment. The user interface shown in Fig. 6 is composed of a registered image addition dialog 601, a registered person name 602, a registered image 603, and a batch input button 604. Also, a candidate image display dialog 611 is a user interface after a process for selecting candidate images is executed for the registered image addition dialog 601.

[0043] In the candidate image display dialog 611, a candidate image 612 is selected as a candidate image by executing a process for selecting a candidate image. The registered image to be added to the face authentication system is confirmed by the user pressing a candidate image confirmation button 613.

[0044] The process shown in Fig. 5 starts when the user presses the batch input button 604. Each process shown in Fig. 5 can be realized by the CPU 101 of the information processing device 100 reading and executing a necessary program.

[0045] First, in the process of step S5001, the image acquisition unit 301 receives an instruction to add a registration image. In this process, the image acquisition unit 301 acquires an image of the person in question. Moreover, the image is an image for which authentication has failed. Moreover, the image acquisition unit 301 may acquire multiple images. The image acquisition unit 301 may acquire the image from the storage 104 included in the information processing device 100. Moreover, the image may be acquired from the storage device 109 via the network 111 through the communication I / F 105. Moreover, the image acquisition unit 301 may acquire video from the camera 110 and input it as multiple images.

[0046] In the process of step S5002, the analysis unit 303 analyzes the face likelihood and face features of the image acquired in step S5001. The image may be a group of multiple images, and when a group of images is acquired, the face likelihood and face organ likelihood are analyzed for all images. The analysis of the face likelihood is performed by the face detection unit 401 of the analysis unit 303. The analysis of the face organ likelihood is performed by the face organ feature amount acquisition unit 402 of the analysis unit 303.

[0047] In the process of step S5003, the analysis unit 303 judges whether the face likelihood and the face organ likelihood acquired in step S5002 exceed the threshold value. If the analysis unit 303 judges that the face likelihood and the face organ likelihood do not exceed the corresponding threshold value (S5003 / No), it ends the process as there is no face suitable for registration. On the other hand, if the analysis unit 303 judges that the face likelihood and the face organ likelihood acquired in step S5002 exceed the threshold value (S5003 / Yes), it executes the process of step S5004.

[0048] Steps S5004 to S5007 represent the process flow for comparing the input image of the subject with registered images and acquiring the image with the lowest similarity.

[0049] After performing the processes from step S5004 to step S5007 on all images of the subject, the similarity calculation unit 404 executes the process of step S5008.

[0050] In the process of step S5008, the candidate image selection unit 405 displays the input image with the lowest similarity as a candidate image on the UI. A candidate image display dialog 611 shown in Fig. 6 is an example of such a display.

[0051] Next, the effect of making face authentication less likely to fail in the processing of the first embodiment will be described with reference to Fig. 7. Fig. 7 is a diagram for explaining the effect of the first embodiment. Fig. 7 shows a face feature space on a two-dimensional plane. Note that while Fig. 7 shows the face feature amount space on a two-dimensional plane, the actual face feature amount space is often a multi-dimensional space of three or more dimensions.

[0052] 7 are images of the same person. The registered image 701 is a registered image of a face authentication system. The candidate images 711 and 721 are examples of input images. Points 702, 712, and 722 are coordinates of facial features corresponding to each face image. Ranges 703, 713, and 723 indicate ranges of features for determining that the image is the same person as the corresponding image, for the coordinates of each facial feature.

[0053] 7, the distance between points 702 and 722 is longer than the distance between points 702 and 712. Therefore, it can be said that candidate image 721 and registered image 701 are less similar than candidate image 711 and registered image 701. Also, in FIG. 7, the overlapping range between range 703 and range 723 is smaller than the overlapping range between range 703 and range 713. Therefore, in a face recognition system, if candidate image 721 is used as the registered image, the range for authenticating the person as the real person is wider, and the rate of authentication is increased.

[0054] As explained with reference to Figure 7, the correct recognition rate can be effectively improved by comparing the image with the existing registered images of the person for whom recognition failures are to be improved, and then registering the image with the lowest similarity as a registration candidate image in the face recognition system.

[0055] (Modification of the first embodiment) In contrast to the first embodiment described above, a process in the case where there are already multiple registered images will be described. Fig. 8 is a flowchart showing the procedure of a process for selecting a candidate image when there are multiple registered images according to a modified example of the first embodiment. Fig. 8 shows the difference from the process shown in Fig. 5. Specifically, this difference is the difference between the process from step S5005 to step S5006 and the process from step S8001 to step S8004.

[0056] Steps S5004 to S5007 are processes for calculating the sum of the similarities between each input image of the target person and all registered images of the target person. The process of step S8001 is executed after it is determined in step S5004 shown in FIG. 5 that there is an input image whose similarity has not yet been calculated.

[0057] In the processes from step S8001 to step S8003, the similarities between the input image of the target person and all registered images are calculated, and the sum is calculated. In the processes from step S8001 to step S8004, the similarity calculation unit 404 selects the image with the lowest sum of similarities. After the candidate image selection unit 405 has finished processing all input images of the target person, it executes the process of step S5008 shown in FIG. 5 to select candidate images.

[0058] As shown in Figure 8, when there are multiple registered images and you want to select a candidate image, calculate the similarity between all registered images and the input image, and select the image with the lowest total as the candidate image. The lowest total similarity means that when the registered image and the candidate image are plotted in feature space, the features are the furthest from all images. Therefore, by comparing the image with the lowest total similarity compared to the existing registered images of the person for whom you want to improve authentication failures, and registering it in the system, you can effectively improve the correct authentication rate.

[0059] Second Embodiment In the second embodiment, when a limit is set on the number of images that can be registered in a face authentication system, the similarity between all combinations of the registered image of the person and the registration candidate images is calculated, and an image other than the registration candidate images that has the maximum sum of the similarity of all combinations is excluded from the registered images.

[0060] In the second embodiment, a case will be described where the number of images that can be registered in the face authentication system is a maximum of four per person. The limit on the number of images that can be registered may be changed within the range of the permissible performance of the face authentication system. Fig. 9 is a flowchart showing the procedure of a process for excluding registered images when a limit is set on the number of face images that can be registered in the face authentication system according to the second embodiment.

[0061] Step S9001 is executed after the batch input button 604 shown in FIG. 6 is pressed. In step S9001, the image acquisition unit 301 determines whether the number of registered images of the target person is less than the limit of the face authentication system. When the image acquisition unit 301 determines that the number of registered images of the target person is less than the limit of the face authentication system (step S9001 / Yes), it is sufficient to select a candidate image. This process has been described in the first embodiment, so a detailed description will be omitted in this embodiment. On the other hand, when the image acquisition unit 301 determines that the number of registered images of the target person is equal to or greater than the limit of the face authentication system (S9001 / No), the process proceeds to step S9002.

[0062] In step S9002, the analysis unit 303 and the calculation unit 304 execute the process shown in FIG. 5 and FIG. 8 in the first embodiment, that is, the process of selecting a registration candidate image.

[0063] In step S9003, the recording unit 302 acquires all existing registered images of the person in question.

[0064] In step S9004, the calculation unit 304 calculates, for all registered images, one-to-one similarities between all registered images of the person corresponding to the person and the registration candidate images, and calculates the sum of these similarities.

[0065] In step S9005, the candidate image selection unit 405 excludes the image with the largest sum of similarities among the registered images. The excluded registered image is replaced with a candidate image. Through the above process, it becomes possible to add candidate images even if a limit is set on the number of images that can be registered in the face authentication system.

[0066] Third embodiment In the third embodiment, a face authentication system is described that compares a registration candidate image with registered images of other people in the face authentication system, and selects an image that does not exceed a matching threshold for determining that the person is the person in the face authentication system and has the lowest similarity with the registered images of the person as the registration candidate image. This makes it possible to select an image that is correctly recognized as the person, including an image that is not erroneously recognized as a different person, as the candidate image.

[0067] A case where a person appearing in a registered image and a registered candidate image is erroneously recognized as a person other than the person will be described with reference to FIG. 10. FIG. 10 is a diagram for explaining a case where the rate of erroneous recognition as a person other than the person according to the third embodiment increases. FIG. 10 shows a registered image of the person, a candidate image of the person, and a registered image of another person in a two-dimensional feature space. As shown in FIG. 10, when a registered image 10001, a candidate image 10021, and a registered image 10011 of another person are shown in the feature space, a range in which the candidate image 10021 is determined to be the same person and a range in which the registered image 10011 of another person is determined to be the same person overlap. When the ranges in which the person is determined to be the same person overlap, a person in the candidate image 10021 may be erroneously recognized as a person in the registered image 10011 of another person.

[0068] The process according to the third embodiment will be described with reference to Fig. 11. Fig. 11 is a flowchart showing the procedure of the process of selecting, as a candidate image, an image that correctly authenticates the person, including preventing erroneous recognition as a person other than the person according to the third embodiment. In the third embodiment, a case where the number of registered images is one will be described following the first embodiment, but the third embodiment can be applied regardless of the number of registered images.

[0069] Steps S5002 to S5005 are the same as the processes described in the first embodiment, and therefore the description will be omitted. In step S11006, the similarity calculation unit 404 records the similarity between the input image and the registered image calculated in the process of step S5005.

[0070] In the process of step S5002, if the process has been completed for all input images of the target person, the process of step S11007 is executed.

[0071] In step S11007, the similarity calculation unit 404 obtains the combination of the input image and the registered image with the lowest similarity from the list of similarities between the recorded input image and the registered image.

[0072] In step S11008, the candidate image selection unit 405 acquires all other people's faces held in the face authentication system from the recording unit 302, compares them with the candidate images, and calculates the similarity.

[0073] In step S11009, if there is another person whose similarity exceeds the threshold value (S11009 / Yes), the candidate image selection unit 405 deletes the item with the lowest similarity from the list of similarities between the recorded input image and registered images.

[0074] In step S11009, if there is no other person whose similarity exceeds the threshold value (S11009 / No), the candidate image selection unit 405 executes the process of step S11010.

[0075] In step S11010, the candidate image selection unit 405 sets the input image, for which no other person with a similarity exceeding the threshold value was found in the process of step S11009, as a registration candidate image.

[0076] Fig. 12 is a diagram for explaining a process for selecting an image that is correctly authenticated as the person in question as a candidate image, including preventing erroneous recognition as a person other than the person in the third embodiment. Fig. 12 shows a registered image 12000 and a table 12001 showing the similarity between the input image and the registered image and the number of similar faces of other people.

[0077] In FIG. 12, there are three input images as shown in the input image table 12001 corresponding to the registered image 12000. The similarities between these three input images and the registered image are 400 pt, 500 pt, and 550 pt, respectively. The number of other people's faces similar to these three input images are "3", "1", and "0", respectively. The processing of the third embodiment makes it possible to select a face image with the lowest similarity, with the number of similar other people's faces being "0". This makes it possible to improve the correct recognition rate so as to prevent erroneous recognition with other people from occurring.

[0078] (Modification of the third embodiment) As described in the third embodiment, adding a registration candidate image may increase the rate of misrecognition as a person other than the person in question. In this case, it is possible to adjust the rate of misrecognition as a person other than the person in question by changing the registration candidate images to images other than the person in question so as not to increase the rate of misrecognition as a person other than the person in question.

[0079] In the third embodiment, a means is described for displaying auxiliary information on a registered image, allowing a user to determine which images to register in the system and which to delete, in order to correctly identify an image that is correctly identified as the person in question, including avoiding misidentification as a third party.

[0080] In the third embodiment, a process for calculating the similarity between an input image and a registered image of a person appearing in the input image, and a process for matching the input image with registered images of other people and calculating the number of face images that include similar faces of other people were described.

[0081] The difference from the process shown in Fig. 11 will be described with reference to Fig. 13. Fig. 13 is a flow chart showing the procedure of a process for displaying auxiliary information on a registration image in order to set an image that is correctly authenticated as the person in question as a registration candidate image, including preventing erroneous recognition of the person as a different person, according to a modified example of the third embodiment. Note that the process from step S5002 to step S11006 in Fig. 13 is the same as that in Fig. 10, and therefore description thereof will be omitted.

[0082] In step S13007, the candidate image selection unit 405 selects the acquired image as a candidate image. The registration candidate image is an image with the lowest similarity between the input image of the target person and the registered image. Alternatively, the registration candidate image is an image with the lowest similarity between the input image of the target person and the registered image among face images in which the number of similar face images of other people is less than a threshold. In the third embodiment, the case will be described in which the registration candidate image is an image with the lowest similarity to the registered image and has two similar faces of other people.

[0083] The operation of the user interface after the process shown in Fig. 13 is completed and a dialog box displaying auxiliary information for the user is displayed will be described with reference to Fig. 14. Fig. 14 is a diagram showing a user interface for displaying auxiliary information on a registered image according to a modified example of the third embodiment. A post-display candidate image display dialog box 14000 includes a candidate image 14001 and a similar face number display 14002.

[0084] Candidate image 14001 displays the registration candidate image selected in the process of step S13007. Similar face count display 14002 displays the number of other person's face images that are similar to the registration candidate image. In this embodiment, since there are two similar other person's face images, "2" is displayed. Note that this display is not limited to numbers, and may be an icon or the like indicating that there are other person's face images with a high degree of similarity. When a registration candidate image is selected by the user's mouse operation, a similar face detail dialog 14003 is displayed.

[0085] The similar face detail dialog 14003 displays a face similar to the candidate image and the name of the person. A registration cancel button 14004 may be displayed, and when the registration cancel button is pressed, it is possible to select not to register the registration candidate image. In this case, the user may select not to register the registration image. The similar face detail dialog 14003 may display an icon 14005 for a face image that shows a face similar to the registration candidate image.

[0086] When the user selects the icon 14005 by operating the mouse, a similar other person's face details dialog 14006 may be displayed. Also, the similar other person's face image details dialog 14006 may be displayed by pressing a button, an image, or the like itself instead of an icon.

[0087] The similar other person's face detail dialog 14006 includes a similar other person's face information display section 14007, a similar other person's face image 14008, similar other person's face matching information 14009, and a second similar other person's face image 14010. The similar other person's face detail dialog 14006 also includes second similar other person's face matching information 14011, a similar other person's face deletion button 14011, and a back button 14012.

[0088] The similar other person's face image 14008 displays a face image of another person similar to the registration candidate image. The similar other person's face matching information 14009 displays information based on the past matching history of the similar other person's face image 14008. For example, the percentage of the similar other person's face image 14008 used for identity matching in a certain past period, the date and time of the last matching, etc. may be displayed. The second similar other person's face image 14010 is another face image of a person appearing in the similar other person's face image 14008 that is not similar to the registration candidate image. The second similar other person's face image 14010 increases or decreases depending on the number of face images registered in the face image system.

[0089] The second similar other person's face matching information 14011 displays information based on the past matching history of the second similar other person's face image 14010. When the delete button 14013 is pressed, the similar other person's face image 14006 is excluded from the registered images. The user can check the similar other person's face image 14008, the similar other person's face matching information 14009, the second similar other person's face image 14010, and the second similar other person's face matching information 14011 and select whether or not to delete the face image of another person similar to the registration candidate image.

[0090] For example, if the matching rate in the similar other face matching information 14009 is extremely low compared to other images, or if matching was last performed more than one year ago, it can be determined whether deletion is appropriate. It is also possible to check whether there is a problem with the face image itself, such as whether the similar other person's face image 14008 has a large portion of its facial features hidden, the face is not in focus, etc. When the user has determined whether to delete the similar other person's face image or whether to register a registration candidate image, the user can end the editing work by pressing the back button 14013. These displays described in this embodiment are merely examples, and the shape, arrangement, screen configuration, etc. are not limited to these as long as they include elements necessary to realize the functions.

[0091] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0092] The present invention includes the following inventions that appropriately combine the above-mentioned features.

[0093] (Configuration 1) an information processing device comprising: an image acquisition means for acquiring a facial image; an analysis means for calculating a facial likelihood and facial feature amount of the facial image acquired by the image acquisition means; a recording means for recording a registration image used for facial authentication; a calculation means for calculating a similarity by comparing feature amounts of at least one registration image recorded in the recording means with at least one facial image whose facial likelihood calculated by the analysis means is equal to or greater than a predetermined facial likelihood and whose facial feature amount calculated by the analysis means is equal to or greater than a predetermined facial feature amount; and a selection means for selecting at least one facial image in order from the facial image with the lowest similarity calculated by the calculation means as a registration candidate image.

[0094] (Configuration 2) The information processing device described in configuration 1, characterized in that a limit based on the performance of a face authentication system is set for the number of registration images that can be recorded in the recording means, and when the number of registration images exceeds the limit when a face image is recorded in the recording means as a registration image, the calculation means calculates a sum of the similarity for each of the face image acquired by the image acquisition means and the registration images already recorded in the recording means, and the selection means sets the face image acquired by the image acquisition means as a registration candidate image and replaces it with the registration image for which the highest sum of the similarity is calculated.

[0095] (Configuration 3) The information processing device according to configuration 1 or 2, characterized in that the recording means records facial images of persons other than the person, and the selection means selects, as a registration candidate image, a facial image with the lowest similarity calculated by the calculation means, when the similarity between each of the facial images of all persons other than the person is less than a predetermined similarity.

[0096] (Configuration 4) the recording means records facial images of persons other than the person and results of past matching between the facial images; and the information processing device further comprises a display means for matching the facial image with the lowest similarity calculated by the calculation means with each of the facial images of all persons other than the person, and, if there is a facial image determined to be of the same person as the facial image with the lowest similarity calculated by the calculation means, displaying auxiliary information based on the facial image and the facial image with the lowest similarity calculated by the calculation means.

[0097] (Configuration 5) The information processing device according to configuration 4, characterized in that the auxiliary information indicates at least one of the most recent date and time when matching was performed on a face image determined to be of the same person as the face image with the lowest similarity calculated by the calculation means, the ratio of the face image matched with the face image to all face images recorded in the recording unit, the face image determined to be of the same person as the face image with the lowest similarity calculated by the calculation means, the date and time when matching was performed on each registered image, and the ratio of matching on each registered image.

[0098] (Configuration 6) the recording means records a facial image of a person different from the person and a result of past matching of the facial images with each other, the facial image having the lowest similarity calculated by the calculation means is matched with each of the facial images of all people different from the person, and when a facial image of a person different from the facial image having the lowest similarity calculated by the calculation means is photographed and the calculation means determines that the similarity with the person is equal to or higher than a predetermined similarity, the information processing device according to any one of configurations 1 to 5 is further characterized in comprising a display means for displaying the facial image and a result of past matching of the facial image.

[0099] (Configuration 7) the recording means records a facial image of a person different from the person and a result of past matching of the facial images with each other, the facial image with the lowest similarity calculated by the calculation means is matched with each of the facial images of all people different from the person, a person different from the facial image with the lowest similarity calculated by the calculation means has been photographed, and when there is a facial image whose similarity with the person is determined by the calculation means to be equal to or greater than a predetermined similarity, the information processing device according to any one of configurations 1 to 6 is further characterized in that the recording means records a facial image of a person different from the person and a result of past matching of the facial images with each other, the recording means further comprises a display means for displaying a user interface for adjusting a threshold value so that the similarity between the facial image of the person and the facial image of the person different from the person becomes less than the predetermined similarity when the facial image with the lowest similarity calculated by the calculation means is photographed and the calculation means determines that the similarity with the person is equal to or greater than a predetermined similarity.

[0100] (Method 1) a facial image acquisition means for acquiring a facial image, a facial likelihood and facial feature amounts of the facial image acquired by the image acquisition means, a similarity degree is calculated by comparing the feature amounts of at least one registration image used for facial authentication and recorded in a recording means with at least one facial image whose facial likelihood calculated by the analysis means is equal to or greater than a predetermined facial likelihood and whose facial feature amounts calculated by the analysis means are equal to or greater than a predetermined facial feature amount, and a method for selecting at least one facial image in order of decreasing similarity degree calculated by the calculation means as a candidate image for registration.

[0101] (Program 1) a program for acquiring a facial image, calculating a facial likelihood and facial feature amounts of the facial image acquired by the image acquisition means, calculating a similarity by comparing the feature amounts of at least one registered image used for facial authentication and recorded in a recording means with at least one facial image whose facial likelihood calculated by the analysis means is equal to or greater than a predetermined facial likelihood and whose facial feature amounts calculated by the analysis means are equal to or greater than a predetermined facial feature amount, and selecting at least one facial image in order from the facial image with the lowest similarity calculated by the calculation means as a candidate image for registration.

[0102] (Storage medium 1) A storage medium for storing a program that acquires a facial image, calculates a facial likelihood and facial feature amounts of the facial image acquired by the image acquisition means, calculates a similarity by comparing the feature amounts of at least one registered image used for facial authentication and recorded in a recording means with at least one facial image whose facial likelihood calculated by the analysis means is equal to or greater than a predetermined facial likelihood and whose facial feature amounts calculated by the analysis means are equal to or greater than a predetermined facial feature amount, and selects at least one facial image in order from the facial image with the lowest similarity calculated by the calculation means as a candidate image for registration.

[0103] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a recording medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit for implementing one or more of the functions, such as an ASIC (Application Specific Integrated Circuit).

[0104] The above describes preferred embodiments of the present invention. However, the present invention is not limited to the above-described embodiments. In other words, the present invention includes embodiments in which various modifications have been made based on the gist of the present invention, and these embodiments are not excluded from the scope of the present invention. [Explanation of symbols]

[0105] 301: Image acquisition unit 302: Recording Department 303:Analysis Department 304: Calculation Department 401: Face detection unit 402: Facial organ feature acquisition unit 403: Feature matching unit 404: Similarity calculation unit 405: Candidate image selection unit

Claims

1. An image acquisition means for acquiring a face image; an analysis means for calculating a face likelihood and a face feature amount of the face image acquired by the image acquisition means; A recording means for recording a registration image used for face authentication; a calculation means for calculating a similarity by comparing features between at least one registered image recorded in said recording means and at least one face image having a face likelihood calculated by said analysis means that is equal to or greater than a predetermined face likelihood and a face feature amount calculated by said analysis means that is equal to or greater than a predetermined face feature amount; a selection means for selecting at least one face image as a registration candidate image in order from the face image with the lowest similarity calculated by the calculation means; An information processing device comprising:

2. A limit is set on the number of registration images that can be recorded in the recording means based on the performance of the face authentication system, the calculation means, when the number of registered images exceeds the limit number when the face image is recorded in the recording means as a registered image, calculates a sum of the similarities for the face image acquired by the image acquisition means and the registered images already recorded in the recording means; the selection means selects the face image acquired by the image acquisition means as a registration candidate image and replaces the registration image with the highest sum of the similarities calculated; 2. The information processing apparatus according to claim 1,

3. the recording means records a face image of a person different from the person; the selection means selects, as a registration candidate image, a face image having the lowest similarity calculated by the calculation means, when all of the similarities between the face images of all persons other than the person are less than a predetermined similarity.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. the recording means records a facial image of a person different from the person and a result of past matching between the facial images, a display means for comparing the face image with the lowest similarity calculated by the calculation means with each face image of all persons other than the person, and when a face image determined by the calculation means to be of the same person as the face image with the lowest similarity calculated by the calculation means is present, displaying auxiliary information based on the face image and the face image with the lowest similarity calculated by the calculation means; 3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

5. The auxiliary information indicates at least one of the latest date and time when matching was performed for a face image determined to contain the same person as the face image with the lowest similarity calculated by the calculation means, the ratio of the face image matched with the face image to all face images recorded in the recording unit, the face image determined to contain the same person as the face image with the lowest similarity calculated by the calculation means, the date and time when matching was performed for each registered image, and the ratio of matching for each registered image.

5. The information processing apparatus according to claim 4.

6. the recording means records a facial image of a person different from the person and a result of past matching between the facial images, a display means for displaying, when the face image with the lowest similarity calculated by the calculation means is compared with each face image of all persons other than the person, and a face image of a person other than the face image with the lowest similarity calculated by the calculation means is photographed and the calculation means determines that the similarity with the person is equal to or higher than a predetermined similarity, said face image and a result of past comparison of said face image, 3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

7. the recording means records a facial image of a person different from the person and a result of past matching between the facial images, a display means for displaying a user interface for adjusting a threshold value so that the similarity between the face image of the person and the face image of the person other than the person is less than the predetermined similarity when the face image with the lowest similarity calculated by the calculation means is compared with each other and a face image of a person other than the person with the lowest similarity calculated by the calculation means is photographed and the similarity between the face image of the person and the face image of the person other than the person is determined by the calculation means to be equal to or greater than a predetermined similarity.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

8. Acquire a face image, Calculating a face likelihood and a face feature amount of the face image acquired by the image acquisition means; calculating a similarity by comparing features between at least one registered image used for face authentication and recorded in a recording means and at least one face image having a face likelihood calculated by said analysis means that is equal to or greater than a predetermined face likelihood and a face feature amount calculated by said analysis means that is equal to or greater than a predetermined face feature amount; selecting at least one face image as a registration candidate image in order from the face image with the lowest similarity calculated by the calculation means; A method comprising:

9. Acquire a face image, Calculating a face likelihood and a face feature amount of the face image acquired by the image acquisition means; calculating a similarity by comparing features between at least one registered image used for face authentication and recorded in a recording means and at least one face image having a face likelihood calculated by said analysis means that is equal to or greater than a predetermined face likelihood and a face feature amount calculated by said analysis means that is equal to or greater than a predetermined face feature amount; selecting at least one face image as a registration candidate image in order from the face image with the lowest similarity calculated by the calculation means; A program characterized by:

10. Acquire a face image, Calculating a face likelihood and a face feature amount of the face image acquired by the image acquisition means; calculating a similarity by comparing features between at least one registered image used for face authentication and recorded in a recording means and at least one face image having a face likelihood calculated by said analysis means that is equal to or greater than a predetermined face likelihood and a face feature amount calculated by said analysis means that is equal to or greater than a predetermined face feature amount; selecting at least one face image as a registration candidate image in order from the face image with the lowest similarity calculated by the calculation means; A storage medium for storing a program comprising:

Images