Information processing device, information processing system, information processing method, and information processing program

Abstract

To realize face authentication that enables a person to visually confirm a face image while protecting feature data extracted from the face image. [Solution] An information processing device performs a one-way transformation based on a predetermined algorithm on feature data extracted from a facial image to generate protection features indicating the protected features of the facial image and auxiliary information to be used for authentication using the protection features, generates a protection image for the facial image based on the facial image and the predetermined protection transformation, and registers the auxiliary information, the protection features, and the protection image in a database.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing system, an information processing method, and an information processing program. [Background technology]

[0002] Biometric Template Protection (BTP) technology, which authenticates biometric data such as facial images while protecting it through special encryption and one-way conversion, has been researched and developed, and efforts are underway to put it into practical use and standardize it.

[0003] Known standard architectures for BTP technology include PIE (Pseudonymous Identifier Encoder), which realizes the registration feature protection transformation function, PIR (Pseudonymous Identifier Recorder), which realizes the authentication feature protection transformation function, and PIC (Pseudonymous Identifier Comparator), which realizes the transformation feature matching function.

[0004] PIE generates AD (Auxiliary Data) and PI (Pseudonymous Identifier) ​​by converting feature data at the time of enrollment. The original feature data cannot be restored from either or both of the AD and PI. PIR generates PI*, which is the PI for authentication, from the feature data at the time of authentication and the AD generated by PIE. The original feature data cannot be restored from PI*. PIC compares PI and PI* and outputs "success (similar)" or "failure (dissimilar)". PIC can determine the similarity between the original feature data and the feature data at the time of authentication without restoring the original feature data.

[0005] A specific example of BTP technology is the Fuzzy Extractor (FE) described in Non-Patent Document 1. PIE in FE first transforms feature data at the time of enrollment and generates data (AD) for correcting errors that may occur during authentication. Examples of transformation methods include concealing feature data by additively applying random error-correcting codewords, or removing stable components useful for authentication from feature data and leaving only error components. Both of these are one-way transformations that are information-theoretically irreversible. PIE in FE further generates data (PI) by applying a one-way transformation, such as a hash function, to the feature data and AD, which is computationally difficult to reverse. Thus, PIE in FE is a one-way transformation that generates AD and PI from feature data, and has the property that it is difficult to restore the original feature data from either or both of AD and PI.

[0006] The PIR in FE performs error correction on the feature data used during authentication using AD to restore the feature data used at the time of registration, and generates PI* by performing one-way transformation such as a hash function on the restored feature data and AD. The PIC in FE compares PI and PI*, outputting a successful match if they match perfectly, or a failed match if they do not. Because PI and PI* are data that cannot be distinguished from random numbers, they can also be used as secure encryption keys.

[0007] By applying BTP technologies such as FE to facial recognition technology, even if the AD or PI registered in the system is leaked, an attacker who obtains it will not be able to restore the facial feature data, making it possible to realize a secure facial recognition system.

[0008] A technology with a different purpose from protecting feature data in user authentication and identity verification is described in Non-Patent Document 2. The technology described in Non-Patent Document 2 aims to prevent attacks that track user behavior and violate privacy by allowing attackers to search for facial photos uploaded to Internet services such as SNS (Social Networking Service) without the user's permission. The technology described in Non-Patent Document 2 uses machine learning to add subtle perturbations to facial images, thereby hindering automatic face matching by image search systems. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] Dodis Y., Reyzin L., Smith A. “Fuzzy Extractors: How to Generate Strong Keys from Biometrics and Other Noisy Data.” In: Cachin C., Camenisch JL (eds) Advances in Cryptology - EUROCRYPT 2004. EUROCRYPT 2004. Lecture Notes in Computer Science, vol 3027. Springer, Berlin, Heidelberg [Non-patent document 2] S.Shan, E.Wenger, J.Zhang,H.Li, H.Zheng and BYZhao, “Fawkes: Protecting privacy against unauthorized deep learning models,” Proc. the 29th USENIX Security Symposium, pp.1589-1604, Aug. 2020. Summary of the Invention [Problem to be solved by the invention]

[0010] Some use cases for facial recognition require a human to visually confirm the facial image in addition to the system's automatic matching process. In such use cases, the system must store the user's facial image at the time of registration and display it on a screen for visual confirmation at the time of authentication.

[0011] Therefore, even if feature data is protected using the technology described in Non-Patent Document 1, there is a risk that the facial image will be leaked when it is displayed on a screen for visual confirmation, and an attacker who obtains the leaked facial image can extract and misuse the feature data. Furthermore, the technology described in Non-Patent Document 2 is a technology intended to hinder the matching of a user's facial image, and therefore its purpose is fundamentally different from user authentication, which performs feature matching based on a facial image (assuming that feature matching may be successful).

[0012] Therefore, one aspect of the present invention realizes face authentication that allows a person to visually confirm a face image while protecting feature data extracted from the face image. [Means for solving the problem]

[0013] In order to solve the above problem, one aspect of the present invention employs the following configuration: An information processing device includes a processor and a storage device, and the storage device holds a database in which a registration facial image, registration feature data extracted from the registration facial image and indicating features of the registration facial image, protection features indicating protected features of the facial image, auxiliary information used for authentication using the protection features, and a protected image generated based on a predetermined protection transformation of the facial image are registered, and the processor performs a unidirectional transformation on the registration feature data based on a predetermined algorithm to generate registration protection features that are the protection features for the registration feature data and the auxiliary information for the registration feature data, generates a registration protected image that is the protection image for the registration facial image based on the registration facial image and the predetermined protection transformation, and registers the generated auxiliary information, the registration protection features, and the registration protected image in the database. [Effects of the Invention]

[0014] According to one aspect of the present invention, it is possible to realize face authentication that allows a person to visually confirm a face image while protecting feature data extracted from the face image.

[0015] Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a block diagram illustrating an example of the system configuration of a face authentication system according to a first embodiment. [Figure 2] 2 is a block diagram illustrating an example of the hardware configuration of a computer that constitutes each of a registration terminal, an authentication terminal, a visual confirmation terminal, and an authentication server in the first embodiment. FIG. [Figure 3] FIG. 10 is a sequence diagram illustrating an example of a registration process according to the first embodiment. [Figure 4] FIG. 10 is a sequence diagram illustrating an example of authentication processing in the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In this embodiment, the same components are generally designated by the same reference numerals, and repeated explanations will be omitted. It should be noted that this embodiment is merely an example for realizing the present invention, and does not limit the technical scope of the present invention. [Example]

[0018] This embodiment relates to a face authentication system that enables a person to visually confirm a face image while protecting feature data extracted from a face image from being leaked and misused.

[0019] The face authentication system of this embodiment not only performs a matching process of features generated based on a face image, but also displays the face image for human visual confirmation. A use case requiring both mechanical feature matching and human visual confirmation is, for example, when verifying the identity of an applicant when opening an online banking account, where a face photo data included in an identification card is mechanically matched with a photograph of the applicant's face image, and an operator is required to compare and visually confirm the two.

[0020] Another use case is when a store makes a payment using facial recognition, and the system displays the facial image of the registered user along with the facial recognition result on a terminal for store staff, who then visually confirms it by comparing it with the user's facial features.Another use case is when immigration control at airports and entrance / exit management at facilities use facial recognition, and when the system's facial recognition fails, an administrator visually confirms the face.

[0021] 1 is a block diagram showing an example of the system configuration of a face authentication system. The face authentication system includes, for example, a registration terminal 100, an authentication terminal 110, a visual confirmation terminal 120, and an authentication server 130, which are connected to one another via a network 150 such as the Internet. Note that any two or all three of the registration terminal 100, the authentication terminal 110, and the visual confirmation terminal 120 may be implemented in the same device (i.e., may be integrated). Note that if there is an integrated device, communication between the devices is omitted.

[0022] The registration terminal 100 acquires the user's ID information and facial image during user registration. The registration terminal 100 has, for example, an ID input function 101, a camera 102, a photographing function 103, and a communication function 104. The ID input function 101 accepts input of a user ID. The camera 102 photographs the user's facial image. The photographing function 103 controls the camera 102 to acquire the user's facial image. The communication function 104 controls communication with other devices connected to the registration terminal 100 via a network 150.

[0023] The authentication terminal 110 acquires the user's ID information and facial image during user authentication. The authentication terminal 110 has, for example, an ID input function 111, a camera 112, a photographing function 113, and a communication function 114. The ID input function 111 accepts input of a user ID. The camera 112 photographs the user's facial image. The photographing function 113 controls the camera 112 to acquire the user's facial image. The communication function 114 controls communication with other devices connected to the authentication terminal 110 via the network 150.

[0024] The visual confirmation terminal 120 displays a facial image of the user for visual identity confirmation by an operator. The visual confirmation terminal 120 has a communication function 121, an image display function 122, and a judgment result input function 123. The communication function 121 controls communication with other devices connected to the visual confirmation terminal 120 via a network 150. The image display function 122 displays the protected image on the screen. The judgment result input function 123 accepts input of the judgment result of identity confirmation by the operator.

[0025] The authentication server 130 performs processes such as registration and matching based on a facial image. The authentication server 130 has, for example, a communication function 131, a facial feature extraction function 132, a registration feature protection conversion function 133, a facial image protection conversion function 134, a protected image determination function 135, an encryption function 136, an authentication feature protection conversion function 137, a protection feature matching function 138, a final determination function 140, and a registration search function 141. The authentication server 130 also holds an auxiliary information DB (DataBase) 142, a protection feature DB 143, and an encrypted protected image DB 144.

[0026] The communication function 131 controls communication with other devices connected to the authentication server 130 via the network 150. The facial feature extraction function 132 extracts feature data from a facial image. The registered feature protection transformation function 133 is realized, for example, by a PIE (Pseudonymous Identifier Encoder). The registered feature protection transformation function 133 generates auxiliary information AD (Auxiliary Data) and a protected feature PI (Pseudonymous Identifier) ​​by performing a one-way transformation using PIE on the feature data at the time of registration. The auxiliary information AD is used, for example, for error correction of the feature data at the time of authentication. The protected feature PI indicates features protected by performing a computationally intensive one-way transformation such as a hash function on the feature data and AD of the facial image.

[0027] The facial image protection conversion function 134 converts the facial image to generate a protected image. The protected image determination function 135 verifies the security of the protected image. The encryption function 136 encrypts the protected image based on the protection feature PI to generate an encrypted protected image.

[0028] The authentication feature protection conversion function 137 is realized by, for example, a PIR (Pseudonymous Identifier Recorder). During authentication, the authentication feature protection conversion function 137 generates an authentication protection feature PI* by performing one-way conversion, such as a hash function, on the feature data and the auxiliary information AD.

[0029] The protection feature matching function 138 matches the protection feature PI with the authentication protection feature PI*. The protection feature matching function 138 is realized by, for example, a PIC (Pseudonymous Identifier Comparator). If the matching is successful, the decryption function 139 decrypts the encrypted protected image using the authentication protection feature PI*.

[0030] The final determination function 140 makes a final determination regarding authentication based on the matching result by the protection feature matching function 138 and the information input by the determination result input function 123. The registration search function 141 registers the auxiliary information AD, the protection feature PI, and the encrypted protected image in the auxiliary information DB 142, the protection feature DB 143, and the encrypted protected image DB 144, respectively. The registration search function 141 searches each DB.

[0031] 2 is a block diagram showing an example of the hardware configuration of a computer 400 constituting each of the registration terminal 100, the authentication terminal 110, the visual confirmation terminal 120, and the authentication server 130. The computer 400 has, for example, a CPU (Central Processing Unit) 401, a memory 402, an auxiliary storage device 403, an input device 404, a display device 405, and a communication device 406, which are connected to each other by an internal communication line such as a bus.

[0032] The CPU 401 is an example of a processor, and executes a program stored in the memory 402. Note that while the CPU 401 and a GPU (Graphics Processing Unit) are examples of processors, other semiconductor devices may be used as long as they are capable of executing predetermined processing.

[0033] The memory 402 includes a ROM (Read Only Memory), which is a nonvolatile storage element, and a RAM (Random Access Memory), which is a volatile storage element. The ROM stores unchanging programs (e.g., a BIOS (Basic Input / Output System)). The RAM is a high-speed, volatile storage element such as a DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by the CPU 401 and data used when the programs are executed.

[0034] The auxiliary storage device 403 is a large-capacity, non-volatile storage device such as a magnetic storage device (HDD (Hard Disk Drive)) or a flash memory (SSD (Solid State Drive)), and stores programs to be executed by the CPU 401 and data to be used when the programs are executed. That is, the programs are read from the auxiliary storage device 403, loaded into the memory 402, and executed by the CPU 401.

[0035] The input device 404 is a device that receives input from the user, such as a keyboard or mouse. The display device 405 is a device that outputs the results of program execution in a format that can be viewed by the user, such as a display or printer.

[0036] The communication device 406 is a network interface device that controls communication with other devices in accordance with a predetermined protocol. The communication device 406 may also include a serial interface such as a USB (Universal Serial Bus).

[0037] A part or all of the programs executed by the CPU 401 may be provided to the computer 400 from a removable medium (such as a CD-ROM or flash memory) which is a non-transitory storage medium, or from an external computer equipped with a non-transitory storage device via the network 150, and may be stored in the non-volatile auxiliary storage device 403 which is a non-transitory storage medium. For this reason, the computer 400 may have an interface for reading data from removable media.

[0038] The registration terminal 100 is a computer system configured on one physical computer 400 or on multiple logically or physically configured computers 400, and may operate in separate threads on the same computer 400, or may operate on a virtual computer constructed on multiple physical computer resources. The same applies to the authentication terminal 110, visual confirmation terminal 120, and authentication server 130.

[0039] 1 of the registration terminal 100. For example, the CPU 401 of the computer 400 constituting the registration terminal 100 operates in accordance with an ID input program loaded into the memory 402 of the computer 400 constituting the registration terminal 100 to realize the ID input function 101, operates in accordance with an image capture program loaded into the memory 402 to realize the image capture function 103, and operates in accordance with a communication program loaded into the memory 402 to realize the communication function 104. Similarly, the functions of the authentication terminal 110, the visual confirmation terminal 120, and the authentication server 130 are realized by the CPU 401 of the computer 400 constituting the device operating in accordance with a program loaded into the memory 402 of the computer 400 constituting the device.

[0040] In addition, some or all of the functions of the registration terminal 100, authentication terminal 110, visual confirmation terminal 120, and authentication server 130 may be realized by dedicated circuits such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

[0041] The auxiliary storage device 403 of the computer 400 constituting the authentication server 130 provides storage areas for realizing the auxiliary information DB 142, the protection feature DB 143, and the encrypted protected image DB 144. Some or all of the information stored in the auxiliary information DB 142, the protection feature DB 143, and the encrypted protected image DB 144 may be stored in the memory 402 of the computer 400 constituting the authentication server 130, or may be stored in an external database connected to the authentication server 130.

[0042] In this embodiment, the information used by the face authentication system does not depend on the data structure and may be represented in any data structure, for example, a table, a list, a database, or a queue may be appropriately selected to store the information.

[0043] 3 is a sequence diagram showing an example of registration processing. The registration terminal 100 starts the registration processing in response to a user's operation on the input device 404 of the computer 400 constituting the registration terminal 100, and the ID input function 101 acquires a user ID in accordance with the user's input on the input device 404 (S200). The photographing function 103 controls the camera 102 to acquire a facial image of the user for registration, and transmits the facial image and the user ID acquired in step S200 to the authentication server 130 (S201).

[0044] The authentication server 130 receives the facial image transmitted in step S201, and the facial feature extraction function 132 extracts feature data from the facial image using a predetermined feature extraction algorithm (S202). The registered feature protection conversion function 133 generates auxiliary information AD and protected features PI by performing one-way conversion based on PIE on the feature data (S203).

[0045] It should be noted that the PIE, PIR, and PIC used in this embodiment are the same as the PIE, PIR, and PIC in FE (Fuzzy Extractor). Therefore, the protection feature PI and auxiliary information AD in this embodiment are a hash value and auxiliary information in FE, respectively. Furthermore, the authentication protection feature PI* in this embodiment is a hash value obtained by applying a hash function to the feature information AD obtained by error-correcting the feature data of the facial image at the time of authentication using the auxiliary information AD. However, the PIE, PIR, and PIC are not limited to the PIE, PIR, and PIC in FE, and the PIE, PIR, and PIC in any BTP (Biometric Template Protection) algorithm can be applied.

[0046] The facial image protection conversion function 134 generates a protected image from the facial image by performing protection conversion on the facial image using a predetermined algorithm (S204). Here, the protection conversion performed by the facial image protection conversion function 134 has the following two properties.

[0047] The first property is feature dissimilarity, which is a property that the similarity between the feature data extracted from the original face image (the feature data extracted in step S202) and the protected image feature data, which is the feature data extracted from the protected image generated in step S204 by the face feature extraction function 132, is sufficiently low (dissimilar based on a predetermined condition (for example, the similarity is equal to or less than a predetermined value)).

[0048] Because the protection transformation has feature dissimilarity, even if the protected image is leaked due to hacking into the system or if a protected image displayed on the operator's screen is illegally photographed, it is not possible to generate an authentication protection feature PI* that will succeed in authentication from the leaked protected image or an illegally photographed protected image, thereby preventing spoofing attacks.

[0049] The second property is symmetric visual similarity, which states that if and only if any two facial images (I1, I2) have visual similarity (the property of being similar to or greater than a predetermined threshold in a facial image similarity measure that models human visual perception functions (e.g., SSIM (Structural Similarity))), then the two protected images (I1', I2') obtained by applying the protection transformation to (I1, I2) also have visual similarity.

[0050] The protection transformation may have asymmetric visual similarity instead of or in addition to symmetric visual similarity. Asymmetric visual similarity is a property that two protected images (I1', I2) also have visual similarity if and only if any two face images (I1, I2) have visual similarity (a property of being similar enough to be determined to be the same person by human visual perception).

[0051] Because the protection transformation has symmetric visual similarity, an operation can be realized in which the operator of the visual confirmation terminal 120 visually checks the protected image of the user both at the time of registration and at the time of authentication on the screen of the visual confirmation terminal 120 to confirm the user's identity without meeting the user face-to-face (hereinafter, this operation will also be referred to as "online identity verification").

[0052] Furthermore, since the protection transformation has asymmetric visual similarity, an operation can be realized in which the operator of the visual confirmation terminal 120 faces the user, visually views only the protected image at the time of registration on the screen of the visual confirmation terminal 120, and compares the protected image at the time of registration with the facial features of the user he is facing to confirm the user's identity (hereinafter, this operation will also be referred to as "face-to-face identity confirmation").

[0053] In this embodiment, to configure the facial image protection transformation with the above properties, the facial image protection transformation function 134 uses an image transformation with preserving visual similarity. Preserving visual similarity is a property in which images (I1, I1') before and after the facial image protection transformation have visual similarity in addition to feature dissimilarity.

[0054] A specific algorithm for image transformation with preserving visual similarity is disclosed in Non-Patent Document 2. When the face image protection transformation function 134 uses image transformation with preserving visual similarity, (I1, I1') and (I2, I2') each have visual similarity. Therefore, if and only if (I1, I2) have visual similarity, it is expected that (I1', I2') also have visual similarity (symmetric visual similarity), and (I1', I2) also have visual similarity (asymmetric visual similarity).

[0055] It should be noted that the face image preserving transformation with feature dissimilarity and symmetric visual similarity can be realized by a method other than the method using the image transformation with the preserving visual similarity described above.

[0056] Specifically, for example, the facial image protection transformation function 134 may use, as the facial image protection transformation, a transformation (morphing transformation) in which a user's facial image I1 (or I2) is input, the user's facial image I1 (or I2) is morphed to approximate a facial image J of another person (the facial image J is stored in advance in the authentication server 130, for example), and an intermediate image between I1 (or I2) and J is output as I1' (or I2').

[0057] Morphing transformation significantly changes the facial image itself, so neither features nor visibility are preserved. Therefore, morphing transformation does not have either preserving visual similarity or asymmetric visual similarity, but has symmetric visual similarity and feature dissimilarity.

[0058] Next, the protected image determination function 135 verifies the security of the protected image and determines whether the protection is successful or unsuccessful, and if the protection is unsuccessful, ends the registration process (without executing the processes from step S206 onwards) (S205).

[0059] A specific example of the protected image determination method in step S205 will be described. The facial feature extraction function 132 extracts feature data of the protected image from the protected image generated in step S204. The authentication feature protection conversion function 137 generates protection features PI' of the protected image from the feature data of the protected image and auxiliary information AD using PIR. The protection feature matching function 138 matches the protection features PI with the protection features PI' of the protected image using PIC. If the matching result is unsimilar (dissimilar), the protected image determination function 135 determines that "protection was successful," and if the matching result is similar, the protected image determination function 135 determines that "protection was unsuccessful."

[0060] As another method for determining a protected image in step S205, the protected image determination function 135 may evaluate the similarity between the feature data extracted in step S202 and the feature data of the protected image using a scale similar to that of a general face recognition algorithm (for example, a predetermined type of similarity such as cosine similarity), and if the similarity is equal to or less than a predetermined threshold, determine that the protection is successful, and if not, determine that the protection is unsuccessful.

[0061] When the method of Non-Patent Document 2 or morphing conversion is used as a method for protecting a facial image by the facial image protection conversion function 134, feature dissimilarity is not necessarily satisfied with 100% accuracy, and depending on the input facial image and the value of the random number used inside the facial image protection conversion, there is a probability that the output protected image will not satisfy feature dissimilarity. In this case, there is a risk that feature data will be leaked from the protected image. Therefore, by performing the protected image determination in step S205, feature data can be reliably protected.

[0062] Next, the encryption function 136 encrypts the protected image generated in step S204 using the protection feature PI generated in step S203 as an encryption key, thereby generating an encrypted protected image (S206). The encrypted protected image is correctly decrypted only if the matching process by PIC is successful (if PI=PI*) in the authentication process described below. The process of step S206 is not essential, but by further encrypting the protected image visible to humans, it is possible to protect the user's privacy even if the data stored in the authentication server 130 is leaked.

[0063] Next, the registration search function 141 links the auxiliary information AD with the user ID and registers it in the auxiliary information DB 142, links the protection feature PI with the user ID and registers it in the protection feature DB 143, and links the encrypted protected image with the user ID and registers it in the encrypted protected image DB 144 (S207).

[0064] 4 is a sequence diagram showing an example of authentication processing. The authentication terminal 110 starts the authentication processing in response to a user operation on the input device 404 of the computer 400 constituting the authentication terminal 110, and the ID input function 111 acquires a user ID according to the user input on the input device 404 (S300). Note that "1:N authentication" in which a user ID is not acquired can also be performed.

[0065] The photographing function 113 controls the camera 112 to acquire a face image for authentication of the user, and transmits the user ID acquired in step S300 and the face image for authentication to the authentication server 130 (S301). Note that when 1:N authentication is performed, the photographing function 113 transmits only the face image for authentication.

[0066] The authentication server 130 receives the authentication face image transmitted in step S301, and the facial feature extraction function 132 extracts authentication feature data from the authentication face image (S302). The registration search function 141 searches the auxiliary information DB 142 for auxiliary information AD corresponding to the user ID transmitted in step S301, and the authentication feature protection conversion function 137 generates authentication protection features PI* by performing unidirectional conversion based on PIR on the authentication feature data and the auxiliary information AD (S303).

[0067] The registration search function 141 searches the protection feature DB 143 for the protection feature PI corresponding to the user ID sent in step S301, and the protection feature matching function 138 matches the protection feature PI with the authentication protection feature PI* using the PIC, and determines whether the matching is successful or unsuccessful as the matching result (S304).

[0068] When performing 1:N authentication, the authentication server 130 repeatedly executes the processes of steps S303 and S304 for all registered user IDs, and generates a list (success list) of user IDs and protection features PI for which matching was successful.

[0069] The facial image protection conversion function 134 generates a protected image for authentication from the facial image for authentication using the same algorithm as in step S204 (S305). The protected image determination function 135 verifies the security of the protected image for authentication using the same method as described in step S205 (a method of comparing, by PIC, the protected features generated by PIE from the feature data for authentication with the protected features of the protected image for authentication generated by PIR from the feature data of the protected image for authentication and auxiliary information (generated by PIE from the feature data for authentication) or a method of evaluating the similarity between the feature data for authentication and the feature data of the protected image for authentication) and determines whether protection is successful or unsuccessful. If protection is unsuccessful, the authentication process is terminated (without executing the processes from step S307 onwards) (S306). Note that, in an operation where only face-to-face identity verification is performed and online identity verification is not performed, the processes of steps S305 and S306 may be omitted.

[0070] The registration search function 141 searches the encrypted protected image DB 144 for an encrypted protected image corresponding to the user ID transmitted in step S301, and the decryption function 139 performs a decryption process on the encrypted protected image using the authentication protection feature PI* as a decryption key, and transmits the authentication protected image and the protected image obtained as a result of the decryption process to the visual confirmation terminal 120 (S307).

[0071] When 1:N authentication is performed, in step S307, the authentication server 130 searches for and decrypts the encrypted protected image for each of all user IDs included in the success list described above, and transmits the authentication protected image and all protected images obtained as a result of the decryption process to the visual confirmation terminal 120.

[0072] If the process of step S206 is omitted, the decryption process by the decryption function 139 in step S307 is omitted.

[0073] Furthermore, if the processing of step S206 is executed and a matching result indicating a matching failure is obtained in the matching processing of step S304, the generation of the authentication protection feature PI* as the decryption key has failed, and therefore the decryption processing of step S307 also fails. Therefore, if the processing of step S206 is executed and a matching result indicating a matching failure is obtained in the matching processing of step S304, the authentication processing may end without executing the processing of step S305 and subsequent steps.

[0074] The visual confirmation terminal 120 receives the protected image and the protected image for authentication transmitted in step S307, and the image display function 122 displays the received protected image and the protected image for authentication on the display device 405 of the computer 400 constituting the visual confirmation terminal 120 (S308).

[0075] In the case of face-to-face identity verification, in step S308, the image display function 122 may display only the received protected image on the display device 405. In the case of 1:N authentication, the image display function 122 may display multiple protected images on the display device 405 (for example, in descending order of the matching score used for matching in step S304), or may not display any protected image on the display device 405 if the success list is empty.

[0076] In the case of online identity verification, an operator visually compares the protected image displayed on the display device 405 with the protected image for authentication, and determines whether or not the user whose face image was captured by the authentication terminal 110 is the registered user. On the other hand, in the case of face-to-face identity verification, an operator visually compares the protected image displayed on the display device 405 with the facial features of the user facing the user, and determines whether or not the user facing the user is the registered user.

[0077] In this case, even if an external attacker takes a photograph of the protected image and / or the protected image for authentication displayed on the display device 405 with a camera, or if an unauthorized operator (internal attacker) leaks the data, the attacker cannot extract feature data that enables successful authentication from the photographed image or the leaked data. Therefore, while enabling visual confirmation by the operator, a high level of security can be ensured against attacks that attempt to spoof by stealing a face image.

[0078] The judgment result input function 123 accepts input of the visual judgment result (judgment result indicating whether visual identity verification was successful or not) by the operator via the input device 404 of the computer 400 constituting the visual confirmation terminal 120, and transmits the judgment result to the authentication server 130 (S309).

[0079] The final judgment function 140 of the authentication server 130 generates a final judgment result based on the matching result in step S304 and the judgment result sent in step S309, and sends the final judgment result to the authentication terminal 110 and the visual confirmation terminal 120 (S310).

[0080] In step S310, if the matching result in step S304 is successful and the judgment result transmitted in step S309 is successful in identity verification, final judgment function 140 determines the final judgment result to be successful in authentication. On the other hand, if the matching result in step S304 is unsuccessful and the judgment result transmitted in step S309 is unsuccessful in identity verification, final judgment function 140 determines the final judgment result to be unsuccessful in authentication.

[0081] Furthermore, if only one of the following conditions is satisfied: the matching result in step S304 is a successful match; and the judgment result sent in step S309 is a successful identity verification, the final judgment function 140 may be configured to determine the final judgment result as a successful authentication, or may be configured to determine the final judgment result as a failed authentication, or may be configured to determine the final judgment result as a conditional successful authentication.

[0082] Note that, for example, the processing executed after the authentication process is completed differs between when the final determination result is authentication success and when it is conditional authentication success. When payment processing is performed based on the result of the authentication process, for example, if the final determination result is conditional authentication success, only payment processing of less than a predetermined amount can be executed, but if the final determination result is authentication success, payment processing of more than the predetermined amount can also be executed.

[0083] Next, the visual confirmation terminal 120 displays the final judgment result transmitted in step S310 on the display device 405 of the computer 400 that constitutes the visual confirmation terminal 120 (S311). The authentication terminal 110 displays the final judgment result transmitted in step S310 on the display device 405 of the computer 400 that constitutes the authentication terminal 110 (S312).

[0084] According to this embodiment, the authentication server 130 converts feature data extracted from a facial image in the registration process into auxiliary information AD and protection features PI using PIE, and then registers the converted data in the DB. The authentication server also performs a facial image protection transformation to create feature dissimilarity on the facial image, and then registers the converted data in the DB. This prevents spoofing attacks because even if all information is leaked from all DBs, it is impossible to generate authentication protection features PI* that will result in successful authentication from the leaked information.

[0085] Furthermore, according to this embodiment, since the facial image protection transformation has symmetric visual similarity (or asymmetric visual similarity), an operator can visually compare the registered protected image with the authentication protected image (or the user's own facial features) to confirm the identity of the user. As described above, according to this embodiment, a highly secure facial authentication and identity verification system can be realized that enables an operator to visually confirm the facial image while firmly protecting feature data for facial authentication from leakage and misuse for impersonation.

[0086] Furthermore, the above-described facial image protection transformation is a technology that aims to cause (machine) matching of feature data extracted from a facial image to fail, and is a technology with a different purpose from user authentication, which assumes that matching of feature data extracted from a facial image may be successful. Therefore, there are obstacles to applying the facial image protection transformation to BTP. On the other hand, in this embodiment, by applying the facial image protection transformation to BTP, it is possible to realize a facial authentication and identity verification system that achieves the above-described effects.

[0087] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0088] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0089] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0090] 100 registration terminal, 110 authentication terminal, 120 visual confirmation terminal, 130 authentication server, 132 facial feature extraction function, 133 registration feature protection conversion function, 134 facial image protection conversion function, 135 protected image judgment function, 136 encryption function, 137 authentication feature protection conversion function, 138 protection feature matching function, 140 final judgment function, 141 registration search function, 142 auxiliary information DB, 143 protection feature DB, 144 encrypted protected image DB, 400 computer, 401 CPU, 402 memory, 403 auxiliary storage device

Claims

1. An information processing device, a processor and a storage device; The storage device A facial image for registration; registration feature data indicating features of the registration face image extracted from the registration face image; a database in which protection features indicating protected features of a facial image, auxiliary information used for authentication using the protection features, and a protection image generated based on a predetermined protection transformation of the facial image are registered; The processor: performing a one-way transformation based on a predetermined algorithm on the feature data for registration to generate a protected feature for registration, which is the protected feature for the feature data for registration, and the auxiliary information for the feature data for registration; generating a registration protected image, which is the protected image for the registration face image, based on the registration face image and the predetermined protection transformation; an information processing device that registers the generated auxiliary information, the protection feature for registration, and the protection image for registration in the database;

2. 2. The information processing device according to claim 1, The processor: extracting protected image characteristic data indicating characteristics of the protected image from the protected image; generating a protected image protection feature based on the protected image feature data and the auxiliary information; The information processing device compares the protection feature for registration with the protection feature of the protected image, and if the comparison fails, registers the protected image for registration in the database.

3. 2. The information processing device according to claim 1, The processor: extracting protected image characteristic data indicating characteristics of the protected image from the protected image; calculating a similarity between the registration feature data and the protected image feature data; If the similarity is equal to or smaller than a predetermined threshold, the information processing device registers the protected image for registration in the database.

4. 2. The information processing device according to claim 1, The storage device A face image for authentication; authentication feature data indicating features of the authentication face image extracted from the authentication face image; The processor: generating an authentication protection feature based on the authentication feature data and the auxiliary information; performing protection feature matching to match the registration protection feature registered in the database with the authentication protection feature; outputting a protected image for registration registered in the database; An information processing device that outputs an authentication result based on the result of the protection feature matching.

5. 5. The information processing device according to claim 4, The processor: generating an authentication protected image, which is the protected image for the authentication face image, based on the authentication face image and the predetermined protection transformation; an information processing device that outputs the authentication protected image;

6. 6. The information processing device according to claim 5, The processor: extracting, from the protected image for authentication, protected image characteristic data indicating characteristics of the protected image for authentication; performing the unidirectional transformation based on the predetermined algorithm on the authentication feature data to generate the protection feature for the authentication feature data and the auxiliary information for the authentication feature data; generating a protected image feature for authentication based on the protected image feature data for authentication and the auxiliary information for the feature data for authentication; an information processing device that compares the protection feature for the authentication feature data with the protection feature for the authentication protected image, and outputs the authentication protected image if the comparison fails;

7. 6. The information processing device according to claim 5, The processor: extracting, from the protected image for authentication, protected image characteristic data indicating characteristics of the protected image for authentication; calculating a similarity between the authentication feature data and the authentication protected image feature data; When the degree of similarity is equal to or less than a predetermined threshold, the information processing device outputs the protected image for authentication.

8. 5. The information processing device according to claim 4, The processor: obtaining a result of the identity determination made by an operator through visual confirmation of the output protected image for registration; an information processing device that outputs the authentication result based on the determination result and the result of the protection feature matching;

9. 5. The information processing device according to claim 4, The processor: generating an image by performing the predetermined protection transformation on the face image for registration, and encrypting the generated image using the protection feature for registration as an encryption key, thereby generating the protected image for registration; decrypting the protected image for registration registered in the database using the protected feature for authentication as a decryption key; an information processing device that outputs the decoded registration image;

10. 2. The information processing device according to claim 1, the predetermined protection transformation has feature dissimilarity and symmetric visual similarity; The feature dissimilarity is a property indicating that the facial images before and after the predetermined protection transformation are not similar based on a predetermined condition, The symmetric visual similarity is a property indicating that a combination of the protected images for the facial images included in the combination has the visual similarity only when the combination of facial images before the specified protection transformation is applied has the visual similarity.

11. 2. The information processing device according to claim 1, the predetermined protection transformation has feature dissimilarity and asymmetric visual similarity; The feature dissimilarity is a property indicating that the facial images before and after the predetermined protection transformation are not similar based on a predetermined condition, The asymmetric visual similarity is a property indicating that one facial image included in a combination of facial images before the specified protection transformation has visual similarity with the protected image for the other facial image included in the combination, only when the combination has visual similarity.

12. 2. The information processing device according to claim 1, the predetermined protection transformation comprises feature dissimilarity and visual similarity preserving; The feature dissimilarity is a property indicating that the facial images before and after the predetermined protection transformation are not similar based on a predetermined condition, The preserved visual similarity is a property indicating that, only when a combination of face images before the specified protection transformation has visual similarity, one face image included in the combination and the protection image for that face image have the visual similarity, and also the other face image included in the combination and the protection image for the other face image have the visual similarity.

13. an information processing device according to any one of claims 1 to 12; a registration terminal that captures the face image for registration and transmits it to the information processing device; a visual confirmation terminal that receives the registration protected image registered in the database from the information processing device and displays it.

14. An information processing method by an information processing device, the information processing device includes a processor and a storage device; The storage device A facial image for registration; registration feature data indicating features of the registration face image extracted from the registration face image; a database in which protection features indicating protected features of a facial image, auxiliary information used for authentication using the protection features, and a protection image generated based on a predetermined protection transformation of the facial image are registered; The information processing method includes: the processor performs a one-way transformation on the registration feature data based on a predetermined algorithm to generate a registration protection feature that is the protection feature for the registration feature data and the auxiliary information for the registration feature data; the processor generates a registration protected image, which is the protected image for the registration face image, based on the registration face image and the predetermined protection transformation; The information processing method, wherein the processor registers the generated auxiliary information, the registration protection feature, and the registration protection image in the database.

15. An information processing program that causes the information processing device to execute the information processing method according to claim 14.

Images