Biometric authentication apparatus, system, method, medium, and computer program product

Abstract

The disclosure relates to biometric authentication devices, systems, methods, media, and computer program products. The biometric authentication apparatus includes: at least one processor; and at least one memory having instructions stored thereon, the instructions, when executed by the at least one processor, causing the biometric authentication device to at least: acquire consent conditions regarding acquisition and use of biometric information of a person; and performing biometric authentication on the person based on the agreement condition.

Description

Biometric authentication devices, systems, methods, media, and computer program products Technical Field

[0001] This disclosure relates to biometric authentication devices, biometric authentication systems, methods, non-transitory computer-readable storage media, and computer program products. Background Technology

[0002] In recent years, the importance of personal information protection has been increasing. Facial authentication systems impose strict restrictions on the acquisition and use of large amounts of facial images and facial features.

[0003] Here, facial feature quantities are obtained by converting the external pattern of the face into a multi-dimensional vector.

[0004] Because facial features can be used to identify individuals, they are considered a type of biometric identifier. For example, the state of Illinois in the United States requires written consent from individuals when obtaining facial features. The European Union is also considering laws banning the use of facial authentication in public places. Due to national and state legislation, the acquisition and use of facial features may be subject to greater restrictions than the acquisition and use of facial images.

[0005] Therefore, methods have been proposed to restrict the acquisition and use of facial images or facial feature data. Japanese Patent No. 6150019 discloses a system in which personal information acquired for a person performing a specific gesture as a subject for facial authentication is deleted. Furthermore, Japanese Patent No. 7126138 discloses a facial authentication system in which prior consent is obtained from the person serving as the subject for facial authentication regarding the acquisition and use of personal information such as facial images.

[0006] Here, the acquisition and use of large amounts of facial images and facial feature data may be subject to legal restrictions in terms of application and / or location. It is important to note that if the person being photographed individually consents to the use of their personal information under such legal restrictions, then it is necessary to make the personal information of the consenting person available. In this way, regarding the acquisition and use of facial images and facial feature data, it is necessary to be able to restrict the use of large amounts of personal information while also being able to adjust the restrictions on the use of personal information on a per-individual basis.

[0007] However, in Japanese Patent No. 6150019, if a person does not perform a gesture to express their intention to prohibit the acquisition and use of their facial image or facial feature data, then even if the person does not consent to the acquisition and use of their personal information, a third party can still acquire and use the person's facial image or facial feature data. Furthermore, in Japanese Patent No. 7126138, regarding the acquisition and use of each individual's facial image or facial feature data, the application and usage terms associated with the facial image or facial feature data cannot be changed for each individual. Summary of the Invention

[0008] In view of this, this disclosure provides a technique for the secure and easy use of biometric information specific to an individual.

[0009] This disclosure provides a biometric authentication device in its first aspect, comprising: at least one processor; and at least one memory having instructions stored thereon, the instructions, when executed by the at least one processor, causing the biometric authentication device to at least: acquire consent conditions for the acquisition and use of biometric information about a person; and perform biometric authentication on the person based on the consent conditions.

[0010] This disclosure provides a biometric authentication system in a second aspect, comprising: a mobile terminal device including a consent condition registration unit for registering consent conditions for the acquisition and use of biometric information about a person; a server device including a consent condition management unit for managing the consent conditions registered by the consent condition registration unit; and a biometric authentication device, wherein the biometric authentication device includes: an acquisition unit for acquiring consent conditions from the server device, and a biometric authentication unit for performing biometric authentication on the person based on the consent conditions.

[0011] This disclosure provides a biometric authentication system in a third aspect, comprising: an instant consent registration device, the instant consent registration device including another consent condition registration unit for registering consent conditions for the acquisition and use of biometric information about a person; a server device including a consent condition management unit for managing the consent conditions registered by the other consent condition registration unit; and a biometric authentication device, wherein the biometric authentication device includes: an acquisition unit for acquiring consent conditions from the server device, and a biometric authentication unit for performing biometric authentication on the person based on the consent conditions.

[0012] This disclosure provides, in its fourth aspect, a method performed by a biometric authentication device, comprising: obtaining consent conditions regarding consent to the acquisition and use of a person's biometric information; and performing biometric authentication on the person based on the consent conditions.

[0013] This disclosure provides, in its fifth aspect, a non-transitory computer-readable storage medium storing a computer program, which, when read and executed by a computer, causes the computer to perform the method according to the fourth aspect.

[0014] This disclosure provides, in its sixth aspect, a method performed by a biometric authentication system, comprising: registering consent conditions for the acquisition and use of biometric information about a person; managing the consent conditions registered at the time of registration; obtaining consent conditions; and performing biometric authentication on the person based on the consent conditions.

[0015] This disclosure provides, in its seventh aspect, a non-transitory computer-readable storage medium storing a computer program, which, when read and executed by a computer, causes the computer to perform the method according to the sixth aspect.

[0016] This disclosure provides, in its eighth aspect, a computer program product comprising a computer program that, when read and executed by a computer, causes the computer to perform the method according to the fourth aspect.

[0017] This disclosure provides, in its ninth aspect, a computer program product comprising a computer program that, when read and executed by a computer, causes the computer to perform the method according to the sixth aspect.

[0018] The features of this disclosure will become clear from the following description of embodiments with reference to the accompanying drawings. The following description of the embodiments is by way of example. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the present disclosure and, together with this description, serve to explain the principles of the embodiments.

[0020] Figure 1 is a diagram illustrating an example of the hardware configuration of an information processing apparatus according to the first and second embodiments.

[0021] Figure 2 is a diagram illustrating an overview of the configuration of the system according to the first embodiment.

[0022] Figure 3A is a diagram illustrating the functional configuration of the access / exit management device according to the first embodiment.

[0023] Figure 3B is a diagram illustrating the functional configuration of the monitoring system according to the first embodiment.

[0024] Figure 4 is an example of a UI screen showing a person setting consent conditions according to the first embodiment.

[0025] Figure 5A is a diagram illustrating data 501 managed by a server device according to the first embodiment.

[0026] Figure 5B is a diagram illustrating data 502 stored in the access / exit management device according to the first embodiment.

[0027] Figure 5C is a diagram illustrating data 503 obtained by combining data 501 and data 502 according to the first embodiment.

[0028] Figure 6A is a flowchart illustrating the access / exit authorization setting process performed by the access / exit management device according to the first embodiment.

[0029] Figure 6B is a flowchart illustrating the process performed by the access / exit management device according to the first embodiment for acquiring registered facial feature data.

[0030] Figure 6C is a flowchart illustrating the entry / exit gate control process performed by the entry / exit management device according to the first embodiment.

[0031] Figure 7 is a diagram illustrating an overview of the system configuration according to the second embodiment.

[0032] Figure 8 is a diagram illustrating the functional configuration of the instant consent registration device according to the second embodiment.

[0033] Figure 9 is a flowchart illustrating the process performed by the instant consent registration device according to the second embodiment. Detailed Implementation

[0034] In the following, embodiments will be described in detail with reference to the accompanying drawings. It should be noted that the following embodiments are not intended to limit the scope of the claims. Several features are described in the embodiments, but not all such features are necessary, and multiple such features can be appropriately combined. Furthermore, in the drawings, the same reference numerals are given to the same or similar configurations, and repeated descriptions thereof are omitted.

[0035] In the first embodiment, detailed consent conditions (application, terms of use) regarding the acquisition and use of a person's facial image or facial feature data can be set for each individual, and consent conditions can be obtained from the individual. In this specification, a person's facial image and facial feature data are collectively referred to as biometric information. Furthermore, in the first embodiment, facial images and facial feature data can be acquired and used based on the consent conditions obtained for the corresponding individual. As a result, the facial authentication system acquires and uses the facial image or facial feature data of consenting individuals, without unintentionally acquiring or using the facial image or facial feature data of unconsenting individuals. As a result, personal privacy can be protected.

[0036] Figure 1 is a diagram illustrating an example of the hardware configuration of an information processing apparatus according to the first and second embodiments.

[0037] Information processing device 101 is connected to input device 102, output device 103, and network 104. In each embodiment, one or more information processing devices 101 are used. Note that information processing device 101 is used as mobile terminal device 201, server device 202, access / exit management device 203A, monitoring device 203B, and instant consent registration device 701, which will be described later.

[0038] The information processing device 101 includes a central processing unit (CPU) 101a, random access memory (RAM) 101b, read-only memory (ROM) 101c, external storage device 101d, input I / F 101e, output I / F 101f, and communication I / F 101g. The components of the information processing device 101 are interconnected with each other in a communication-enabled manner via a system bus 101h.

[0039] CPU 101a performs overall control of information processing device 101.

[0040] RAM 101b temporarily stores data received from external storage device 101d, and / or data received from external devices (not shown) via input I / F 101e and communication I / F 101g. RAM 101b serves as the main memory and the working area of ​​CPU 101a.

[0041] ROM 101c stores, for example, a control program executed by CPU 101a.

[0042] External storage device 101d is a storage device such as a hard disk and / or memory card that is fixedly provided in information processing device 101. Note that external storage device 101d may include storage devices that can be attached to and detached from information processing device 101, examples of which include floppy disks (FD), optical disks (such as optical discs (CD)), magnetic cards or optical cards, IC cards, and memory cards.

[0043] Input I / F 101e is the interface between information processing device 101 and input device 102.

[0044] Output I / F 101f is the interface between information processing device 101 and output device 103.

[0045] The communication I / F 101g is the interface between the information processing device 101 and an external device (not shown) connected to the network 104.

[0046] Input device 102 accepts user input. Input device 102 is, for example, a pointing device or keyboard, through which the user inputs data.

[0047] Output device 103 is a display used to display data held by information processing device 101 and program processing results. The display is, for example, a liquid crystal display (LCD) or an organic electroluminescent display (EL).

[0048] Network 104 is a communication device through which information processing apparatus 101 communicates with external devices (not shown). Information processing apparatus 101 can communicate with, for example, a network camera used to capture images of a subject, a database from which data is obtained, and an external server that queries its services.

[0049] It should be noted that the hardware configuration of the information processing device 101 is not limited to the above configuration, and can be any desired configuration.

[0050] First Embodiment

[0051] In a first embodiment, a system (more specifically, a biometric authentication system) including a surveillance device 203B and an access control device 203A using facial authentication will be described. This system is, for example, a facial authentication system, such as an access control system for corporate employees or a missing child search system used in large commercial facilities. In the facial authentication system according to the first embodiment, the user of the facial authentication system (i.e., the person being photographed for facial authentication) can pre-set consent information regarding the application and usage terms of biometric information (facial images and / or facial feature quantities) in various situations. As mentioned above, biometric information includes at least a facial image or facial feature quantities of a person. In one embodiment, biometric authentication is facial authentication. Note that when the subject is a child (e.g., a minor), the child's guardian sets consent information regarding the child. Accordingly, the user of the facial authentication system can perform settings such that the acquisition and use of facial images and / or facial feature quantities are only permitted when necessary.

[0052] When the system of the first embodiment is used as an employee access control system for an enterprise, advantages such as the following can be obtained: When there are access control devices for multiple affiliated companies, employees do not need to repeatedly go through the consent process for facial recognition. Moreover, by expanding the scope of employee consent for facial recognition outside of work, employees do not need to go through the facial recognition consent process when entering or leaving another company.

[0053] Furthermore, when the system of the first embodiment is used as a missing child search system, and multiple facial recognition devices are available within the facility (e.g., a commercial facility), the guardian of the missing child does not need to repeatedly perform the facial recognition consent process for the child. Moreover, by expanding the scope of child facial recognition consent to other facilities (e.g., other commercial facilities), it is possible to eliminate the need for the guardian of the missing child to perform the child facial recognition consent process for other facilities located in other places.

[0054] Here, in the first embodiment, when a facial image or facial feature quantity is acquired or used, the application and usage terms permitted by the person being photographed as a facial authentication subject will be referred to as "consent conditions." Consent conditions include conditions under which the person allows the acquisition or use of the person's biometric information. Regarding the acquisition / use of personal information (biometric information), consent conditions include the purpose of use, the period of use, the storage period and the validity period of consent, the type of biometric information (e.g., facial image or facial feature quantity), the processor, the administrator, the responsible department, and the method of acquisition (e.g., image acquisition method). Consent conditions include at least the consent conditions pre-registered by the person using the UI shown in Figure 4, or consent conditions predicted based on information about non-identifiable individuals obtained from the person's biometric information (e.g., an image of the person), or both. Furthermore, in addition to the information listed above, consent conditions may also include detailed conditions allowing the use of the person's personal information, such as country, location, and time zone. Consent conditions may include any settings desired by the system's users, and various items may be subject to consent conditions. Moreover, the system administrator may store other necessary information as metadata when storing consent conditions. For example, when obtaining consent to use a person's personal information in accordance with the laws or regulations of a region or organization, it may be necessary to store items such as the processor, manager, and responsible department of the personal information along with the consent conditions. The stored metadata is not limited to the forms of metadata mentioned above, and may include various types of data as long as necessary and legally appropriate.

[0055] (System Configuration)

[0056] Figure 2 is a diagram illustrating an overview of the configuration of the system according to the first embodiment.

[0057] System 20 includes a mobile terminal device 201, a server device 202, an access / exit management device 203A, and a monitoring device 203B. System 20 is a biometric authentication system that performs biometric authentication on persons, and is, for example, a facial authentication system. As described with reference to FIG1, information processing device 101 is used as mobile terminal device 201, server device 202, access / exit management device 203A, and monitoring device 203B. Therefore, mobile terminal device 201, server device 202, access / exit management device 203A, and monitoring device 203B each have a configuration similar to that of information processing device 101. Access / exit management device 203A and monitoring device 203B are examples of biometric authentication devices. In one embodiment, the biometric authentication device is used for access management or monitoring.

[0058] Mobile terminal device 201 is used by a person acting as a subject for facial authentication to set (input) consent conditions, whether to consent, related facial images, etc., regarding the acquisition and use of facial images or facial feature data. System 20 includes one or more mobile terminal devices 201. The input devices of mobile terminal device 201 (input device 102 shown in FIG. 1) include image capture devices, key input devices, and pointing devices. The output device of mobile terminal device 201 (output device 103 shown in FIG. 1) is a display device. Mobile terminal device 201 can be an individually owned smartphone or PC, but is not limited to these. Note that mobile terminal device 201 can be any device capable of registering consent conditions, which will be described later. The detailed method for registering consent conditions will be described later with reference to FIG. 4.

[0059] Server device 202 manages the consent conditions of individuals set using mobile terminal device 201. The input device of server device 202 (input device 102 in Figure 1) is a keyboard. The output device of server device 202 (output device 103 in Figure 1) is a display device. Server device 202 consists of one or more information processing devices 101. Server device 202 can be a standalone PC. Server device 202 can be a cloud server connected to multiple PCs via a network. The management of consent conditions for individuals will be described below using Figures 5A to 5C.

[0060] The access control device 203A controls the opening and closing of access gates via facial recognition. The access control device 203A manages the entry and exit of company employees. The input devices of the access control device 203A (input device 102 in Figure 1) include a keyboard, mouse, and image capture device. The output devices of the access control device 203A (output device 103 in Figure 1) include a display device and the access gate device.

[0061] Surveillance device 203B monitors specific individuals designated through facial recognition. For example, it may be assumed that surveillance device 203B is used to search for lost children in a large commercial facility. The input devices of surveillance device 203B (input device 102 in FIG. 1) include a keyboard, mouse, and image capture device. The output devices of surveillance device 203B (output device 103 in FIG. 1) include a display device.

[0062] Here, the access control devices 203A(multiple) and monitoring devices 203B(multiple) are devices used to illustrate various services using the facial recognition system according to the first embodiment. It should be noted that the first embodiment is applicable to systems other than access control and monitoring systems, such as electronic payment systems using facial recognition, and recommendation systems for detecting frequent customers via facial recognition and providing services tailored to the preferences of frequent customers. When the first embodiment is applied to such systems, the consent management mechanism described in the first embodiment can be used.

[0063] (Function Configuration)

[0064] Figure 3A is a diagram illustrating the functional configuration of the access / exit management device according to the first embodiment. Figure 3B is a diagram illustrating the functional configuration of the monitoring system according to the first embodiment. Reference numerals 201, 202, 203A, and 203B in Figures 3A and 3B correspond to 201, 202, 203A, and 203B in Figure 2. Furthermore, in Figures 3A and 3B, functional blocks labeled with the same reference numerals have the same function.

[0065] Access control device 203A manages the entry and exit of employees registered in system 20 who have agreed to access control via facial recognition. The functional configurations of the consent condition registration unit 301 of mobile terminal device 201 and the consent management unit 302 of server device 202 will be described below. The consent condition registration unit 301 is a consent condition registration unit used to register consent conditions for the acquisition and use of a person's biometric information. The consent management unit 302 is a consent condition management unit used to manage the consent conditions registered by the consent condition registration unit 301.

[0066] The consent registration unit 301 of the mobile terminal device 201 obtains consent conditions and a facial image from a person being photographed as a subject for facial authentication. The consent registration unit 301 registers the obtained consent conditions and facial image in the consent management unit 302 of the server device. The method for obtaining and registering consent conditions and facial images will be described later with reference to FIG4.

[0067] The consent management unit 302 of server device 202 manages the consent conditions and facial images registered by the consent condition registration unit 301. For example, the consent management unit 302 uses the data structure shown in FIG. 5A to manage the data. Here, the consent management unit 302 assigns a unique person ID to each individual (person) and manages the consent conditions and facial images in association with the person ID. Therefore, various types of data associated with the person ID can be appropriately referenced. The person being photographed as the subject of facial authentication can check their own person ID via the consent condition registration unit 301.

[0068] The following block diagram will be used to describe the functional configuration of the access / exit management device 203A.

[0069] The consent condition query unit 303 queries the consent management unit 302 of the server device 202 for consent conditions and facial images of an individual (employee). The consent condition query unit 303 is a unit for obtaining consent conditions for the acquisition and use of a person's biometric information. Here, the consent condition query unit 303 obtains a person ID (a string or number that can uniquely identify a person) from the database 306 via the recording unit 305. The process for storing the person ID in the database 306 will be described later in the description of the system setup unit 304. The consent condition query unit 303 obtains consent conditions for the acquisition and use of a person's biometric information from the server device 202, which communicates with the access / exit management device 203A (biometric authentication device). The consent condition query unit 303 transmits the person ID to the consent management unit 302 and obtains the consent conditions and facial image corresponding to the transmitted person ID from the consent management unit 302. The consent condition query unit 303 stores the obtained consent conditions and facial image in the database 306 via the recording unit 305.

[0070] The system setting unit 304 obtains the person IDs of all permitted individuals and a list of turnstiles that can be used by all permitted individuals, and stores the obtained information in the database 306 via the recording unit 305. Here, the administrator of system 20 queries the person IDs of permitted individuals to determine which turnstiles each individual can use, and inputs the person IDs and the turnstiles they can use into the system setting unit 304.

[0071] Recording unit 305 controls the storage, updating, and deletion of data in database 306.

[0072] Database 306 is a database that stores data using the data structures shown in Figures 5A to 5C.

[0073] The consent condition determination unit 307 is used to determine whether to allow the acquisition and use of a person's biometric information based on consent conditions. The consent condition determination unit 307 determines whether to allow the acquisition and use of a person's biometric information based on whether there are contradictory items in the consent conditions. For each person, the consent condition determination unit 307 determines the consent conditions and determines whether facial authentication should be performed. The consent condition determination unit 307 checks whether there are contradictory items in the consent conditions. More specifically, the consent condition determination unit 307 checks whether the usage / storage period or the validity period of the consent has expired. The consent condition determination unit 307 also checks whether the purpose of use and the facial image acquisition method are consistent. The consent condition determination unit 307 also checks whether the person has already been allowed to use facial feature data. As described at the beginning of the first embodiment, the consent conditions include various items. Therefore, if the consent conditions include items other than those mentioned above, then the consent condition determination unit 307 checks such items.

[0074] In the first embodiment, it is assumed that facial authentication is performed on individuals for whom there are no contradictory items in the consent conditions. Furthermore, the consent condition determination unit 307 transmits the person ID and facial image of the individual to be subject to facial authentication to the facial authentication unit 308.

[0075] The facial authentication unit 308 is a biometric authentication unit used to perform biometric authentication on a person based on consent conditions. If the consent condition determination unit 307 determines that it is permissible to obtain and use the person's biometric information, then the facial authentication unit 308 performs biometric authentication on that person. The facial authentication unit 308 identifies the person corresponding to the face detected by the facial detection unit 309. Specifically, the facial authentication unit 308 converts the facial images registered in the recording unit 305 and the facial images detected by the facial detection unit 309 into facial feature values. The facial authentication unit 308 assigns a person ID by comparing the facial feature values ​​of the facial images with the facial feature values ​​of the detected facial images.

[0076] When the facial authentication unit 308 converts a facial image into facial feature quantities, it uses a neural network that transforms the external pattern of the face into a multidimensional vector (feature quantity). This neural network is pre-trained to convert a pair of facial images of the same person into feature quantities that are close to each other in the feature space, and a pair of facial images of different people into feature quantities that are far apart from each other in the feature space. The neural network described above is merely an example, and other methods can be used to assign person IDs. Examples of other techniques include dimensionality reduction techniques called Principal Component Analysis (PCA) and clustering techniques called k-means. The other methods described above are merely examples, and it is possible to use any method capable of extracting a multidimensional vector (feature quantity) that distinguishes a pair of facial images of the same person from a pair of facial images of different people.

[0077] When comparing facial features, cosine similarity is calculated between the features. If a pair of features has a cosine similarity exceeding a preset threshold, then those features are considered to have been extracted from the same person. The comparison of features is not limited to the methods described above. For example, two people with the highest cosine similarity exceeding the threshold can be considered the same person. Furthermore, Euclidean distance or Manhattan distance can be used, for example. The comparison methods are not limited to these examples, and it is possible to use any method that can quantitatively calculate the distance between two features.

[0078] The face detection unit 309 detects facial regions in the image acquired by the image capture unit 310. This detection is performed using a face detection neural network called Retinaface (Non-Patent Document 1: Deng, Jiankang et al., “Retinaface: Single-shot multi-level face localization in the wild”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2020). The invention is not limited thereto, and it is possible to use any method capable of detecting facial regions.

[0079] Image capture unit 310 is an image capture device that acquires an image of a person as a subject for facial authentication. Here, a surveillance camera is used as image capture unit 310, but the invention is not limited thereto. Image capture unit 310 is not limited thereto, and can be any device capable of capturing or moving images and acquiring image data.

[0080] The gate control unit 311 controls the opening and closing of the access gate 312 based on the authentication result from the facial authentication unit 308. When the facial authentication unit 308 successfully assigns a person ID, the gate control unit 311 transmits an open gate command to the access gate 312. On the other hand, if the facial authentication unit 308 fails to assign a person ID, the gate control unit 311 does not transmit an open gate command to the access gate 312.

[0081] If the facial recognition unit 308 fails to assign a person ID, the gate control unit 311 performs error handling to notify the user of the failed person ID assignment using audio and visual display. The control method is not limited to this type of method, and it is possible to use any method that can open the access gate 312 for authorized persons and prevent it from opening for unauthorized persons.

[0082] If a command to open the entrance / exit gate 312 is received from the gate control unit 311, then the entrance / exit gate 312 will open; otherwise, if no command to open the entrance / exit gate 312 is received, then the entrance / exit gate 312 will not open.

[0083] The following is a description of the surveillance device 203B shown in Figure 3B. Surveillance device 203B is used to search for lost children within a large commercial facility.

[0084] In Figures 3A and 3B, function blocks with the same names and symbols have the same function. Note that the system setting unit 304 uses a person ID to specify the person to be monitored on the settings screen. For example, in a use case corresponding to the search for a missing child, the guardian (parents) of the missing child notifies the administrator of the monitoring device 203B of the child's person ID. The administrator then sets the child's person ID via the system setting unit 304.

[0085] Here, similar to the case of the access / exit management device 203A, the monitoring device 203B transmits the person IDs of all persons to be monitored from the consent condition query unit 303 to the consent management unit 302, and obtains the consent conditions and facial images. Furthermore, the facial authentication unit 308 converts the obtained facial images into facial feature values. The facial authentication unit 308 compares the facial feature values ​​obtained from the consent management unit 302 with the facial feature values ​​obtained from the facial image acquired by the facial detection unit 309 for verification, and identifies the person ID.

[0086] Image capture unit 313 is an image capture device that acquires images of people as subjects for facial authentication. The example in Figure 3B differs from the example in Figure 3A in that multiple image capture units 313 are installed at the locations to be monitored within the facility. Here, the image capture unit 313 in Figure 3B acquires moving images, reduces the frame rate of the moving images to the minimum necessary for searching for lost children, and transmits the resulting images to face detection unit 309. For example, if there are ten image capture devices capturing moving images at 30fps, then not all 300 images per second are transmitted to face detection unit 309. For each image capture device, one image is extracted from every ten images, and 30 images are transmitted to face detection unit 309 per second. However, the image transmission method is not limited to this method. Designers of facial authentication systems can determine the image transmission method based on the processing performance of image capture unit 313 and the search speed required by the facial authentication system.

[0087] Note that each image capture unit 313 in Figure 3B assigns its identifier and installation location as metadata to the image to be transmitted to the face detection unit 309, and transmits the image and metadata together to the face detection unit 309.

[0088] The person monitoring unit 314 acquires the authentication result from the face authentication unit 308 and the metadata from the image capture unit 313. More specifically, the person monitoring unit 314 transmits the identification number and installation location of the image capture unit 313, which has captured an image of the subject to be monitored, to the notification unit 315.

[0089] The notification unit 315 notifies the system administrator via a display of the identification number and installation location of the image capture unit 313, which has captured an image of a monitored target matching the search target. The notification method for the identification number and installation location of the image capture unit 313 is not limited to this. For example, the notification unit 315 may directly notify the guardian of the monitored target of the identification number and installation location of the image capture unit 313 via email, or it may publish the identification number and installation location of the image capture unit 313 via a web server.

[0090] (UI: User Interface)

[0091] Figure 4 is an example of a UI screen in which a person sets consent conditions according to the first embodiment.

[0092] The consent registration unit 301 of the mobile terminal device 201 displays a user interface (UI) through which a person being photographed for facial authentication can set (input) consent conditions. The consent registration unit 301 of the mobile terminal device 201 presents consent conditions including various facial authentication services to the person being photographed for facial authentication, as shown in items 401 to 405 of FIG4.

[0093] The consent condition registration unit 301 registers the consent conditions in the consent management unit 302 based on the results set by the person in the consent condition setting UI.

[0094] Furthermore, although the access control device 203A and the surveillance device 203B are illustrated as examples in the first embodiment, the present invention is not limited to these examples. According to this disclosure, a person being photographed for facial authentication can pre-set consent conditions for all facial authentication services that person can use. The application examples of the UI in Figure 4 are not limited to access control devices and surveillance systems. An example in which a person sets consent conditions for payment functions will be described below.

[0095] Item 401 allows individuals using facial recognition to set consent conditions for payment functions. For example, if a person using facial recognition consents to all payment functions, there's no need to perform a separate consent process for each function. Furthermore, security is enhanced by limiting the validity period of the consent. Additionally, the person using facial recognition can easily demonstrate their consent by pressing a button on the UI.

[0096] Items 402 and 403 are settings that allow guardians of missing children to set consent conditions within the facial recognition system used to search for missing children. For example, by selecting the checkbox in item 403, a guardian can allow access to and use of the child's facial image or facial feature data only during the period the child was missing. Guardians can also select the radio button "All Locations" in the "Allowed Locations" section to allow facial recognition of the child at all locations during the period the child was missing. Furthermore, from a risk management perspective, written consent is required.

[0097] Items 404 and 405 allow employees to set consent conditions within the company-implemented employee facial recognition system. For example, the person being photographed for facial recognition can set "Working Hours" as the consent period. Therefore, when the person being photographed for facial recognition (here, the employee) changes their work location due to internal changes, temporary relocation, or a change in job duties, they do not need to reset the consent conditions for the acquisition and use of their facial image. For example, as shown in Figure 4, the person being photographed for facial recognition can choose to consent via a signature (the method of consent: a signature on a document printed at the workplace).

[0098] Here, facial recognition services with configurable settings (i.e., consent conditions) are not limited to the examples described above. For instance, the consent condition registration unit 301 may display settings related to a recommendation system that presents recommended products to individuals. Furthermore, the consent condition registration unit 301 may display settings related to identity verification via facial recognition.

[0099] The settings are not limited to the examples above. Settings may include, for example, specifying the timeframe for a person to consent to facial recognition, and the maximum payment amount for the payment function.

[0100] The methods for entering settings items are not limited to those described above. Other examples of entry methods include combo boxes, multi-select options, and toggle switches.

[0101] The content and input methods for setting up the project are not limited to the examples above, and it is possible to use any content and input methods that can be quantitatively set and are technically feasible in a facial recognition system.

[0102] Facial image registration method 406 is a method for registering facial images used for facial authentication. Here, the person being photographed for facial authentication uploads their facial image data pre-stored in the mobile terminal device 201. Furthermore, as described in items 402 and 403, in the case of a child's facial image, the guardian uploads the child's facial image on behalf of the child. Here, the facial image registration method is not limited to the examples described above. For example, the mobile terminal device 201 can capture the subject's facial image on-site. The present invention is not limited to these methods, and it is possible to use any method that can acquire facial images that can be used for facial authentication.

[0103] Consent acquisition method 407 is a method for obtaining consent from a person who is the subject of facial authentication. Here, a consent button is presented to the person who is the subject of facial authentication (illustrated as consent acquisition method 407). By pressing this consent button, the person who is the subject of facial authentication can express their intention to consent to facial authentication under the set (input) consent conditions.

[0104] Consent acquisition methods are not limited to the examples above. For instance, consent can be obtained by allowing a person to sign with their finger on a touchscreen. It is also possible to double-check the person's intent to consent by displaying a confirmation screen again after they have pressed the consent button. Furthermore, consent can be obtained individually, as described in item 404. Alternatively, electronic signatures can be used in item 404. Alternatively, consent can be obtained using different consent acquisition methods for each item. Consent acquisition methods are not limited to these examples, and it is possible to use any method that enables a person to understand the content of their consent and voluntarily agree.

[0105] The UI displayed by the consent registration unit 301 of the mobile terminal device 201 is not limited to these examples, and it is possible to use any screen display and operation method or any consent unit that enables detailed setting of consent conditions.

[0106] (Data Structures)

[0107] Figures 5A to 5C illustrate examples of data structures according to the first embodiment. More specifically, Figure 5A is a diagram illustrating data managed by a server device according to the first embodiment. Figure 5B is a diagram illustrating data 502 stored in an access / exit management device according to the first embodiment. Figure 5C is a diagram illustrating data 503 obtained by combining data 501 and data 502 according to the first embodiment.

[0108] The following describes an overview of the data 501 shown in Figure 5A. Data 501 is managed by the consent management unit 302 of server device 202 and includes consent conditions and facial images corresponding to multiple individuals. The tabular data 501 is updated when an individual, acting as a subject for facial authentication, registers new consent conditions or changes existing ones. The consent management unit 302 assigns a unique person ID to each individual and manages the consent conditions and facial images of multiple individuals in association with these person IDs. Therefore, based on the person ID, it is possible to appropriately reference the data associated with that person ID.

[0109] The first row in Figure 5A indicates the name of the item constituting data 501. The item name includes the person ID, registered facial image, consent conditions, metadata, etc. As described at the beginning of the first embodiment, the consent conditions here include the purpose of use, usage / storage period, data type (facial image or facial feature quantity), facial image acquisition method, etc. The metadata includes the processor, manager, and responsible department. For example, the second to fourth rows in data 501 show the consent conditions and metadata set by the person acting as the subject of facial authentication using the UI shown in Figure 4. Moreover, as described in the UI description in Figure 4, there are cases where a guardian registers consent conditions for a child on behalf of the child. In data 501, the person ID of the person who actually performed the registration using the UI in Figure 4 is displayed as "A", and the child's person ID is displayed as "a".

[0110] In addition to the project name mentioned above, Data 501 may also include, for example, age, gender, and email address. Data 501 may also include the date / time and location of obtaining the person's consent. Data 501 may also have an electronic signature to ensure that no fraud has occurred. The project name only needs to contain the information necessary for operating the system based on consent conditions, and this information is not limited to any specific information.

[0111] The following is an overview of the data 502 shown in Figure 5B. Data 502 is data stored in the recording unit 305 by the system setting unit 304 shown in Figure 3A, and includes the person IDs of all persons allowed to enter / exit the access / exit gate 312, as well as information associated with these person IDs. Furthermore, the data 502 used in Figure 3B includes the person IDs to be monitored, as well as information associated with such person IDs.

[0112] The following describes an overview of data 503 in Figure 5C. Data 503 is obtained by combining data 501 and data 502. Data 503 used in Figure 3A is obtained by combining the consent conditions for entry / exit management and the registered facial image from data 501 with the entry / exit authorization information from data 502. Data 503 used in Figure 3B is obtained by combining the consent conditions for surveillance and the registered facial image from data 501 with the surveillance purpose information from data 502. Data 501 and data 502 are combined by associating data corresponding to the same person ID with each other.

[0113] (Processing flowchart)

[0114] Figure 6A is a flowchart illustrating the access / exit authorization setting process performed by the access / exit management device according to the first embodiment. Note that the process shown in Figure 6A is implemented by the CPU 101a of the access / exit management device 203A (information processing device 101) executing the control program in ROM 101c.

[0115] Steps S601 to S604 are the processes by which the administrator of the access / exit management device 203A sets access / exit authorization for all personnel who are allowed to enter and exit.

[0116] In step S601, the access / exit management device 203A determines the system termination. If the system becomes difficult to continue for some reason, or a system stop command has been issued ("No" in step S601), then the access / exit management device 203A stops the system. On the other hand, if the access / exit management device 203A does not detect any abnormality in the system ("Yes" in step S601), then the process moves to step S602.

[0117] In step S602, the administrator of the access / exit management device 203A inputs access / exit authorization information for all persons who are allowed to enter and exit into the system setting unit 304.

[0118] In step S603, the system setting unit 304 records the data input by the administrator in step S602 in the recording unit 305. The data recorded here adopts the data 502 format.

[0119] In step S604, the system setting unit 304 waits for a preset time until it receives the next input.

[0120] Figure 6B is a flowchart illustrating the registration facial feature acquisition process performed by the access / exit management device according to the first embodiment. Note that the process in Figure 6B is implemented by the CPU 101a of the access / exit management device 203A (information processing device 101) executing the control program in ROM 101c.

[0121] Steps S605 to S611 are processes for obtaining registered facial images of persons who consent to the acquisition and use of facial images or facial feature quantities, and converting the registered facial images into registered facial feature quantities.

[0122] In step S605, the access / exit management device 203A determines the system termination. If the system becomes difficult to continue for some reason, or a system stop command has been issued ("No" in step S605), then the access / exit management device 203A stops the system. On the other hand, if the access / exit management device 203A does not detect any abnormality in the system ("Yes" in step S605), then the process moves to step S606.

[0123] In step S606, the consent condition query unit 303 reads the person IDs and access / exit authorization lists of all persons recorded in the data 502 from the recording unit 305. Then, the consent condition query unit 303 determines whether the access / exit management device 203A allows access / exit for all persons based on the obtained access / exit authorization list. The consent condition query unit 303 extracts the person IDs of all persons who are allowed access / exit.

[0124] In step S607, the consent condition query unit 303 obtains the corresponding consent conditions and registered facial images for the person IDs of all persons allowed to enter / exit as obtained in step S606.

[0125] More specifically, the consent condition query unit 303 transmits the person IDs of all persons allowed to enter / exit to the consent management unit 302. The consent management unit 302 extracts all data corresponding to the person IDs of all persons allowed to enter / exit from the data 501.

[0126] Next, the consent management unit 302 further extracts data related to entry / exit management from the extracted data. The consent management unit 302 transmits the extracted data to the consent condition query unit 303. Finally, the consent condition query unit 303 stores the obtained data in the recording unit 305.

[0127] In step S608, the consent condition determination unit 307 acquires data 503 (see Figure 5C).

[0128] More specifically, the consent condition determination unit 307 obtains the data 502 obtained in step S606 and the data extracted from data 501 in step S607 from the recording unit 305. The consent condition determination unit 307 obtains data 503 by combining elements with the same person ID from the two data sets (data 502 and the data extracted from data 501). Finally, the consent condition determination unit 307 stores data 503 in the recording unit 305.

[0129] In step S609, the consent condition determination unit 307 obtains the registered facial images of all persons who are permitted to obtain and use facial images or facial feature quantities from the data 503 obtained in step S608.

[0130] More specifically, for all the consent conditions of the individuals in data 503, the consent condition determination unit 307 determines whether there are any contradictions between the purpose and time period of use of the consent conditions. If there are no contradictions between the purpose and time period of use of a certain individual's consent conditions, then the consent condition determination unit 307 extracts the registered facial image of that individual that does not contradict the purpose and time period of use of the consent conditions. If there are contradictions between the purpose or time period of use of a certain individual's consent conditions, then the consent condition determination unit 307 deletes the data corresponding to that individual from the recording unit 305. The consent condition determination unit 307 obtains the registered facial images of all individuals whose purpose and time period of use of the consent conditions do not contradict each other.

[0131] In step S610, the face authentication unit 308 converts all registered face images extracted in step S609 into facial feature quantities. As a result, "registered facial feature quantities" are obtained.

[0132] In step S611, the system waits for a preset time.

[0133] The processing steps S605 to S611 are repeated periodically to continuously update the registered facial feature quantity of the facial authentication unit 308.

[0134] Figure 6C is a flowchart illustrating the process by which the access / exit management device according to the first embodiment controls the access / exit gate. Note that the process in Figure 6C is implemented by the CPU 101a of the access / exit management device 203A (information processing device 101) executing the control program in ROM 101c.

[0135] Steps S612 to S618 include processing for identifying the person shown in the image acquired by the image capture unit 310 from the registered facial image and processing for controlling the entry / exit gate 312.

[0136] In step S612, the access / exit management device 203A determines the system termination. If the system becomes difficult to continue for some reason, or a system stop command has been issued ("No" in step S612), then the access / exit management device 203A stops the system. On the other hand, if the access / exit management device 203A does not detect any abnormality in the system ("Yes" in step S612), then the process moves to step S613.

[0137] In step S613, the image capture unit 310 acquires an image of the person.

[0138] In step S614, the face detection unit 309 detects the facial region of a person in the image acquired by the image capture unit 310.

[0139] In step S615, the face detection unit 309 determines whether a facial region can be detected in step S614. If a facial region can be detected ("Yes" in step S615), then the face detection unit 309 moves to the processing in step S616. On the other hand, if a facial region cannot be detected ("No" in step S615), then the face detection unit 309 returns to the processing in step S612, and the image capture unit 310 acquires the image again.

[0140] In step S616, the face authentication unit 308 converts the facial region detected in step S614 into facial feature quantities.

[0141] Therefore, the "verification facial feature quantity" is obtained.

[0142] In step S617, the face authentication unit 308 compares the registered facial feature quantity obtained in step S610 with the verified facial feature quantity obtained in step S616 to specify the person ID of the person detected by the face detection unit 309 in step S614. If the person ID can be specified ("Yes" in step S617), then the face authentication unit 308 moves to the processing in step S618. If the person ID cannot be specified ("No" in step S617), then the face authentication unit 308 returns to the processing in step S612, and the image capture unit 310 acquires the image again.

[0143] In step S618, the gate control unit 311 opens the entrance / exit gate 312 by controlling the entrance / exit gate 312.

[0144] The process returns to step S612, and facial authentication is performed while the system is operating.

[0145] The basic processing procedure in monitoring device 203B is similar to that in access / exit management device 203A described above. The following is a supplementary description of the processing performed by monitoring device 203B.

[0146] In steps S602 and S603 of Figure 6A, the administrator inputs a list of all persons who have been allowed to be monitored into the system setting unit 304. The system setting unit 304 stores the person IDs of the monitored targets and the information associated with those person IDs in the recording unit 305.

[0147] In steps S606 to S610 of Figure 6B, monitoring device 203B obtains consent conditions and registers facial feature data. Monitoring device 203B assigns person IDs to all persons to be monitored, similar to the case of access / exit management device 203A. Therefore, using a process similar to that of access / exit management device 203A, monitoring device 203B can query the consent management unit 302 of server device 202 using the person IDs.

[0148] In steps S613 to S617 of Figure 6C, the monitoring device 203B performs a process from acquiring verification images to comparing feature quantities. As shown in Figure 3B, the monitoring device 203B includes multiple image capture units 310 (multiple image capture devices). However, the monitoring device 203B transmits the images acquired by the image capture units 310 to the face detection unit 309 one by one. Accordingly, similar to the case described in the processing of the entry / exit management device 203A, the face images can be compared one by one.

[0149] If a person ID can be specified, then the monitoring device 203B performs a different process than that performed by the entry / exit management device 203A. Specifically, the monitoring device 203B notifies the system administrator of the installation location of the image capture device that captured the person's image. Specifically, the person monitoring unit 314 controls the notification unit 315 to notify the system administrator of the installation location of the image capture unit 310 (image capture device) that captured the person's image.

[0150] (Effect)

[0151] According to the aforementioned facial recognition system, facility users (i.e., the individuals being photographed for facial recognition) can pre-set their intention to consent to the application and use of facial images or facial feature data. Accordingly, facility users can only authorize the acquisition and use of facial images and facial feature data when necessary. Furthermore, facility users do not need to repeatedly perform the consent process with multiple facial recognition devices within the facility. Moreover, by extending the scope of consent set by facility users to other facilities, facility users do not need to repeat the consent process at facilities in other locations.

[0152] (Changes to the first embodiment)

[0153] In the first embodiment, an example is described in which the access control device 203A has one access gate 312, but the invention is not limited thereto, and multiple access gates 312 may be provided. Furthermore, multiple access gates 312 may be connected to enable communication with each other via a network. The number of access control devices 203A and the number of monitoring devices 203B are not limited to one, and there may be more than one of each.

[0154] Furthermore, the facial recognition system according to this disclosure is not limited to including access control equipment 203A and surveillance equipment 203B. For example, the facial recognition system may include electronic payment devices and identity verification systems. Moreover, for example, the facial recognition system may be a facial recognition system provided by local governments in public institutions, or a facial recognition function provided by an individual digital camera.

[0155] In the first embodiment, an example of setting consent conditions regarding the acquisition and use of facial images or facial feature data is described. However, consent conditions are not limited to those concerning the acquisition and use of facial images or facial feature data. The content for setting consent conditions may include, for example, personal information (such as address, age, and gender) and biometric information (such as fingerprints, iris information, or vein information). In this way, there are no restrictions on the content for setting consent conditions, as long as it is information requiring consent to the acquisition and use of some information held by the person.

[0156] Second Embodiment

[0157] In the first embodiment, it is necessary for the person to pre-set, input, and register consent conditions regarding the acquisition and use of facial images or facial feature data. If the person does not pre-register consent conditions, then the person being photographed as the subject of facial authentication needs to use a cumbersome and lengthy method to set consent conditions.

[0158] Accordingly, the second embodiment illustrates an example of a facial authentication system in which, when a person serving as the subject of facial authentication has not pre-registered consent conditions, consent conditions for that person are predicted and presented on the spot, and consent is obtained from that person. The facial authentication system according to the second embodiment predicts and presents consent conditions with a high probability of being agreed to by the person. As a result, compared to presenting all hypothetical consent conditions to the person on the spot and selecting from them, the burden associated with choosing consent conditions for the person can be reduced.

[0159] Furthermore, if consent conditions that are clearly unnecessary for the person are obtained (e.g., even if the person makes an input error), privacy issues may arise, depending on the country or region. To address this issue, the second embodiment has the effect of preventing privacy-related problems by predicting the appropriate consent conditions to be presented to the person.

[0160] Furthermore, in event settings where multiple consent conditions need to be obtained, obtaining consent conditions using conventional, lengthy methods can be very time-consuming. To address this use case, the second embodiment presents the minimum required consent conditions using a simple expression. Accordingly, consent conditions for multiple individuals can be obtained in a shorter time compared to using conventional, lengthy methods. As a result, it is possible not only to reduce the burden associated with selecting consent conditions for individuals serving as subjects for facial authentication, but also to reduce the workload required of event operators when obtaining consent conditions for multiple individuals. In the second embodiment, the differences from the first embodiment will be described.

[0161] (System Configuration)

[0162] Figure 7 is a diagram illustrating an overview of the system configuration according to the second embodiment. An overview of the situation in system 70 where no pre-set consent conditions have been established for handling a person is described below. This system is similar to the first embodiment and includes an access / exit management device 203A and a monitoring device 203B. System 70 is a biometric authentication system that performs biometric authentication on a person, and is, for example, a facial recognition system. In this case, the system modules labeled with the same reference numerals as in the first embodiment are the same system modules.

[0163] If the person being photographed for facial authentication has not pre-set consent conditions, the instant consent registration device 701 shown in Figure 7 immediately obtains the consent conditions and registers the facial image from the person. The input devices of the instant consent registration device 701 (input device 102 shown in Figure 1) include an image capture device, a keypad input device, and a pointing device. The output device of the instant consent registration device 701 (output device 103 in Figure 1) is a display device. The instant consent registration device 701 is installed at the entrance of a company or large commercial facility. The instant consent registration device 701 prompts individuals who have not pre-set consent conditions to do so. Detailed functions of the instant consent registration device 701 will be described below with reference to Figures 8 and 9.

[0164] (Function Configuration)

[0165] Figure 8 is a diagram illustrating the functional configuration of an instant consent registration device according to a second embodiment. The instant consent registration device 701 predicts and presents consent conditions regarding the acquisition and use of facial images or facial feature data to the person on the spot. The instant consent registration device 701 obtains consent from the person as a subject undergoing facial authentication. The instant consent registration device 701 includes an image capture unit 801, a consent condition prediction unit 802, and a consent condition registration unit 301 described with reference to Figures 3A and 3B. The consent condition registration unit 301 of the instant consent registration device 701 is another consent condition registration unit for registering consent conditions regarding the acquisition and use of a person's biometric information. The consent condition registration unit 301 of the instant consent registration device 701 has a function different from that of the consent condition registration unit 301 of the mobile terminal device 201.

[0166] Image capture unit 801 is an image capture device that acquires images of a person as a subject for facial authentication. Here, image capture unit 801 is a surveillance camera. Image capture unit 801 acquires moving images. Image capture unit 801 transmits frames (one image at a time) as image data to consent condition prediction unit 802.

[0167] The consent condition prediction unit 802 is a consent condition prediction unit that predicts consent conditions for the acquisition and use of a person's biometric information based on at least one of the person's attributes, physical characteristics, behavior, and associated information. The consent condition prediction unit 802 uses an image acquired from the image capture unit 801 to predict the person's consent conditions for the acquisition and use of facial images or facial feature data. The consent condition prediction unit 802 uses the person's facial authentication image to predict appropriate consent conditions for presentation to the person.

[0168] Appropriate consent conditions are those to which the person presented with the consent conditions is highly likely to consent and which require the person's consent. If consent conditions for searching for a missing child, as illustrated in the first embodiment, are presented to a person (here, an adult), the person is unlikely to agree to the consent conditions for searching for a missing child. Even if the person (here, an adult) agrees to the consent conditions for searching for a missing child, that person will not use the function for searching for a missing child.

[0169] On the other hand, families with young children are more likely to agree to the conditions for searching for missing children. Such families are more likely to actually use the missing children search function.

[0170] If the consent conditions required for a person are presented to the person being photographed as a subject for facial authentication first, it is possible to reduce the burden on that person regarding consenting to the consent conditions.

[0171] To predict appropriate consent conditions for the individual, the instant consent registration device 701 uses images to predict suitable consent conditions. Specifically, the instant consent registration device 701 uses a neural network capable of inferring the subject's age from an image. In the case of a group (family) including children, the instant consent registration device 701's function of predicting the search for lost children, as described in the first embodiment, is necessary. On the other hand, in the case of a group (family) excluding children, the instant consent registration device 701's function of predicting the search for lost children is unnecessary.

[0172] As described in the first embodiment, the consent registration unit 301 obtains consent conditions and facial images from the person being photographed as the subject of facial authentication, and registers the obtained consent conditions and facial images in the consent management unit 302.

[0173] The consent condition registration unit 301 presents the consent conditions based on the predictions made by the consent condition prediction unit 802. Similar to the first embodiment, the consent conditions are presented using the UI presentation method shown in FIG. 4. Here, when the group (family) includes children, the consent condition registration unit 301 displays items related to the missing children search function in the UI. On the other hand, when the group only includes adults, the consent condition registration unit 301 does not display items related to the missing children search function in the UI. The person being photographed, as the subject of facial authentication, can change the consent conditions presented in the UI. The person can use, for example, a keyboard and a touch panel to change the consent conditions.

[0174] (UI: User Interface)

[0175] The UI presented by the consent condition registration unit 301 of the instant consent registration device 701 is similar to the UI shown in Figure 4. The consent conditions presented in the UI include consent conditions regarding the acquisition and use of a person's biometric information, which are predicted based on non-personally identifiable information obtained from the person's biometric information.

[0176] (Processing flowchart)

[0177] Figure 9 is a flowchart illustrating the process performed by the instant consent registration device according to the second embodiment. The process shown here is the instant consent registration device 701 performing the instant registration of consent conditions. Note that the process shown in Figure 9 is implemented by the CPU 101a of the instant consent registration device 701 (information processing device 101) executing the control program in ROM 101c.

[0178] In step S901, the immediate consent registration device 701 determines the system termination. If the system becomes difficult to continue for some reason, or a system stop command has been issued ("No" in step S901), then the immediate consent registration device 701 stops the system. On the other hand, if the immediate consent registration device 701 does not detect any abnormality in the system ("Yes" in step S901), then the process moves to step S902.

[0179] In step S902, the image capture unit 801 acquires an image of a person (hereinafter referred to as "person") who serves as the subject for facial authentication.

[0180] In step S903, the consent condition prediction unit 802 uses the image acquired by the image capture unit 801 to predict the consent conditions to be presented to the person. More specifically, the consent condition prediction unit 802 uses information about the person (non-personally identifiable information) obtained from the image acquired by the image capture unit 801 to predict the consent conditions to be presented to the person.

[0181] In step S904, the consent condition registration unit 301 presents the consent conditions predicted by the consent condition prediction unit 802 in step S903 to the person via the UI (Figure 4).

[0182] In step S905, the consent registration unit 301 obtains the consent conditions of the person and registers the facial image.

[0183] In step S906, the consent registration unit 301 uploads the consent conditions and registered facial image of the person obtained in step S905 to the server device 202.

[0184] In step S907, the immediate consent registration device 701 waits for a preset time until it accepts the next input.

[0185] After the processing in Figure 9 is completed, the entry / exit management device 203A and monitoring device 203B perform processing, but this processing is similar to the processing described in the first embodiment, so it will not be described again here.

[0186] (Effect)

[0187] According to the second embodiment, if the person being photographed for facial authentication has not pre-registered consent conditions, appropriate consent conditions can be predicted and presented to that person on the spot. This makes it possible to easily obtain consent conditions from the person being photographed for facial authentication.

[0188] Compared to presenting all assumed consent conditions to the person being facially authenticated and requiring them to choose on the spot, the facial authentication system according to the second embodiment reduces the burden on the person regarding consent conditions. As described at the beginning of the second embodiment, if consent conditions that are clearly unnecessary for the person being facially authenticated (e.g., even if the person being facially authenticated makes an input error) are obtained, privacy issues may arise, depending on the country or region. According to the second embodiment, to address this privacy issue, appropriate consent conditions are predicted and presented to the person, thereby making it possible to avoid privacy-related problems.

[0189] (Changes in the second embodiment)

[0190] In the second embodiment, an example is described where a function to predict whether the search for a missing child is necessary before presenting consent conditions to a person is described, but the invention is not limited to this example. For example, the access control device 203A may have access gates as in the first embodiment. Moreover, the access control device 203A may have multiple access gates. Furthermore, the multiple access gates may be connected to enable them to communicate with each other via a network. The number of access control devices 203A and the number of surveillance devices 203B are not limited to one, and there may be more than one of each. In addition, the facial recognition system according to this disclosure is not limited to including access control devices 203A and surveillance devices 203B. For example, the facial recognition system may include electronic payment devices and identity verification systems. The facial recognition system may also be a facial recognition system provided by local governments or public institutions.

[0191] (Changes in methods for predicting appropriate consent conditions)

[0192] In the second embodiment, a neural network capable of inferring the age of a subject from an image is used to present consent conditions for the search for missing children to a group (family) including children.

[0193] While this invention uses image prediction to determine consent conditions for groups (families) including children, it is not limited to this, and from a privacy perspective, prediction can be made using only non-personally identifiable information. First, personally identifiable information is information that can identify an individual, alone or in combination with other information. Personally identifiable information includes biometric information such as fingerprints, vein patterns, iris scans, and facial images.

[0194] On the other hand, it is also possible to use information that cannot identify an individual to predict consent conditions. Information that cannot identify an individual includes, for example, height, weight, age, gender, gait, clothing, and facial expressions. For example, when inferring information that cannot identify a person, images can be acquired from multiple surveillance cameras, and height or behavior can be inferred through triangulation. It is also possible to provide an information processing terminal that allows manual input of information that cannot identify a person. It is possible to allow a person being photographed for facial authentication to input information that cannot identify them on the spot via an information processing terminal. It is also possible to use neural networks capable of inferring information that cannot identify a person from images. Alternatively, a person's height and body type can be obtained by using a neural network capable of inferring the three-dimensional position of a person from an image.

[0195] Furthermore, combinations of the aforementioned inference methods (neural networks) can be used to infer information about the identity of unidentifiable individuals. There are no restrictions on these inference techniques; the only requirement is that it is possible to infer information about the identity of unidentifiable individuals.

[0196] The instant consent registration device 701 according to the second embodiment determines whether to request a missing children search system based on the age of the person being photographed as a facial authentication subject, and determines whether to display consent conditions related to the missing children search system. However, there is no limitation on using age as a predictive reference to predict the appropriate consent conditions to be presented to the person. The instant consent registration device 701 can predict consent conditions by using information about the person's attributes, physical characteristics, behavior, or related factors. The following are examples of installing the instant consent registration device 701 at the entrance of a station, hospital, or commercial facility and predicting consent conditions.

[0197] (property)

[0198] Minors are subject to the precondition that they cannot use payment functions linked to their accounts and credit cards. The instant consent registration device 701 infers the age of the person being photographed for facial authentication and does not present the consent conditions related to payment functions to the minor. Here, the attributes of the person to be inferred are not limited to their age. For example, the instant consent registration device 701 may infer the person's gender and / or ethnicity and determine the consent conditions to be presented to that person.

[0199] (Physical characteristics)

[0200] The instant consent registration device 701 can present consent conditions related to a guidance service that directs individuals with mobility impairments, such as those using wheelchairs or canes, to more accessible routes (accessible routes). The instant consent registration device 701 performs recognition using images acquired from pre-installed surveillance cameras and detects / identifies individuals with mobility impairments using wheelchairs or canes. The instant consent registration device 701 also combines the guidance service with presenting consent conditions related to facial authentication to the individuals with mobility impairments. The guidance service can be provided using, for example, audio guidance devices and surveillance cameras pre-installed in appropriate locations within the facility. The audio guidance device of the guidance service only guides a specific individual to a ramp and / or elevator upon detection of pre-consent. The physical characteristics inferred here are not limited to wheelchairs and / or canes. For example, the instant consent registration device 701 can infer at least one of a white cane, hearing aid, eye mask, glasses, height, and weight, and determine the consent conditions to be presented to the individual.

[0201] (Behavior)

[0202] The instant consent registration device 701 can determine that a person whose face is obscured is sensitive to the handling of their personal information and can present only the minimum consent conditions regarding payment functionality. Here, the behavior inferred by the instant consent registration device 701 is not limited to the act of obscuring their face. For example, the instant consent registration device 701 can determine the consent conditions to be presented to the person by inferring at least continuous back-and-forth movement or the execution of specific gestures.

[0203] (Related information)

[0204] Upon detecting a person wearing a sign indicating a need for assistance, the instant consent registration device 701 can present consent conditions regarding the use of a monitoring service involving facial recognition. Signs indicating a need for assistance include, for example, "help sign" and "pregnancy sign." The instant consent registration device 701 detects signs indicating a need for assistance by recognizing images acquired from pre-installed surveillance cameras. To provide the monitoring service, the instant consent registration device 701 presents consent conditions regarding the implementation of facial recognition to the wearer of the sign indicating a need for assistance. The monitoring service is a service used to assist persons who require assistance from others. The instant consent registration device 701 acquires images of persons from surveillance cameras installed in stations or hospitals and determines the person's physical condition by inferring their posture.

[0205] Upon detecting a person's physical impairment, the immediate consent registration device 701 identifies the individual through facial recognition. In the case of a seriously injured person, the immediate consent registration device 701 quickly establishes contact with emergency contacts or requests emergency transportation. The relevant information inferred by the immediate consent registration device 701 is not limited to signs indicating a request for assistance from others. For example, the immediate consent registration device 701 can determine the consent conditions to be presented to the person by inferring the surrounding circumstances (such as accompanying persons, guide dogs, or objects).

[0206] As a result, it may be possible to perform facial recognition only on individuals who consent to use services at train stations, hospitals, or commercial facilities. It's important to note that the prediction of consent conditions to be presented to individuals is not limited to this. For example, a neural network capable of inferring appropriate consent conditions directly from an image can be used to predict the consent conditions to be presented to an individual. Alternatively, the consent conditions to be presented to an individual can be predicted by using a combination of the inference methods described above. References to predicting consent conditions are not limited to these examples, and only require the method to predict consent conditions that indicate a high probability of consent from the individual being photographed for facial recognition and that require that individual's consent.

[0207] (Additional use of non-image information)

[0208] Imagine a scenario where a person enters an event venue with a two-dimensional barcode ticket. In this case, information about that person (user) can be obtained through means other than image recognition. This information can be used to determine the consent conditions to be presented to the user. Examples of information about a person obtained without using image recognition include the date / time of visit, time period, location, event content, number of visitors, and weather. This type of information can be used to predict the circumstances surrounding the person.

[0209] (Consent to use non-facial biometric information)

[0210] In the second embodiment, an example of setting consent conditions related to the acquisition and use of facial images or facial feature quantities is illustrated. However, consent conditions are not limited to those concerning the acquisition and use of facial images or facial feature quantities. The content for setting consent conditions may include, for example, personal information (such as address, age, and gender) and biometric information (such as fingerprints, iris information, or vein information). In this way, there are no restrictions on the content for setting consent conditions, as long as it is information requiring consent to the acquisition and use of some information held by the person.

[0211] (Presentation of multiple conditions for consent)

[0212] In the second embodiment, aspects are described in which consent conditions, inferred to be suitable for presentation to a person (user), are presented to the person (user), allowing the person to determine whether to consent, and allowing the person to make changes to items not consented to. Examples of UI elements for making changes to items include checkboxes and radio buttons, as shown in Figure 4. Alternatively, a simpler consent method can be envisioned, where consent condition candidates are presented to the person using sentences, etc., and the person is allowed to select only one candidate from the candidates. An example of this method is presenting the person with three options, as described below.

[0213] Option 1 for agreeing to the terms: Agree to use facial recognition data for payment services (facial recognition data will be deleted immediately after payment is completed).

[0214] Option 2 for consent: Consent to use facial data for the missing children search service (facial data will be deleted after the child is found or 24 hours later).

[0215] Other: Presenting other consent option candidates

[0216] In this way, the method of presenting the consent conditions to be presented to the person may include aspects other than those shown in Figure 4.

[0217] According to this disclosure, it is possible to use biometric information about individuals safely and easily.

[0218] Other embodiments

[0219] The embodiments of this disclosure can also be implemented by a computer that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a "non-transitory computer-readable storage medium") to perform the functions of one or more embodiments described above and / or includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing the functions of one or more embodiments described above, and by a method executed by a computer of a system or device, for example, by reading and executing computer-executable instructions from a storage medium to perform the functions of one or more embodiments described above and / or controlling one or more circuits to perform the functions of one or more embodiments described above. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessor unit (MPU)) and may include a network of individual computers or individual processors to read and execute computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or storage medium. The storage medium may include, for example, a hard disk, random access memory (RAM), read-only memory (ROM), storage devices for distributed computing systems, optical discs (such as CDs, DVDs, or Blu-ray discs). TM One or more of the following: flash memory devices, memory cards, etc. Embodiments of the present invention can also be implemented by providing software (including computer program products of computer programs) that performs the functions of the above embodiments to a system or device via a network or various storage media, and the computer (central processing unit (CPU), microprocessor unit (MPU) of the system or device) reads and executes the computer program.

[0220] While this disclosure has been described with reference to exemplary embodiments, it should be understood that this disclosure is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. A biometric authentication device, comprising: At least one processor; And at least one memory thereon storing instructions, which, when executed by the at least one processor, cause the biometric authentication device to at least: obtain consent conditions for the acquisition and use of biometric information about a person; And perform biometric authentication on the person based on consent conditions.

2. The biometric authentication device according to claim 1, wherein it is determined whether to allow the acquisition and use of the person's biometric information based on consent conditions, and if it is determined that the acquisition and use of the person's biometric information is allowed, biometric authentication is performed on the person.

3. The biometric authentication device according to claim 2, wherein the determination of whether to allow the acquisition and use of the person's biometric information is based on whether the consent conditions include contradictory items.

4. The biometric authentication device according to claim 1, wherein the consent condition instructs the person to allow the acquisition and use of the person's biometric information.

5. The biometric authentication device according to claim 1, wherein the consent conditions include at least one of the following regarding the acquisition and use of the biometric information of the person: purpose of use, time period of use, storage period, type of biometric information, processor, manager, responsible department, and acquisition method.

6. The biometric authentication device according to claim 1, wherein the consent conditions include at least one of consent conditions pre-registered by the person and consent conditions predicted based on non-personally identifiable information obtained from the person's biometric information.

7. The biometric authentication device according to claim 1, wherein the consent conditions for obtaining and using the biometric information of the person are obtained from a server device capable of communicating with the biometric authentication device.

8. The biometric authentication device according to claim 1, wherein the biometric authentication device is used for entry / exit management or for monitoring.

9. The biometric authentication device according to claim 1, wherein the biometric information includes at least one of the person's facial image and the person's facial feature quantity, and the biometric authentication is facial authentication.

10. The biometric authentication device of claim 1, wherein the consent conditions are associated with a unique person ID for each person.

11. A biometric authentication system, comprising: A mobile terminal device, the mobile terminal device including a consent registration unit for registering consent conditions for the acquisition and use of biometric information about a person; The server device includes an consent condition management unit for managing consent conditions registered by the consent condition registration unit; as well as A biometric authentication device, wherein the biometric authentication device includes: an acquisition unit for acquiring consent conditions from a server device, and a biometric authentication unit for performing biometric authentication on the person based on the consent conditions.

12. The biometric authentication system according to claim 11, wherein the consent condition registration unit registers the consent conditions in the consent condition management unit based on the result set by the person through the user interface where the consent conditions are set.

13. A biometric authentication system, comprising: An instant consent registration device, the instant consent registration device including another consent condition registration unit for registering consent conditions for the acquisition and use of a person's biometric information; The server device includes a consent condition management unit for managing consent conditions registered by the other consent condition registration unit; The device includes a biometric authentication unit, wherein the biometric authentication unit comprises: an acquisition unit for acquiring consent conditions from the server device, and a biometric authentication unit for performing biometric authentication on the person based on the consent conditions.

14. The biometric authentication system of claim 13, wherein the other consent condition registration unit registers consent conditions in the consent condition management unit based on the results set by the person through a user interface in which consent conditions are set, and the consent conditions presented in the user interface include consent conditions for the acquisition and use of the person's biometric information predicted based on non-personally identifiable information obtained from the person's biometric information.

15. The biometric authentication system of claim 13, wherein the instant consent registration device further comprises a consent condition prediction unit, the consent condition prediction unit being used to predict consent conditions for the acquisition and use of the person's biometric information based on at least one of the person's attributes, physical characteristics, behavior and associated information.

16. The biometric authentication system according to claim 13, wherein the biometric information includes at least one of the person's facial image and the person's facial feature quantity, and the biometric authentication is facial authentication.

17. The biometric authentication system of claim 13, wherein the consent conditions are associated with a unique person ID for each person.

18. A method performed by a biometric authentication device, comprising: Obtain consent conditions regarding the acquisition and use of a person's biometric information; And perform biometric authentication on the person based on consent conditions.

19. A non-transitory computer-readable storage medium storing a computer program, which, when read and executed by a computer, causes the computer to perform the method according to claim 18.

20. A method performed by a biometric authentication system, comprising: Register the conditions for obtaining and using a person's biometric information; Manage the consent conditions registered during the registration process; Conditions for obtaining consent; And perform biometric authentication on the person based on consent conditions.

21. A non-transitory computer-readable storage medium storing a computer program, which, when read and executed by a computer, causes the computer to perform the method according to claim 20.

22. A computer program product comprising a computer program, which, when read and executed by a computer, causes the computer to perform the method according to claim 18.

23. A computer program product comprising a computer program, which, when read and executed by a computer, causes the computer to perform the method according to claim 20.

Images