Biometric devices, biometric systems, methods, and programs

Abstract

The present invention aims to safely and easily utilize biometric information for each individual. [Solution] The system includes an acquisition means for acquiring consent conditions regarding the acquisition and use of a person's biometric information, and a biometric authentication means for performing biometric authentication on the person based on the consent conditions.

Description

Technical Field

[0001] The present invention relates to a biometric authentication device, a biometric authentication system, a method, and a program.

Background Art

[0002] In recent years, the importance of personal information protection has been increasing. In face recognition systems, strict restrictions have been imposed on the acquisition and use of face images and face feature amounts of an unspecified large number of people. Here, the face feature amount is obtained by converting the appearance pattern of a face into a multi-dimensional vector. Since an individual can be identified using the face feature amount, the face feature amount is regarded as a kind of biometric identifier. For example, in the state of Illinois in the United States, written consent of the person himself / herself is required at the time of obtaining the face feature amount. In addition, in the EU, a law prohibiting the use of a face recognition function in a public space is being considered. Depending on national and state laws and regulations, the acquisition and use of face feature amounts may be more restricted than the acquisition and use of face images.

[0003] Therefore, methods for restricting the acquisition and use of face images or face feature amounts have been proposed. Patent Document 1 proposes a system for deleting acquired personal information of a person when the person who is the subject of face recognition shows a specific gesture. Patent Document 2 proposes a face recognition system that obtains consent in advance from a person who is the subject of face recognition regarding the acquisition and use of personal information such as face images.

[0004] Here, the acquisition and use of face images and face feature amounts of an unspecified large number of people may be legally restricted depending on the use and / or location. However, when a person who is the subject under such legal restrictions individually consents to the use of personal information, there is a need to make the personal information of the person who consents to the use available. Thus, there is a need to adjust the restrictions on the use of personal information for each individual while restricting the use of personal information of an unspecified large number with respect to the acquisition and use of face images and face feature amounts.

Prior Art Documents

Patent Documents

[0005] [Patent Document 1] Patent No. 6150019 [Patent Document 2] Patent No. 7126138 [Non-patent literature]

[0006] [Non-Patent Document 1] Deng, Jiankang, et al."Retinaface:Single-shot multi-level face localization in the wild."Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.2020. [Overview of the initiative] [Problems that the invention aims to solve]

[0007] However, in Patent Document 1, if a person does not express their refusal to have their facial image or facial features acquired and used through gestures, a third party can acquire and use the person's facial image or facial features even if the person has not consented to the acquisition and use of their personal information. Furthermore, in Patent Document 2, the purpose and usage conditions associated with each individual's facial image or facial features cannot be changed.

[0008] Therefore, the present invention aims to safely and easily utilize biometric information for each individual. [Means for solving the problem]

[0009] To achieve the objectives of the present invention, an information processing device according to one embodiment of the present invention comprises the following configuration: an acquisition means for acquiring consent conditions regarding the acquisition and use of a person's biometric information, and a biometric authentication means for performing biometric authentication on the person based on the consent conditions. [Effects of the Invention]

[0010] According to the present invention, biometric information for each individual can be used safely and easily. [Brief explanation of the drawing]

[0011] [Figure 1] A diagram showing an example of the hardware configuration of the information processing device according to the first and second embodiments. [Figure 2] A diagram showing an overview of the system configuration according to the first embodiment. [Figure 3A] A diagram illustrating the functional configuration of the access control device according to the first embodiment. [Figure 3B] A diagram illustrating the functional configuration of the monitoring system according to the first embodiment. [Figure 4] A diagram showing an example of a UI screen in which a person according to the first embodiment sets consent conditions. [Figure 5A] A diagram illustrating the data 501 managed by the server device according to the first embodiment. [Figure 5B] A diagram illustrating the data 502 stored in the access control device according to the first embodiment. [Figure 5C] A diagram illustrating data 503 obtained by combining data 501 and data 502 according to the first embodiment. [Figure 6A] A flowchart illustrating the process for setting access permissions for the access control device according to the first embodiment. [Figure 6B] A flowchart illustrating the process for acquiring registered facial features of an access control device according to the first embodiment. [Figure 6C] A flowchart illustrating the process by which the access control device according to the first embodiment controls the access gate. [Figure 7] A diagram showing an overview of the system configuration according to the second embodiment. [Figure 8] A diagram showing an overview of the functional configuration of the instant consent registration device according to the second embodiment. [Figure 9] A flowchart illustrating the process performed by the consent immediate registration device according to the second embodiment. [Modes for carrying out the invention]

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0013] The first embodiment can set in detail consent conditions (usage, usage conditions) regarding the acquisition and use of a person's face image or face feature amount for each person, and can acquire the consent conditions of the person. In this specification, a person's face image and face feature amount are collectively referred to as biometric information. Further, the first embodiment can acquire and use a face image or face feature amount based on the consent conditions acquired for each person. Thereby, the face authentication system acquires and uses the face image or face feature amount of a person who has given consent, and does not inadvertently acquire and use the face image or face feature amount of a person who has not given consent. As a result, the privacy of the person can be protected.

[0014] FIG. 1 is a diagram showing an example of the hardware configuration of an information processing apparatus according to the first and second embodiments.

[0015] The information processing apparatus 101 is connected to an input device 102, an output device 103, and a network 104. In each embodiment, one or more information processing apparatuses 101 are used. Note that the information processing apparatus 101 is used as a mobile terminal device 201, a server device 202, an entrance / exit management device 203A, a monitoring device 203B, and a consent immediate registration device 701, which will be described later.

[0016] The information processing apparatus 101 includes a CPU 101a, a RAM 101b, a ROM 101c, an external storage device 101d, an input I / F 101e, an output I / F 101f, and a communication I / F 101g. Each component of the information processing apparatus 101 is communicably connected via a system bus 101h. ...

[0017] The CPU (Central Processing Unit) 101a provides comprehensive control over the entire information processing unit 101.

[0018] The RAM (Random Access Memory) 101b temporarily stores data supplied from an external device (not shown) via the external storage device 101d and / or the input I / F 101e and communication I / F 101g. The RAM 101b functions as the main memory and work area of ​​the CPU 101a.

[0019] The ROM (Read Only Memory) 101c stores control programs and other data executed by the CPU 101a.

[0020] The external storage device 101d is a storage device such as a hard disk and / or a memory card that is permanently installed on the information processing device 101. The external storage device 101d may also include optical discs such as flexible disks (FD) and compact disks (CD), magnetic or optical cards, IC cards, and memory cards that are removable from the information processing device 101.

[0021] The input I / F 101e is the interface between the information processing device 101 and the input device 102.

[0022] Output I / F 101f is the interface between the information processing device 101 and the output device 103.

[0023] The communication interface 101g is an interface between the information processing device 101 and an external device (not shown) connected to the network 104.

[0024] The input device 102 accepts user input. The input device 102 is, for example, a pointing device and a keyboard for the user to input data.

[0025] The output device 103 is a display for displaying the data and program processing results held by the information processing device 101. The display is, for example, an LCD (Liquid Crystal Display) or an organic EL (Organic Electro-luminescence) display.

[0026] Network 104 is a communication device for the information processing device 101 to communicate with external devices (not shown). The information processing device 101 communicates with, for example, a network camera that takes pictures of subjects, a database that acquires data, and an external server that queries for services.

[0027] The hardware configuration of the information processing device 101 is not limited to the above, and may have any configuration.

[0028] (First Embodiment) In the first embodiment, a system (more specifically, a biometric authentication system) consisting of an access control device 203A and a monitoring device 203B using facial recognition is described. The system is a facial recognition system for, for example, an employee access control system in a company, and a lost child search system in a large commercial facility. In the facial recognition system of the first embodiment, the user of the facial recognition system (i.e., the person who is the subject of facial recognition) can pre-set consent information regarding the use and conditions of biometric information (facial image or facial features) in various scenarios. Thus, the biometric information includes at least one of the person's facial image and facial features. In one embodiment, the biometric authentication is facial recognition. If the subject is a child (e.g., a minor), the child's guardian sets the child's consent information. This allows the user of the facial recognition system to permit the acquisition and use of facial images and facial features only when necessary.

[0029] Furthermore, when the system of the first embodiment is used as an employee access control system in a company, the following advantages are available: Employees will no longer need to repeatedly go through the consent procedure for facial recognition at multiple access control devices in affiliated companies. In addition, by extending the scope of employee consent for facial recognition beyond their workplace, employees will no longer need to go through the consent procedure for facial recognition when entering or leaving other companies.

[0030] Furthermore, when the system of the first embodiment is used as a lost child search system, the parents of the lost child will no longer need to go through the consent procedure for facial recognition of their child at multiple facial recognition devices within a facility (e.g., a commercial facility). In addition, by extending the scope of consent for facial recognition of the child to other facilities (e.g., other commercial facilities), the parents of the lost child will no longer need to go through the consent procedure for facial recognition of their child at other facilities in different locations.

[0031] In this first embodiment, the uses and conditions of use permitted by the person who will be the subject of facial recognition when acquiring and using facial images or facial features are referred to as "consent conditions." Consent conditions are the conditions under which a person permits the acquisition and use of their biometric information. Consent conditions include the purpose of use, usage period, storage period, validity period of consent, type of biometric information (e.g., facial image, facial features), handler, manager, department in charge, and acquisition method (e.g., image acquisition method) related to the acquisition and use of the person's personal information (biometric information). Consent conditions include at least one of the consent conditions that the person has registered in advance using the UI in Figure 4, and consent conditions predicted based on non-personally identifiable information obtained from the person's biometric information (e.g., the person's image). Furthermore, consent conditions may include detailed conditions other than those listed above, such as the country, location, and time period under which the use of the person's personal information is permitted. Consent conditions can be any setting conditions desired by the system user, and various items can be subject to consent conditions. Furthermore, system administrators may store other necessary information as metadata along with the consent conditions. For example, when obtaining consent for the use of an individual's personal information in accordance with local or organizational laws or rules, it may be necessary to store items such as the person handling the personal information, the person responsible for management, and the department in charge, along with the consent conditions. The stored metadata may include various types of data, not limited to the aforementioned metadata forms, as long as it is necessary and legally appropriate.

[0032] (System Configuration) Figure 2 is a diagram showing an overview of the system configuration according to the first embodiment.

[0033] System 20 consists of a mobile terminal device 201, a server device 202, an access control device 203A, and a monitoring device 203B. System 20 is a biometric authentication system for biometric authentication of individuals, such as a facial recognition system. As explained in Figure 1, the information processing device 101 is used as the mobile terminal device 201, the server device 202, the access control device 203A, and the monitoring device 203B. Therefore, each of the mobile terminal device 201, the server device 202, the access control device 203A, and the monitoring device 203B has the same configuration as the information processing device 101. The access control device 203A and the monitoring device 203B are examples of biometric authentication devices. In one embodiment, the biometric authentication device is used for access control or monitoring.

[0034] The mobile terminal device 201 is a device on which a person who is the subject of facial recognition sets (inputs) consent conditions, consent status, and related facial images, etc., regarding the acquisition and use of facial images or facial features. The system 20 comprises one or more mobile terminal devices 201. The input device of the mobile terminal device 201 (input device 102 in Figure 1) includes an imaging device, key input, and pointing device. The output device of the mobile terminal device 201 (output device 103 in Figure 1) is a display device. The mobile terminal device 201 is a smartphone or PC owned by an individual, but is not limited to these. However, the mobile terminal device 201 only needs to be a device that can register the consent conditions described later. A detailed method for registering the consent conditions will be described later using Figure 4.

[0035] The server device 202 manages the consent conditions of multiple individuals set on multiple mobile terminal devices 201. The input device of the server device 202 (input device 102 in Figure 1) is a keyboard device. The output device of the server device 202 (output device 103 in Figure 1) is a display device. The server device 202 consists of one or more information processing devices 101. The server device 202 may be a single PC. Alternatively, the server device 202 may be a cloud server connecting multiple PCs via a network. The consent conditions of the individuals to be managed will be described later using Figures 5A to 5C.

[0036] Access control device 203A is a device that controls the opening and closing of access gates using facial recognition. In this case, access control device 203A manages the entry and exit of employees in a company. The input device of access control device 203A (input device 102 in Figure 1) includes a keyboard, mouse, and imaging device. The output device of access control device 203A (output device 103 in Figure 1) includes a display device and an access gate device.

[0037] The monitoring device 203B monitors a specific person designated by facial recognition. Here, for example, it is assumed that the monitoring device 203B will be used to search for a lost person in a large commercial facility. The input device of the monitoring device 203B (input device 102 in Figure 1) includes a keyboard, mouse, and imaging device. The output device of the monitoring device 203B (output device 103 in Figure 1) includes a display device.

[0038] Here, the access control devices 203A (multiple) and monitoring devices 203B (multiple) are devices exemplified to illustrate various services utilizing the facial recognition system of the first embodiment. The first embodiment can also be applied to systems other than access control and monitoring systems, such as an electronic payment system using facial recognition, and a recommendation system that detects regular customers using facial recognition and provides services tailored to their preferences. When applying the first embodiment to these systems, the consent management mechanism described in the first embodiment can be utilized.

[0039] (Functional Configuration) Figure 3A is a diagram illustrating the functional configuration of the access control device according to the first embodiment. Figure 3B is a diagram illustrating the functional configuration of the monitoring system according to the first embodiment. Note that reference numerals 201, 202, 203A, and 203B in Figures 3A and 3B correspond to reference numerals 201, 202, 203A, and 203B in Figure 2. Also, functional blocks assigned the same reference numeral in Figures 3A and 3B have the same function.

[0040] The access control device 203A manages the access of employees who have consented to access control using facial recognition registered with the system 20. The functional configuration of the consent condition registration unit 301 of the mobile terminal device 201 and the consent management unit 302 of the server device 202 will be described below. The consent condition registration unit 301 is a consent condition registration means for registering consent conditions regarding the acquisition and use of a person's biometric information. The consent management unit 302 is a consent condition management means for managing the consent conditions registered by the consent condition registration unit 301.

[0041] The consent condition registration unit 301 of the mobile terminal device 201 acquires consent conditions and a facial image from the person to be the subject of facial recognition. The consent condition registration unit 301 registers the acquired consent conditions and facial image with the consent management unit 302 of the server device. The method for acquiring and registering consent conditions and facial images will be described later with reference to Figure 4.

[0042] The consent management unit 302 of the server device 202 manages the consent conditions and facial images registered by the consent condition registration unit 301. For example, the consent management unit 302 manages data using the data structure shown in Figure 5A. Here, the consent management unit 302 assigns a unique person ID to each individual (person) and manages the consent conditions and facial images in association with the person ID. Therefore, various data based on the person ID can be referenced as appropriate. In addition, the person who will be the subject of facial recognition can confirm their own person ID via the consent condition registration unit 301.

[0043] The following is a block diagram illustrating the functional configuration of the access control device 203A.

[0044] The consent condition inquiry unit 303 queries the consent management unit 302 of the server device 202 for the consent conditions and facial image for each individual (employee). The consent condition inquiry unit 303 is an acquisition means for acquiring consent conditions regarding the acquisition and use of a person's biometric information. Here, the consent condition inquiry unit 303 acquires a person ID (a string or sequence of numbers that can uniquely identify a person) from the database 306 via the recording unit 305. The process of saving the person ID to the database 306 will be described later in the explanation of the system setting unit 304. The consent condition inquiry unit 303 acquires consent conditions regarding the acquisition and use of a person's biometric information from the server device 202, which communicates with the access control device 203A (biometric authentication device). The consent condition inquiry unit 303 transmits the person ID to the consent management unit 302 and acquires the consent conditions and facial image corresponding to the transmitted person ID from the consent management unit 302. The consent condition inquiry unit 303 saves the acquired consent conditions and facial image to the database 306 via the recording unit 305.

[0045] The system configuration unit 304 obtains the person IDs of all persons permitted to enter and exit, and a list of gates that all persons permitted to enter and exit can use. The obtained information is then saved to the database 306 via the recording unit 305. At this point, the administrator of system 20 obtains the person IDs from the persons permitted to enter and exit, determines the gates that each person can use, and inputs the person IDs and available gates into the system configuration unit 304.

[0046] The recording unit 305 controls the saving, updating, and deletion of data in the database 306.

[0047] Database 306 is a database that stores data with the data structure shown in Figures 5A to 5C.

[0048] The consent condition determination unit 307 is a consent condition determination means that determines whether the acquisition and use of a person's biometric information is valid based on the consent conditions. The consent condition determination unit 307 determines whether the acquisition and use of a person's biometric information is valid based on whether or not there are contradictory consent conditions in the consent conditions. The consent condition determination unit 307 determines the consent conditions for each person and decides whether or not to perform facial recognition. Here, the consent condition determination unit 307 checks whether there are any contradictions in each item of the consent conditions. Specifically, the consent condition determination unit 307 checks whether the usage and storage period and the validity period of the consent have expired. The consent condition determination unit 307 also checks whether the purpose of use and the method of acquiring facial images are appropriate for the situation. The consent condition determination unit 307 also checks whether the person has permitted the use of facial features. Furthermore, as explained at the beginning of the first embodiment, the consent conditions include various items. Therefore, if the consent conditions include items other than those mentioned above, the consent condition determination unit 307 determines each item individually.

[0049] In the first embodiment, facial recognition is performed on individuals who meet all the consent conditions without any inconsistencies. The consent condition determination unit 307 also transmits the person ID and facial image of the person it determines to be eligible for facial recognition to the facial recognition unit 308.

[0050] The facial recognition unit 308 is a biometric authentication means that performs biometric authentication on a person based on consent conditions. The facial recognition unit 308 performs biometric authentication on a person if the consent condition determination unit 307 determines that the acquisition and use of the person's biometric information is valid. The facial recognition unit 308 identifies whose face is detected by the face detection unit 309. Specifically, the facial recognition unit 308 converts multiple registered face images in the recording unit 305 and the face image detected by the face detection unit 309 into facial feature quantities. The facial recognition unit 308 identifies the person ID by comparing the facial feature quantities of the multiple face images with the facial feature quantities of the detected face image.

[0051] When the facial recognition unit 308 converts a facial image into facial features, it uses a neural network that converts the outward appearance patterns of the face into multidimensional vectors (features). This neural network is pre-trained to convert pairs of facial images of the same person into features that are close together in the feature space, and pairs of facial images of different people into features that are far apart in the feature space. The above neural network is just one example, and other methods may be used to identify the person ID. Other methods include, for example, a dimensionality reduction method called principal component analysis (PCA) and a clustering method called k-means. Any other method that extracts multidimensional vectors (features) that distinguish between pairs of facial images of the same person and pairs of facial images of different people is acceptable, and is not limited to the above example.

[0052] When comparing facial features, the cosine similarity between each feature is calculated. If a pair of features has a cosine similarity exceeding a predetermined threshold, that pair is considered to have been extracted from the same person. However, the comparison of features is not limited to the method described above. For example, the person with the highest cosine similarity that exceeds the threshold may be considered to be the same person. Alternatively, Euclidean distance or Manhattan distance may be used. The comparison method is not limited to this, as long as the distance between the two features can be calculated quantitatively.

[0053] The face detection unit 309 detects face regions from images acquired by the imaging unit 310. Here, a face detection neural network called Retinaface (see Non-Patent Literature 1) is used. However, any network capable of detecting face regions is acceptable.

[0054] The imaging unit 310 is an imaging device that acquires images of a person who is the subject of facial recognition. The imaging unit 310 uses, but is not limited to, a surveillance camera. The imaging unit 310 is not limited to, as long as it can capture images or videos and acquire image data.

[0055] The gate control unit 311 controls the opening and closing of the entrance / exit gate 312 based on the authentication result of the facial recognition unit 308. If the facial recognition unit 308 successfully identifies the person ID, the gate control unit 311 sends a command to open the entrance / exit gate 312. On the other hand, if the facial recognition unit 308 fails to identify the person ID, the gate control unit 311 does not send a command to open the entrance / exit gate 312. If the facial recognition unit 308 fails to identify the person ID, the gate control unit 311 performs error processing to inform the user of the failure to identify the person ID through voice and screen display. The control method is not limited to these, as long as it is possible to open the entrance / exit gate 312 for persons with entry / exit privileges and not open it for persons without entry / exit privileges.

[0056] The entrance / exit gate 312 opens when it receives a command to open the entrance / exit gate 312 from the gate control unit 311, and does not open the entrance / exit gate 312 when it does not receive a command to open the entrance / exit gate 312.

[0057] Next, we will describe the monitoring device 203B shown in Figure 3B. The monitoring device 203B is used to search for lost children in large commercial facilities.

[0058] In Figures 3A and 3B, functional blocks assigned the same name and code have the same function. However, the system setting unit 304 specifies the person to be monitored by their person ID via the settings screen. For example, in the lost child search use case, the parent of the lost child provides the child's person ID to the administrator of the monitoring device 203B. The administrator then sets the child's person ID via the system setting unit 304.

[0059] Here, the monitoring device 203B, similar to the access control device 203A, transmits the person IDs of all persons to be monitored from the consent condition inquiry unit 303 to the consent management unit 302, and obtains the consent conditions and facial images. The facial recognition unit 308 also converts the acquired facial images into facial features. The facial recognition unit 308 compares the facial features obtained from the consent management unit 302 with the matching facial features obtained from the facial images acquired from the facial detection unit 309 to identify the person ID.

[0060] The imaging unit 313 is an imaging device that acquires images of people who are to be the subject of facial recognition. Unlike Figure 3A, the imaging unit 313 in Figure 3B consists of multiple imaging devices installed in locations to be monitored within the facility. Here, the imaging unit 313 in Figure 3B acquires moving images and transmits the images, which have been downsampled to the necessary number of frames for lost person search, to the face detection unit 309. For example, if there are 10 imaging devices that acquire images at 30fps, all 300 images per second are not transmitted to the face detection unit 309. Instead, one out of every 10 images from each imaging device is downsampled, and 30 images per second are transmitted to the face detection unit 309. However, the image transmission method is not limited to this. The designer of the facial recognition system can determine the image transmission method based on the processing performance of the imaging unit 313 and the search speed required for the facial recognition system.

[0061] Furthermore, the imaging unit 313 in Figure 3B adds its identification number and installation location as metadata to the image transmitted to the face detection unit 309, and transmits the image and metadata together to the face detection unit 309.

[0062] The person monitoring unit 314 acquires the authentication result from the facial recognition unit 308 and metadata from the imaging unit 313. Specifically, the person monitoring unit 314 transmits the identification number and installation location of the imaging unit 313 that captured the image of the subject to be monitored to the notification unit 315.

[0063] The notification unit 315 notifies the system administrator via the display of the identification number and installation location of the imaging unit 313 that captured the person being monitored who matches the search criteria. The notification method for notifying the identification number and installation location of the imaging unit 313 is not limited to this. For example, the notification unit 315 may directly notify the guardian of the person being monitored of the identification number and installation location of the imaging unit 313 via email, or it may publish the identification number and installation location of the imaging unit 313 via a web server.

[0064] (UI; User Interface) Figure 4 shows an example of a UI screen in which a person sets consent conditions according to the first embodiment.

[0065] The consent condition registration unit 301 of the mobile terminal device 201 displays a UI (User Interface) for the person to be the subject of facial recognition to set (input) consent conditions. As shown in items 401 to 405 of Figure 4, the consent condition registration unit 301 of the mobile terminal device 201 presents the person to be the subject of facial recognition with various consent conditions for facial recognition services. Based on the results set by the person in the UI for setting consent conditions, the consent condition registration unit 301 registers the consent conditions to the consent management unit 302.

[0066] Furthermore, while the first embodiment provided examples of an access control device 203A and a monitoring device 203B, the invention is not limited to these. According to the present invention, a person who is the subject of facial recognition can pre-set consent conditions for all facial recognition services that they may use. The application example of the UI in Figure 4 is not limited to an access control device and a monitoring system. Here, we will explain an example in which a person sets consent conditions for a payment function.

[0067] Item 401 allows individuals to set consent conditions regarding payment functions for those being used for facial recognition. For example, by agreeing to all payment functions, individuals being used for facial recognition can avoid having to go through individual consent procedures for each function. Furthermore, security can be enhanced by limiting the period during which the consent of the person being used for facial recognition is valid. In addition, individuals being used for facial recognition can easily indicate their consent by pressing a button on the UI.

[0068] Items 402 and 403 are settings for a facial recognition system to locate lost children, allowing parents to set consent conditions. For example, by selecting the checkbox in item 403, parents can allow the acquisition and use of their child's facial image or facial features only during the period their child is missing. Alternatively, by selecting "All locations" using the "Locations to consent" radio button, parents can allow facial recognition of their child in all locations during the period their child is missing. Furthermore, from a risk management perspective, a written consent form will be issued.

[0069] Items 404 and 405 are items that allow employees to set consent conditions for a facial recognition system implemented by a company for its employees. For example, the person being facially recognized may set the consent period for facial recognition to "during employment." Therefore, when the person being facially recognized (in this case, an employee) changes workplaces due to transfer, secondment, or job change, they do not need to re-enter the consent conditions regarding the acquisition and use of their facial images. For example, as shown in Figure 4, the person being facially recognized can choose a consent method by signature (consent method: illustrated by signing a document printed at their workplace).

[0070] Here, the facial recognition service that allows setting items (i.e., consent conditions) is not limited to the above. For example, the consent condition registration unit 301 may present setting items related to a recommendation system that presents recommended products to each individual. The consent condition registration unit 301 may also present setting items related to identity verification by facial recognition.

[0071] The contents of the settings are not limited to those listed above. For example, the settings may include fields to specify the time period during which a person will consent to facial recognition, and fields to input the maximum amount that can be used for payment functions.

[0072] Furthermore, the input methods for entering setting items are not limited to those described above. Input methods include, for example, combo boxes, multi-selectors, and toggle switches.

[0073] The content and input methods of the settings items are not limited to those described above, as long as they can be set quantitatively and are technically feasible in the facial recognition system.

[0074] The facial image registration method 406 is a method for registering facial images for facial recognition. In this method, the person to be the subject of facial recognition uploads their own facial image data that has been previously stored on the mobile terminal device 201. As described in the explanations of items 402 and 403, for children's facial images, the guardian uploads the child's facial image on behalf of the child. However, the method of registering facial images is not limited to the above. For example, the mobile terminal device 201 may take a picture of the subject's face on the spot. Any method that can obtain a facial image usable for facial recognition is acceptable.

[0075] Method 407 for obtaining consent is a method for obtaining consent for facial recognition from the person to be the subject of facial recognition. In this method, the person to be the subject of facial recognition is presented with a consent button (illustrated in Method 407 for obtaining consent). By pressing the consent button, the person to be the subject of facial recognition can indicate their consent to perform facial recognition under the set (input) consent conditions.

[0076] The methods for obtaining consent are not limited to those described above. For example, consent may be obtained by having a person sign with their finger on a touch panel. Alternatively, a confirmation screen may be displayed again after the person presses the consent button to double-check their consent. Furthermore, as described in item 404, the person's consent may be obtained separately. Electronic signatures may also be used in item 404. In addition, the person's consent may be obtained using different methods for each item. The method of obtaining consent is not limited to those described above, as long as it is possible to obtain consent in a way that shows the person understands the content of the consent and that it was given voluntarily.

[0077] The UI displayed by the consent condition registration unit 301 of the mobile terminal device 201 is not limited to these, as long as it allows for detailed setting of consent conditions and enables the implementation of any consent method.

[0078] (Data structure) Figures 5A to 5C show an example of the data structure according to the first embodiment. Specifically, Figure 5A is a diagram illustrating the data managed by the server device according to the first embodiment. Figure 5B is a diagram illustrating the data 502 stored in the access control device according to the first embodiment. Figure 5C is a diagram illustrating the data 503 obtained by combining data 501 and data 502 according to the first embodiment.

[0079] The overview of data 501 in Figure 5A is described below. Data 501 is data managed by the consent management unit 302 of the server device 202, and includes the consent conditions and facial images of multiple individuals. When a person who is the subject of facial recognition registers new consent conditions or changes existing consent conditions, this table-formatted data 501 is updated. Here, the consent management unit 302 assigns a unique person ID to each individual and manages the consent conditions and facial images of multiple individuals in association with the person ID. Therefore, based on the person ID, the respective data linked to the person ID can be referenced as appropriate.

[0080] The first row of Figure 5A shows the names of the items that make up Data 501. The item names include Person ID, Registered Face Image, Consent Conditions, and Metadata. As described in the introductory explanation of the First Embodiment, the consent conditions here include the purpose of use, usage and storage period, data type (face image or face features), and face image acquisition method. The metadata includes the handler, manager, and department in charge. For example, the second to fourth rows of Data 501 show the consent conditions and metadata set by the person who will be the subject of face recognition using the UI shown in Figure 4. Also, as described in the explanation of the UI in Figure 4, a parent may register the consent conditions for their child on behalf of the child. Here, in Data 501, the Person ID of the person actually registered using the UI in Figure 4 is displayed as "A", and the Person ID of the child is displayed as "a".

[0081] Data 501 may also contain, for example, age, gender, and email address, in addition to the item names described above. Furthermore, Data 501 may also contain the date and location where the person's consent was obtained. Additionally, Data 501 may contain an electronic signature guaranteeing that no one is impersonating the person. The item names only need to contain information necessary for operating the system based on the consent conditions, and the information is not limited to specific information.

[0082] The overview of data 502 in Figure 5B is described below. Data 502 is data stored in the recording unit 305 by the system setting unit 304 in Figure 3A, and includes the person IDs of all persons who can enter and exit the entrance / exit gate 312, and information associated with the person IDs. Furthermore, the data 502 applied to Figure 3B shows all the person IDs of the monitored persons and the information associated with the person IDs.

[0083] The overview of data 503 in Figure 5C is described below. Data 503 is data that combines data 501 and data 502. The data 503 applied to Figure 3A is data that combines the consent conditions and registered facial images related to access control from data 501 and the access rights from data 502. The data 503 applied to Figure 3B is data that combines the consent conditions and registered facial images related to monitoring from data 501 and the monitoring purpose from data 502. The combination of data 501 and data 502 is performed by associating data for the same person ID.

[0084] (Processing flowchart) Figure 6A is a flowchart illustrating the process for setting access rights for the access control device according to the first embodiment. The process shown in Figure 6A is realized when the CPU 101a of the access control device 203A (information processing device 101) executes the control program in the ROM 101c.

[0085] S601-S604 are processes in which the administrator of the access control device 203A sets the access permissions for all persons who are permitted to enter or exit.

[0086] In S601, the access control device 203A determines whether the system is terminated. If the access control device 203A determines that it is difficult to continue the system for any reason, or if a system stop command is issued (No in S601), it stops the system. On the other hand, if the access control device 203A does not detect any system abnormality (Yes in S601), it proceeds to S602.

[0087] In S602, the administrator of the access control device 203A enters the access permissions of all persons who are permitted to enter and exit into the system setting unit 304.

[0088] In S603, the system setting unit 304 records the data entered by the administrator in S602 in the recording unit 305. The data to be recorded has the format of data 502.

[0089] In S604, the system setting unit 304 waits for a predetermined amount of time before accepting the next input.

[0090] Figure 6B is a flowchart illustrating the process of acquiring registered facial features of the access control device according to the first embodiment. The process shown in Figure 6B is realized when the CPU 101a of the access control device 203A (information processing device 101) executes the control program of the ROM 101c.

[0091] S605-S611 are processes that retrieve registered facial images of individuals who have consented to the acquisition and use of facial images or facial features, and convert the registered facial images into registered facial features.

[0092] In S605, the access control device 203A determines whether the system is terminated. If the access control device 203A determines that it is difficult to continue the system for any reason, or if a system stop command is issued (No in S605), it stops the system. On the other hand, if the access control device 203A does not detect any system abnormality (Yes in S605), it proceeds to S606.

[0093] In S606, the consent condition inquiry unit 303 reads a list of all individuals' person IDs and entry / exit privileges recorded in data 502 from the recording unit 305. Here, the consent condition inquiry unit 303 determines from the acquired list of entry / exit privileges whether all individuals are permitted to enter or exit using the entry / exit control device 203A. The consent condition inquiry unit 303 extracts the person IDs of all individuals who are permitted to enter or exit.

[0094] In S607, the consent condition inquiry unit 303 retrieves the corresponding consent conditions and registered facial images for the person IDs of all persons authorized to enter and exit, which were obtained in S606.

[0095] Specifically, the consent condition inquiry unit 303 sends the person IDs of all persons permitted to enter and exit to the consent management unit 302. The consent management unit 302 extracts all data from data 501 that matches the person IDs of all persons permitted to enter and exit.

[0096] Next, the consent management unit 302 further extracts data related to access control from the extracted data. The consent management unit 302 then transmits the extracted data to the consent condition inquiry unit 303. Finally, the consent condition inquiry unit 303 stores the acquired data in the recording unit 305.

[0097] In S608, the consent condition determination unit 307 acquires data 503 (see Figure 5C).

[0098] Specifically, the consent condition determination unit 307 obtains the data 502 acquired in S606 and the extracted data of data 501 acquired in S607 from the recording unit 305. The consent condition determination unit 307 obtains data 503 by combining elements of the two data (data 502 and the extracted data of data 501) that have the same person ID. Finally, the consent condition determination unit 307 saves data 503 to the recording unit 305.

[0099] In S609, the consent condition determination unit 307 obtains registered facial images of all individuals who have given permission to acquire and use facial images or facial features from the data 503 acquired in S608.

[0100] Specifically, the consent condition determination unit 307 determines whether there is any inconsistency between the purpose of use and the period of use of the consent conditions for all individuals in the data 503. If there is no inconsistency between the purpose of use and the period of use of the consent conditions, the consent condition determination unit 307 extracts the registered face images of individuals for whom there is no inconsistency between the purpose of use and the period of use of the consent conditions. If there is an inconsistency between the purpose of use and the period of use of the consent conditions, the consent condition determination unit 307 deletes the data of individuals for whom there is an inconsistency between the purpose of use and the period of use of the consent conditions from the recording unit 305. Then, the consent condition determination unit 307 obtains the registered face images of all individuals for whom there is no inconsistency between the purpose of use and the period of use of the consent conditions.

[0101] In S610, the facial recognition unit 308 converts all registered facial images extracted in S609 into facial features. This yields the "registered facial features".

[0102] In S611, the system waits for a predetermined amount of time.

[0103] By periodically repeating the processes in S605 to S611, the registered facial features of the facial recognition unit 308 are continuously updated.

[0104] Figure 6C is a flowchart illustrating the process by which the access control device according to the first embodiment controls the access gate. The process shown in Figure 6C is realized when the CPU 101a of the access control device 203A (information processing device 101) executes the control program in the ROM 101c.

[0105] S612 to S618 include the process of identifying the person in the image acquired by the imaging unit 310 from among the registered face images, and the process of controlling the entrance / exit gate 312.

[0106] In S612, the access control device 203A determines when the system should terminate. If the access control device 203A determines that it is difficult to continue the system for any reason, or if a system shutdown command is issued (No in S612), it will shut down the system. On the other hand, if the access control device 203A does not detect any system abnormalities (Yes in S612), it will proceed to S613.

[0107] In S613, the imaging unit 310 acquires an image of a person.

[0108] In S614, the face detection unit 309 detects the face region of a person from the image acquired by the imaging unit 310.

[0109] In S615, the face detection unit 309 determines whether it was able to detect a face region in S614. If the face detection unit 309 was able to detect a face region (Yes in S615), it proceeds to S616. On the other hand, if the face detection unit 309 was unable to detect a face region (No in S615), it returns to S612, and the imaging unit 310 reacquires the image.

[0110] In S616, the facial recognition unit 308 converts the facial region detected in S614 into facial features. This yields "matched facial features".

[0111] In S617, the facial recognition unit 308 identifies the person ID of the person detected by the face detection unit 309 in S614 by comparing the registered facial features acquired in S610 with the matched facial features acquired in S616. If the facial recognition unit 308 can identify the person ID (Yes in S617), it proceeds to S618. If the facial recognition unit 308 cannot identify the person ID (No in S617), it returns to S612, and the imaging unit 310 reacquires the image.

[0112] In S618, the gate control unit 311 opens the entrance / exit gate 312 by controlling it.

[0113] The process returns to S612, and facial recognition is performed while the system is running.

[0114] The basic processing procedure for monitoring device 203B is the same as that for access control device 203A described above. The following provides supplementary information about the processing performed by monitoring device 203B.

[0115] In steps S602 and S603 of Figure 6A, the administrator inputs a monitoring list of all individuals who are permitted to be monitored into the system configuration unit 304. The system configuration unit 304 stores the IDs of the monitored individuals and the information associated with those IDs in the recording unit 305.

[0116] In steps S606-S610 of Figure 6B, the monitoring device 203B acquires consent conditions and registered facial features. Similar to the access control device 203A, the monitoring device 203B assigns a person ID to all persons being monitored. Therefore, the monitoring device 203B can query the consent management unit 302 of the server device 202 using the person ID, following the same procedure as the access control device 203A.

[0117] In steps S613 to S617 of Figure 6C, the monitoring device 203B performs processing from the acquisition of matching images to the comparison of feature quantities. As shown in Figure 3B, the monitoring device 203B has multiple imaging units 310 (multiple imaging devices). However, the monitoring device 203B transmits the images acquired by the multiple imaging units 310 to the face detection unit 309 one by one. This allows for face image matching one by one, similar to the process described in the access control device 203A.

[0118] If the monitoring device 203B can identify a person's ID, it performs a different process than the access control device 203A. Specifically, the monitoring device 203B notifies the system administrator of the location of the imaging device that captured the person's image. Specifically, the person monitoring unit 314 controls the notification unit 315 to notify the system administrator of the location of the imaging unit 310 (imaging device) that captured the person's image.

[0119] (effect) With the facial recognition system described above, facility users (i.e., the individuals whose faces are to be recognized) can pre-set their consent regarding the use and conditions of their facial images or facial features. This allows facility users to permit the acquisition and use of their facial images and facial features only when necessary. Furthermore, facility users will no longer need to repeat the consent process at multiple facial recognition devices within the facility. In addition, by extending the scope of the user's consent to other facilities, facility users will no longer need to repeat the consent process at facilities located elsewhere.

[0120] (Variation of the first embodiment) In the first embodiment, an example was shown in which the access control device 203A has one access gate 312, but it is not limited to this, and may have multiple access gates 312. Furthermore, multiple access gates 312 may be connected to each other via a network so that they can communicate with one another. Also, the number of access control devices 203A and monitoring devices 203B is not limited to one, but may be multiple.

[0121] Furthermore, the facial recognition system of the present invention is not limited to the access control device 203A and the monitoring device 203B. For example, the facial recognition system may be an electronic payment device and an identity verification system. Also, the facial recognition system may be a facial recognition system provided by a local government to a public institution and a facial recognition function provided by an individual digital camera.

[0122] In the first embodiment, an example of setting consent conditions for the acquisition and use of facial images or facial features was shown. However, consent conditions are not limited to those for the acquisition and use of facial images or facial features. The subjects for which consent conditions are set include, for example, personal information such as address, age, and gender, and biometric information such as fingerprints, iris, and vein patterns. Thus, the subjects for which consent conditions are set are not limited to any information that requires consent for the acquisition and use of any information held by an individual.

[0123] (Second Embodiment) In the first embodiment, individuals had to pre-set, input, or register consent conditions for the acquisition and use of facial images or facial features. If an individual had not pre-registered consent conditions, the individual to be the subject of facial recognition had to set the consent conditions in a redundant and burdensome manner.

[0124] Therefore, the second embodiment presents an example of a facial recognition system that predicts and presents the consent conditions of a person to be the subject of facial recognition, and obtains consent from the person, when the person has not registered the consent conditions in advance. The facial recognition system of the second embodiment predicts and presents consent conditions that the person is most likely to agree to. This reduces the burden on the person in selecting consent conditions compared to presenting and having the person select from all possible consent conditions on the spot.

[0125] Furthermore, if consent conditions that are clearly unnecessary for an individual are obtained, even if it is due to a data entry error by the individual, it may become a privacy issue in some countries or regions. In contrast, the second embodiment has the effect of preventing privacy-related troubles by predicting appropriate consent conditions to be presented to the individual.

[0126] Furthermore, in event venues where consent conditions must be obtained from multiple individuals, conventional redundant methods can be time-consuming. To address such use cases, the second embodiment presents the minimum consent conditions in a simplified manner. This allows for obtaining consent conditions from multiple individuals in a shorter time compared to conventional redundant methods. As a result, not only is the burden of selecting consent conditions for individuals to be subject to facial recognition reduced, but the effort required by event organizers to obtain consent conditions from multiple individuals is also reduced. The differences between the second embodiment and the first embodiment will be explained below.

[0127] (System Configuration) Figure 7 is a diagram showing an overview of the system configuration according to the second embodiment. Similar to the first embodiment, the overview of the system 70, which includes an access control device 203 and a monitoring device 203B, will be described to address a situation in which a person has not set prior consent conditions. System 70 is a biometric authentication system that biometrically authenticates a person, for example, a facial recognition system. Here, system blocks assigned the same reference numerals in the figure as in the first embodiment mean that they are the same system.

[0128] The instant consent registration device 701 in Figure 7 acquires consent conditions and a registered facial image from a person on the spot if that person has not set consent conditions in advance. The input device of the instant consent registration device 701 (input device 102 in Figure 1) is an imaging device, a key input, and a pointing device. The output device of the instant consent registration device 701 (output device 103 in Figure 1) is a display device. The instant consent registration device 701 is installed at the entrances of companies and large commercial facilities. The instant consent registration device 701 then prompts the person who has not set consent conditions in advance to set them. A detailed explanation of the functions of the instant consent registration device 701 will be described later using Figures 8 and 9.

[0129] (Functional Configuration) Figure 8 is a diagram showing an overview of the functional configuration of the instant consent registration device according to the second embodiment. The instant consent registration device 701 predicts and presents consent conditions on the spot when acquiring and using a person's face image or facial features. The instant consent registration device 701 then obtains consent from the person who will be the subject of facial recognition. The instant consent registration device 701 comprises an imaging unit 801, a consent condition prediction unit 802, and a consent condition registration unit 301 as described in Figures 3A and 3B. The consent condition registration unit 301 of the instant consent registration device 701 is another consent condition registration means for registering consent conditions regarding the acquisition and use of a person's biometric information. The consent condition registration unit 301 of the instant consent registration device 701 has different functions from the consent condition registration unit 301 of the mobile terminal device 201.

[0130] The imaging unit 801 is an imaging device that acquires images of a person who will be the subject of facial recognition. In this case, the imaging unit 801 is a surveillance camera. The imaging unit 801 acquires moving images. The imaging unit 801 transmits each frame (one image at a time) as image data to the consent condition prediction unit 802.

[0131] The consent condition prediction unit 802 is a consent condition prediction means that predicts the consent conditions for acquiring and using a person's biometric information based on at least one of the person's attributes, physical characteristics, behavior, and associated information. The consent condition prediction unit 802 uses images acquired from the imaging unit 801 to predict the consent conditions of a person regarding the acquisition and use of face images or face features. The consent condition prediction unit 802 uses images of a face-recognized person to predict appropriate consent conditions for presenting to the person, depending on the situation.

[0132] Appropriate consent conditions for a given situation are those that the person is highly likely to agree to and that necessitate obtaining the person's consent. If the consent conditions for searching for a lost child, as exemplified in the first embodiment, are presented to a person (in this case, one adult), the person is unlikely to agree to the conditions for searching for a lost child. Even if consent is obtained from the person (in this case, one adult) regarding the conditions for searching for a lost child, this person will not use the function to search for a lost child.

[0133] On the other hand, families with young children are more likely to agree to the terms of the search for their lost child. And these families are more likely to actually use the lost child search function.

[0134] By prioritizing the presentation of consent conditions required by the person being used for facial recognition, the burden of consenting to those conditions can be reduced.

[0135] The instant consent registration device 701 predicts appropriate consent conditions using images in order to predict appropriate consent conditions according to the person's situation. Specifically, the instant consent registration device 701 uses a neural network that can estimate the age of the subject from the image. Here, the instant consent registration device 701 predicts that for groups (families) that include children, the function to find lost children, as mentioned in the first embodiment, is necessary. On the other hand, the instant consent registration device 701 predicts that for groups (families) that do not include children, the function to find lost children is unnecessary.

[0136] As described in the first embodiment, the consent condition registration unit 301 obtains consent conditions and a facial image from the person who will be the subject of facial recognition, and registers the obtained consent conditions and facial image in the consent management unit 302.

[0137] However, the consent condition registration unit 301 presents consent conditions based on the predictions of the consent condition prediction unit 802. The method for presenting consent conditions is the same as in the first embodiment, by presenting the UI shown in Figure 4. Here, the consent condition registration unit 301 displays items related to lost child search on the UI for groups (families) that include children. On the other hand, the consent condition registration unit 301 does not display items related to lost child search on the UI for groups of adults only. Furthermore, the person who is the subject of facial recognition may make changes to the consent conditions presented on the UI. The person can make changes to the consent conditions, for example, using a keyboard and touch panel.

[0138] (UI; User Interface) The UI presented by the consent condition registration unit 301 of the instant consent registration device 701 is the same as the UI in Figure 4. Here, the consent conditions presented in the UI include consent conditions regarding the acquisition and use of a person's biometric information, which are predicted based on non-personally identifiable information obtained from the person's biometric information.

[0139] (Processing flowchart) Figure 9 is a flowchart illustrating the process performed by the instant consent registration device according to the second embodiment. It shows the process procedure by which the instant consent registration device 701 immediately registers the consent conditions. Note that the process in Figure 9 is realized by the CPU 101a of the instant consent registration device 701 (information processing device 101) executing the control program of the ROM 101c.

[0140] In S901, the instant consent registration device 701 determines when the system is finished. If the instant consent registration device 701 determines that it is difficult to continue the system for any reason, or if a system stop command is issued (No in S901), it stops the system. On the other hand, if the instant consent registration device 701 does not detect any system abnormality (Yes in S901), it proceeds to S902.

[0141] In S902, the imaging unit 801 acquires an image of the person who will be the subject of facial recognition (hereinafter referred to as "person").

[0142] In S903, the consent condition prediction unit 802 uses the image acquired by the imaging unit 801 to predict the consent conditions to be presented to the person. More specifically, the consent condition prediction unit 802 uses information that does not identify who the person is (information that does not identify an individual) obtained from the image acquired by the imaging unit 801 to predict the consent conditions to be presented to the person.

[0143] In S904, the consent condition registration unit 301 presents the consent conditions predicted by the consent condition prediction unit 802 in S903 to the person via the UI (Figure 4).

[0144] In S905, the consent condition registration unit 301 obtains the person's consent conditions and registered facial image.

[0145] In S906, the consent condition registration unit 301 uploads the consent conditions and registered facial image of the person obtained in S905 to the server device 202.

[0146] In S907, the instant consent registration device 701 waits for a predetermined amount of time before accepting the next input.

[0147] After the process shown in Figure 9 is completed, the access control device 203A and the monitoring device 203B perform their respective processes. However, these processes are the same as those described in the first embodiment, so their explanation will be omitted.

[0148] (effect) According to the second embodiment, if a person to be the subject of facial recognition has not registered consent conditions in advance, appropriate consent conditions can be predicted and presented to the person on the spot. This makes it possible to easily obtain consent conditions from the person to be the subject of facial recognition.

[0149] The facial recognition system of the second embodiment can reduce the consent burden on a person subject to facial recognition compared to a system where all conceivable consent conditions are presented and the person is asked to select them on the spot. As explained at the beginning of the second embodiment, if consent conditions that are clearly unnecessary for the person subject to facial recognition are obtained, even if it is due to an input error by the person subject to facial recognition, it may become a privacy issue in some countries or regions. To address such privacy issues, the second embodiment can avoid privacy troubles by predicting appropriate consent conditions that should be presented to the person.

[0150] (Modified version of the second embodiment) In the second embodiment, an example was shown in which consent conditions are presented to a person after predicting whether a function to find a lost child is necessary, but the invention is not limited to this. For example, the access control device 203A may have an access gate, similar to the first embodiment. The access control device 203A may also have multiple access gates. Furthermore, multiple access gates may be connected so that they can communicate with each other via a network. In addition, the number of access control devices 203A and monitoring devices 203B is not limited to one, but may be multiple. The facial recognition system of the present invention is not limited to the access control device 203A and monitoring device 203B. For example, the facial recognition system may be an electronic payment device and an identity verification system. Furthermore, the facial recognition system may be a facial recognition system provided by a local government to a public institution.

[0151] (Variations of methods for predicting appropriate consent conditions) In the second embodiment, a neural network capable of estimating the age of a subject from an image was used to present consent conditions for searching for a lost child to a group (family) including a child.

[0152] Here, we predicted the consent conditions for a group (family) including children from the image, but this is not limited to this if predictions are made using information that does not identify individuals from a privacy perspective. First, personally identifiable information is information that can identify who an individual is, either individually or in combination. Personally identifiable information includes, for example, biometric information such as fingerprints, vein patterns, iris patterns, and facial images.

[0153] On the other hand, consent conditions may be predicted using information that does not identify an individual. Information that does not identify an individual includes, for example, height, weight, age, gender, gait, clothing, and facial expression. In estimating information that does not identify an individual, for example, images may be acquired from multiple surveillance cameras and height and behavior may be estimated by triangulation. Alternatively, an information processing terminal may be provided that allows manual input of information that does not identify an individual. The person to be the subject of facial recognition may then be asked to input the information that does not identify an individual via the information processing terminal. Alternatively, a neural network capable of estimating information that does not identify an individual from an image may be used. Alternatively, a neural network capable of estimating the 3D position of a person from an image may be used to determine the person's height and body type. Alternatively, a combination of the above estimation methods (neural networks) may be used to estimate information that does not identify an individual. The estimation method is not limited to these methods, as long as it can estimate information that does not identify an individual.

[0154] Furthermore, the instant consent registration device 701 of the second embodiment determined whether a lost child search system was necessary based on the age of the person being facially recognized, and decided whether to display consent conditions related to the lost child search system. However, age is not the only predictive criterion for appropriate consent conditions to be presented to a person. The instant consent registration device 701 can predict consent conditions using the person's attributes, physical characteristics, behavior, and supplementary information. Hereinafter, examples of how the instant consent registration device 701 installed at the entrances of train stations, hospitals, and commercial facilities predicts consent conditions are listed below.

[0155] (attribute) It is assumed that minors cannot use payment functions linked to accounts and credit cards. The instant consent registration device 701 estimates the age of the person being facially recognized and does not present consent conditions regarding payment functions to minors. Here, the attributes of the person to be estimated are not limited to age. For example, the instant consent registration device 701 may estimate the gender and / or race of the person and determine the consent conditions to be presented to the person.

[0156] (Physical characteristics) The instant consent registration device 701 may present consent conditions for a guidance service that directs wheelchair users or crutch users to easily accessible routes (barrier-free routes). The instant consent registration device 701 detects and identifies wheelchair users or crutch users by recognizing images acquired from pre-installed surveillance cameras. Furthermore, the instant consent registration device 701 presents consent conditions for the use of facial recognition as part of the guidance service to wheelchair users. The guidance service may be provided, for example, by using an audio guide device and surveillance cameras pre-installed in appropriate locations within the facility. The audio guide device for the guidance service will provide audio guidance only when it detects a specific person who has given prior consent, and will guide that person to a ramp and / or elevator. Here, the physical characteristics to be estimated are not limited to wheelchairs and / or crutches. For example, the instant consent registration device 701 may determine the consent conditions to be presented to a person by estimating at least one of the following: a white cane, hearing aid, eye patch, glasses, height, and weight.

[0157] (action) The instant consent registration device 701 may determine that a person who is making movements to conceal their face is a person who is sensitive to the handling of personal information, and may present only the minimum consent conditions regarding payment functions. However, the actions of a person that the instant consent registration device 701 estimates are not limited to movements to conceal the face. For example, the instant consent registration device 701 may determine the consent conditions to present to a person by estimating at least one of the following: movements of looking around restlessly, and certain gestures.

[0158] (Additional information) The instant consent registration device 701 may present consent conditions for a facial recognition-based monitoring service to a person wearing a mark indicating they need assistance from others. These marks include, for example, the Help Mark and the Maternity Mark. The instant consent registration device 701 detects these marks by recognizing images acquired from pre-installed surveillance cameras. Furthermore, the instant consent registration device 701 presents consent conditions for facial recognition to the person wearing the mark indicating they need assistance, in order to provide the monitoring service. The monitoring service is a service to support people who need assistance from others. The instant consent registration device 701 acquires images of people from surveillance cameras installed at stations or hospitals and determines if a person is unwell by estimating their posture. When the instant consent registration device 701 detects unwellness, it identifies the individual through facial recognition. The instant consent registration device 701 promptly contacts the emergency contact of the unwell person or notifies them to arrange for emergency transport. Here, the supplementary information estimated by the instant consent registration device 701 is not limited to the mark indicating they need assistance from others. For example, the instant consent registration device 701 may determine the consent conditions to present to a person by estimating the circumstances surrounding the person, such as companions, guide dogs, and personal belongings.

[0159] This allows facial recognition to be performed only on individuals who have consented to using services at train stations, hospitals, and commercial facilities. However, the prediction of consent conditions to be presented to individuals is not limited to this. For example, a neural network that can directly estimate appropriate consent conditions from an image may be used to predict the consent conditions to be presented to individuals. Alternatively, a combination of the above estimation methods may be used to predict the consent conditions to be presented to individuals. The prediction criteria for consent conditions are not limited to these, as long as the method can predict consent conditions that are likely to be agreed to by the person being facially recognized and for which consent from the person is necessary.

[0160] (Integration of information other than images) Let's consider a scenario where an individual enters an event venue by scanning a 2D barcode ticket. In this case, information about the individual (user) may be obtained through means other than image recognition. The consent conditions presented to the user may be determined using the information obtained in this way. Examples of information about an individual obtained without relying on image recognition include the date, time, and location of their visit, the content of the event, the number of attendees, and the weather on that day. This information may be useful in predicting the circumstances surrounding the individual.

[0161] (Consent to use biometric information other than facial features) In the second embodiment, an example of setting consent conditions for the acquisition and use of facial images or facial features was shown. However, consent conditions are not limited to those for the acquisition and use of facial images or facial features. The subjects for which consent conditions are set include, for example, personal information such as address, age, and gender, and biometric information such as fingerprints, iris, and vein patterns. Thus, the subjects for which consent conditions are set are not limited to any information that a person possesses for which consent is required for acquisition and use.

[0162] (Presentation of multiple consent conditions) In the second embodiment, we described a form in which consent conditions presumed to be appropriate for a person (user) are presented to the person (user), who is asked to decide whether or not to consent, and the person is asked to modify the items they do not consent to. UI elements for modifying items include checkboxes and radio buttons as shown in Figure 4. On the other hand, a simpler method of consent can also be considered in which candidate consent conditions are presented to the person in text or other format, and the person is asked to select only one from the candidates. This method can take the form of presenting the person with three options, as shown below.

[0163] Consent Condition 1: I agree to the payment service that uses facial features (facial features will be deleted immediately after payment is completed). Consent Condition 2: I agree to the use of facial features for the lost child search service (facial features will be deleted once the person is found or after 24 hours). Other: Suggest additional terms of agreement.

[0164] Thus, the method of presenting consent conditions to individuals can include forms other than those shown in Figure 4.

[0165] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0166] The disclosures herein include the following biometric devices, biometric systems, methods, and programs. (Item 1) A means of obtaining consent conditions regarding the acquisition and use of a person's biometric information, The system includes a biometric authentication means that performs biometric authentication on the person based on the aforementioned consent conditions. Biometric authentication device. (Item 2) The system further includes a consent condition determination means for determining whether the acquisition and use of the person's biometric information is valid based on the aforementioned consent conditions. The biometric authentication means performs the biometric authentication on the person if the consent condition determination means determines that the acquisition and use of the person's biometric information is valid. The biometric authentication device described in item 1. (Item 3) The consent condition determination means determines whether the acquisition and use of the person's biometric information is valid based on whether or not there are any contradictory items in the consent conditions. The biometric authentication device described in item 2. (Item 4) The aforementioned consent conditions are the conditions under which the person authorizes the acquisition and use of their biometric information. A biometric authentication device as described in any one of items 1 through 3. (Item 5) The aforementioned consent conditions include at least one of the following regarding the acquisition and use of the person's biometric information: purpose of use, period of use, retention period, type of biometric information, handler, person responsible for management, department in charge, and method of acquisition. A biometric authentication device as described in any one of items 1 through 4. (Item 6) The aforementioned consent conditions include at least one of the consent conditions previously registered by the person and the consent conditions predicted based on non-personally identifiable information obtained from the person's biometric information. A biometric authentication device described in any one of items 1 through 5. (Item 7) The acquisition means obtains consent conditions regarding the acquisition and use of the person's biometric information from a server device that communicates with the biometric authentication device. A biometric authentication device as described in any one of items 1 through 6. (Item 8) For access control or monitoring purposes, A biometric authentication device as described in any one of items 1 through 7. (Item 9) The aforementioned biometric information includes at least one of the person's facial image and facial features, The aforementioned biometric authentication is facial recognition. A biometric authentication device as described in any one of items 1 through 8. (Item 10) The aforementioned consent conditions are linked to a unique person ID for each individual. A biometric authentication device as described in any one of items 1 through 9. (Item 11) A mobile terminal device equipped with a consent condition registration means for registering consent conditions regarding the acquisition and use of a person's biometric information, A server device comprising consent condition management means for managing the consent conditions registered by the consent condition registration means, A biometric authentication device, An acquisition means for obtaining the consent conditions from the server device, A biometric authentication device comprising: biometric authentication means for performing biometric authentication on the person based on the aforementioned consent conditions; Biometric authentication system. (Item 12) The consent condition registration means registers the consent conditions to the consent condition management means based on the results set by the person to the user interface for setting the consent conditions. The biometric authentication system described in item 11. (Item 13) A consent instant registration device equipped with other consent condition registration means for registering consent conditions regarding the acquisition and use of a person's biometric information, A server device comprising consent condition management means for managing the consent conditions registered by the other consent condition registration means, A biometric authentication device, An acquisition means for obtaining the consent conditions from the server device, A biometric authentication device comprising: biometric authentication means for performing biometric authentication on the person based on the aforementioned consent conditions; Biometric authentication system. (Item 14) The other consent condition registration means registers the consent condition to the consent condition management means based on the results set by the person to the user interface for setting the consent condition, The consent conditions presented in the user interface include consent conditions regarding the acquisition and use of the person's biometric information, which are predicted based on non-personally identifiable information obtained from the person's biometric information. The biometric authentication system described in item 13. (Item 15) The instant consent registration device further comprises consent condition prediction means that predicts consent conditions regarding the acquisition and use of a person's biometric information based on at least one of the person's attributes, physical characteristics, behavior, and associated information. A biometric authentication system as described in item 13 or 14. (Item 16) The aforementioned biometric information includes at least one of the person's facial image and facial features, The aforementioned biometric authentication is facial recognition. A biometric authentication system described in any one of items 11 through 15. (Item 17) The aforementioned consent conditions are linked to a unique person ID for each individual. A biometric authentication system described in any one of items 11 through 16. (Item 18) A method performed by a biometric authentication device, The acquisition process involves obtaining consent conditions regarding the acquisition and use of a person's biometric information, The system includes a biometric authentication step that performs biometric authentication on the person based on the aforementioned consent conditions. method. (Item 19) A program that causes a computer to perform the actions described in item 18. (Item 20) A method performed by a biometric authentication system, A consent conditions registration process for registering consent conditions regarding the acquisition and use of a person's biometric information, A consent condition management step for managing the consent conditions registered by the consent condition registration step, The acquisition process for obtaining the aforementioned consent conditions, The system includes a biometric authentication step that performs biometric authentication on the person based on the aforementioned consent conditions. method. (Item 21) A program that causes a computer to perform the actions described in item 20. (Item 22) A method performed by a biometric authentication system, Other consent registration steps include registering consent conditions regarding the acquisition and use of a person's biometric information, A consent condition management step for managing the consent conditions registered by the other consent condition registration step, The acquisition process for obtaining the aforementioned consent conditions, The system includes a biometric authentication step that performs biometric authentication on the person based on the aforementioned consent conditions. method. (Item 23) A program that causes a computer to perform the actions described in item 22.

[0167] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0168] 20 Systems 201 Mobile terminal device 202 Server Device 203A Access control device 203B Monitoring device

Claims

1. A means of obtaining consent conditions regarding the acquisition and use of a person's biometric information, The system includes a biometric authentication means that performs biometric authentication on the person based on the aforementioned consent conditions. Biometric authentication device.

2. The system further includes a consent condition determination means for determining whether the acquisition and use of the person's biometric information is valid based on the aforementioned consent conditions. The biometric authentication means performs the biometric authentication on the person if the consent condition determination means determines that the acquisition and use of the person's biometric information is valid. The biometric authentication device according to claim 1.

3. The consent condition determination means determines whether the acquisition and use of the person's biometric information is valid based on whether or not there are any contradictory items in the consent conditions. The biometric authentication device according to claim 2.

4. The aforementioned consent conditions are the conditions under which the person authorizes the acquisition and use of their biometric information. The biometric authentication device according to claim 1.

5. The aforementioned consent conditions include at least one of the following regarding the acquisition and use of the person's biometric information: purpose of use, period of use, retention period, type of biometric information, handler, person responsible for management, department in charge, and method of acquisition. The biometric authentication device according to claim 1.

6. The aforementioned consent conditions include at least one of the consent conditions previously registered by the person and the consent conditions predicted based on non-personally identifiable information obtained from the person's biometric information. The biometric authentication device according to claim 1.

7. The acquisition means obtains consent conditions regarding the acquisition and use of the person's biometric information from a server device that communicates with the biometric authentication device. The biometric authentication device according to claim 1.

8. For access control or monitoring purposes, The biometric authentication device according to claim 1.

9. The aforementioned biometric information includes at least one of the person's facial image and facial features, The aforementioned biometric authentication is facial recognition. The biometric authentication device according to claim 1.

10. The aforementioned consent conditions are linked to a unique person ID for each individual. A biometric authentication device according to any one of claims 1 to 9.

11. A mobile terminal device equipped with a consent condition registration means for registering consent conditions regarding the acquisition and use of a person's biometric information, A server device comprising consent condition management means for managing the consent conditions registered by the consent condition registration means, A biometric authentication device, An acquisition means for obtaining the consent conditions from the server device, A biometric authentication device comprising: biometric authentication means for performing biometric authentication on the person based on the aforementioned consent conditions; Biometric authentication system.

12. The consent condition registration means registers the consent conditions to the consent condition management means based on the results set by the person to the user interface for setting the consent conditions. The biometric authentication system according to claim 11.

13. A consent instant registration device equipped with other consent condition registration means for registering consent conditions regarding the acquisition and use of a person's biometric information, A server device comprising consent condition management means for managing the consent conditions registered by the other consent condition registration means, A biometric authentication device, An acquisition means for obtaining the consent conditions from the server device, A biometric authentication device comprising: biometric authentication means for performing biometric authentication on the person based on the aforementioned consent conditions; Biometric authentication system.

14. The other consent condition registration means registers the consent condition to the consent condition management means based on the results set by the person to the user interface for setting the consent condition, The consent conditions presented in the user interface include consent conditions regarding the acquisition and use of the person's biometric information, which are predicted based on non-personally identifiable information obtained from the person's biometric information. The biometric authentication system according to claim 13.

15. The instant consent registration device further comprises consent condition prediction means that predicts consent conditions regarding the acquisition and use of a person's biometric information based on at least one of the person's attributes, physical characteristics, behavior, and associated information. The biometric authentication system according to claim 13.

16. The aforementioned biometric information includes at least one of the person's facial image and facial features, The aforementioned biometric authentication is facial recognition. The biometric authentication system according to claim 11 or 13.

17. The aforementioned consent conditions are linked to a unique person ID for each individual. The biometric authentication system according to claim 11 or 13.

18. A method performed by a biometric authentication device, The acquisition process involves obtaining consent conditions regarding the acquisition and use of a person's biometric information, The system includes a biometric authentication step that performs biometric authentication on the person based on the aforementioned consent conditions. method.

19. A program for causing a computer to perform the method described in claim 18.

20. A method performed by a biometric authentication system, A consent conditions registration process for registering consent conditions regarding the acquisition and use of a person's biometric information, A consent condition management step for managing the consent conditions registered by the consent condition registration step, The acquisition process for obtaining the aforementioned consent conditions, The system includes a biometric authentication step that performs biometric authentication on the person based on the aforementioned consent conditions. method.

21. A program for causing a computer to perform the method described in claim 20.

22. A method performed by a biometric authentication system, Other consent registration steps include registering consent conditions regarding the acquisition and use of a person's biometric information, A consent condition management step for managing the consent conditions registered by the other consent condition registration step, The acquisition process for obtaining the aforementioned consent conditions, The system includes a biometric authentication step that performs biometric authentication on the person based on the aforementioned consent conditions. method.

23. A program for causing a computer to perform the method described in claim 22.

Images