Authentication guide device and authentication guide method

Abstract

To provide an authentication guide device capable of supporting the efficiency of authentication by adaptively presenting a guide to effective authentication to users depending on the usage of authentication.SOLUTION: The authentication guide device includes: one or more acquisition units that acquire ambient condition information when obtaining one or more pieces of biological information that is used in multiple types of biometric authentication to identify a person; a guide determination unit that determines a guide method for presenting the execution order to give priority to the person among multiple types of biometric authentication based on surrounding environment information; and a guide display unit that displays the guide method on the display unit.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present disclosure relates to an authentication guidance device and an authentication guidance method.

Background Art

[0002] Patent Document 1 discloses an information processing apparatus that acquires a captured image obtained by capturing the face of a person to be photographed, refers to authentication data in which a captured image of a person's face is associated with specific information for identifying the person, and identifies the person corresponding to the person to be photographed based on the acquired captured image. In particular, when the information processing apparatus cannot identify the person corresponding to the person to be photographed, it accepts an input of the spoken voice of the person to be photographed, extracts specific information for identifying the person to be photographed from the spoken voice, and registers authentication data in which the captured image of the person to be photographed is associated with the extracted specific information.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is known a personal authentication technique for confirming (authenticating) whether a person to receive authentication matches a person registered in a database in advance by using the biometric information of the person. In recent years, in personal authentication techniques, the spread of multimodal biometric authentication that acquires multiple types (for example, face, voice) of biometric information from a person to receive authentication and performs authentication in multiple stages is expected. Thereby, it is expected to improve security against forgery by others and the like.

[0005] The authentication process performed by the information processing device described in Patent Document 1 includes facial recognition and voice recognition, but the execution order is predetermined, specifically, facial recognition followed by voice recognition. In other words, the execution order of each biometric authentication (hereinafter referred to as "modal biometric authentication") is predetermined, and it may not necessarily be an execution order suitable for the environment in which the person being authenticated (the subject of the photograph) acquires the biometric authentication. For example, if the biometric information is acquired in a bright environment, the accuracy of facial recognition is likely to improve, but if it is in a dark environment, the accuracy of facial recognition may deteriorate. However, authentication with a fixed execution order that does not take into account the environment in which the biometric information is acquired has problems such as poor authentication accuracy and long authentication times, so improvements in the efficiency of authentication are expected.

[0006] This disclosure was devised in light of the conventional circumstances described above, and aims to provide an authentication guide device and authentication guide method that adaptively presents effective authentication guides to users according to their authentication usage, thereby supporting the efficiency of authentication. [Means for solving the problem]

[0007] This disclosure comprises: one or more acquisition units that acquire information on the surrounding environmental conditions when acquiring one or more biometric information of a person used for multiple types of biometric authentication; a guide determination unit that determines a guide method for presenting the person with an execution order for preferentially executing the multiple types of biometric authentication based on the environmental conditions information; a guide display unit that displays the guide method on a display unit; and an overall determination unit that performs an authentication determination of the person using the processing results of the multiple types of biometric authentication executed according to the execution order of the displayed guide method. The guide determination unit determines the switching conditions for switching the execution order in the guide method, and the guide display unit, when the switching conditions are met, switches the execution order in the guide method that is displayed on the display unit when the switching conditions are met and displays it on the display unit. To provide an authentication guide device.

[0008] Furthermore, this disclosure relates to an authentication guiding method performed by an authentication guiding device, comprising the steps of: acquiring information on the surrounding environmental conditions when acquiring one or more biometric information of a person used for multiple types of biometric authentication; determining a guiding method that presents the person with an execution order for preferentially performing the multiple types of biometric authentication based on the environmental conditions information; displaying the guiding method on a display unit; and performing an authentication determination of the person using the processing results of the multiple types of biometric authentication performed according to the execution order of the displayed guiding method. Furthermore, in determining the guiding method, a switching condition for switching the execution order in the guiding method is determined, and in displaying the guiding method, if the switching condition is satisfied, the execution order in the guiding method displayed on the display unit at the time the switching condition is satisfied is switched and displayed on the display unit. To do so, provide an authentication guide method.

[0009] These comprehensive or specific embodiments may be implemented as systems, devices, methods, integrated circuits, computer programs, or recording media, or as any combination of systems, devices, methods, integrated circuits, computer programs, and recording media. [Effects of the Invention]

[0010] According to this disclosure, it is possible to adaptively present effective authentication guides to users depending on their authentication usage, thereby supporting the efficiency of authentication. [Brief explanation of the drawing]

[0011] [Figure 1] Block diagrams showing example system configurations for biometric authentication systems and hardware configurations for multimodal authentication devices. [Figure 2] Block diagram showing an example of the internal configuration of the processor in a multimodal authentication device. [Figure 3] This diagram shows a table that defines examples of conditions for determining whether or not to use facial recognition. [Figure 4] This diagram shows a table that defines examples of conditions for determining whether to use voice authentication. [Figure 5] This diagram shows a table that defines examples of conditions for determining whether fingerprint authentication can be used. [Figure 6] This diagram shows the first example of a guide method for modal biometric authentication that transitions over time. [Figure 7]Figure showing the second display example of the guiding method for modal biometric authentication that transitions over time [Figure 8] Figure showing an example of a display screen for successful authentication [Figure 9] Figure showing an example of a display screen for failed authentication [Figure 10] Flowchart showing an example of the overall operation procedure of a multimodal authentication device [Figure 11] Flowchart showing an example of the operation procedure of the first subroutine for determining the guiding method [Figure 12] Flowchart showing an example of the operation procedure of the second subroutine for determining the guiding method [Figure 13] Flowchart showing an example of the operation procedure of the subroutine for switching the guiding method [Figure 14] Flowchart showing an example of the operation procedure of the subroutine for determining the priority of modal biometric authentication

Mode for Carrying Out the Invention

[0012] Hereinafter, with appropriate reference to the drawings, embodiments specifically disclosing an authentication guiding device and an authentication guiding method according to the present disclosure will be described in detail. However, detailed descriptions that are more than necessary may be omitted. For example, detailed descriptions of well-known matters and duplicate descriptions of substantially the same configurations may be omitted. This is to avoid making the following description unnecessarily redundant and to facilitate the understanding of those skilled in the art. Note that the accompanying drawings and the following description are provided for those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims by these.

[0013] <Definition of Terms> First, terms used in the following embodiments (which may be referred to as "the present embodiments") will be defined.

[0014] (1) "Modal Biometric Authentication": It is a biometric authentication process performed by an authentication guide device (e.g., the multimodal authentication device 10) by acquiring or referring to the biometric information of the authentication target person in order for the authentication target person to perform an action that satisfies a specific use or purpose. It is a biometric authentication that comprehensively encompasses the following processes of face authentication, fingerprint authentication, and voice authentication. (2) "Face Authentication": It is one of the modal biometric authentications. By referring to the face image (face imaging data) of the authentication target person registered in the user information database (to be referred to later) or the feature amount for each feature point of the imaging data (an example of biometric information), it is an authentication that performs identity verification according to whether the authentication target person receiving face authentication matches the registered authentication target person corresponding to the face image or its feature amount. (3) "Fingerprint Authentication": It is one of the modal biometric authentications. By referring to the fingerprint image of the authentication target person registered in the user information database (to be referred to later) or its feature amount (an example of biometric information), it is an authentication that performs identity verification according to whether the authentication target person receiving fingerprint authentication matches the registered authentication target person corresponding to the fingerprint image (fingerprint imaging data) or the feature amount for each feature point of the imaging data. In the following embodiments, fingerprint authentication will be described as a non-contact type of authentication in which the finger of the authentication target person is executed non-contact with the sensor (e.g., the fingerprint camera 202), but it may also be a contact type of authentication executed in contact with the sensor. (4) "Voice Authentication": It is one of the modal biometric authentications. By referring to the voice of the authentication target person registered in the user information database (to be referred to later) or its feature amount (an example of biometric information), it is an authentication that performs identity verification according to whether the authentication target person receiving voice authentication matches the registered authentication target person corresponding to the voice or its feature amount. (5) "Surrounding Condition Information": It is information indicating conditions including various environments around the authentication target person at the time of acquiring one or more biometric information of the authentication target person used for multiple types of modal biometric authentications. It is information that comprehensively encompasses the following system condition information, environmental condition information, usage condition information, and user information respectively. (6) "System Condition Information": This is fixed (static) information that indicates the purpose of the biometric authentication system (for example, where the multimodal authentication device 10 will be placed and what actions will be permitted to the person being authenticated if biometric authentication is successful), the location where the sensor group (see below) will be placed, or the specifications of the sensor group (see below) at the time of designing the biometric authentication system or installing it at the actual installation site. (7) "Environmental Condition Information": This refers to information that indicates the environment surrounding the person being authenticated when acquiring one or more biometric data of the person being authenticated for use in multiple types of modal biometric authentication (e.g., brightness, sound pressure of noise signals), and unlike system condition information, it is information that can constantly change from time to time of acquisition. (8) "Usage Condition Information": This is information indicating the purpose for which the person being authenticated will use the biometric authentication system, and is information that the person being authenticated has decided when they visit the location where the biometric authentication system is installed. Usage condition information is information that requires the security level (see below) to be changed for each use, such as cash withdrawal, transfer, balance inquiry, deposit, etc. (9) "User Information": This refers to information that indicates the preferences or characteristics of the person being authenticated (e.g., proficiency level or personal attributes), and is fixed or may change depending on the mood of the person being authenticated when they visit the location where the biometric authentication system is installed. Personal attributes include, for example, information indicating that the person being authenticated is a twin.

[0015] <System Configuration> First, with reference to Figure 1, the system configuration of the biometric authentication system and the hardware configuration example of the multimodal authentication device according to this embodiment will be described. Figure 1 is a block diagram showing an example of the system configuration of the biometric authentication system 1 and an example of the hardware configuration of the multimodal authentication device 10.

[0016] The biometric authentication system 1 includes a multimodal authentication device 10, a user information database 30, and a drive device 40. In Figure 1, the database is denoted as DB. At least one of the user information database 30 and the drive device 40 may be included in the multimodal authentication device 10.

[0017] The multimodal authentication device 10 (an example of an authentication guide device) acquires ambient condition information (see above) when acquiring one or more biometric information of the person to be authenticated (user) to be used for multiple types of biometric authentication (modal biometric authentication), and determines a guide method that specifies the execution order for which multiple types of biometric authentication should be preferentially executed for the person to be authenticated (user) based on the acquired ambient condition information. The multimodal authentication device 10 displays the determined guide method on the display unit (see below), and in parallel with the display of the guide method, sequentially executes modal biometric authentication for the person to be authenticated (user) according to the determined guide method.

[0018] The multimodal authentication device 10 includes a sensor group 20, a display 21, a processor 22, a memory 23, a communication IF 24, and an input device 25. In Figure 1, the interface is denoted as IF. The multimodal authentication device 10 has a housing equipped with a sensor group 20 that includes sensors for acquiring one or more biometric information of an authenticated person (user) used for multiple types of modal biometric authentication, and may be, for example, a tablet, a PC (Personal Computer), or a mobile terminal. The multimodal authentication device 10 may also be composed of a single housing equipped with multiple sensors, or it may be composed of multiple housings equipped with at least one sensor. The multimodal authentication device 10 is installed, for example, in commercial facilities, amusement parks, tourist spots, or building entrances to verify the identity of authenticated persons (users) who are entering. The multimodal authentication device 10 is also installed, for example, in ATMs (Automatic Teller Machines) of financial institutions such as banks or post offices to verify the identity of authenticated persons (users). The locations mentioned above are merely examples of possible installation locations for the multimodal authentication device 10, and are not limited to these.

[0019] The sensor group 20 includes a face camera 201, a fingerprint camera 202, and a microphone 203. Note that the face camera 201 and the fingerprint camera 202 may use the same camera.

[0020] The face camera 201 is a camera device (an example of a sensor) that captures images of the face of the person to be authenticated. Hereinafter, image data obtained by capturing images with the face camera 201 or the fingerprint camera 202 will be referred to as image data. The face camera 201 acquires image data of the face of the person to be authenticated, which is biometric information necessary for face authentication, by capturing images according to a predetermined frame rate, or by calculating feature quantities (e.g., vector data) for each feature point in the image data using the image data. The face camera 201 outputs the image data or feature quantity data based on the image data to the processor 22. Note that the calculation of feature quantity data based on the image data of the face of the person to be authenticated may be performed by the processor 22.

[0021] The fingerprint camera 202 is a camera device (an example of a sensor) that captures the fingerprints of the person to be authenticated. The fingerprint camera 202 acquires image data of the person to be authenticated, which is biometric information necessary for fingerprint authentication, by capturing images according to a predetermined frame rate, or by calculating feature data (e.g., vector data) for each feature point in the image data using the image data. The fingerprint camera 202 outputs the image data or the feature data based on the image data to the processor 22. Note that the calculation of feature data based on the image data of the person to be authenticated may be performed by the processor 22.

[0022] Each of the face camera 201 and the fingerprint camera 202 may acquire ambient condition information (see above) for the multimodal authentication device 10 by calculating it based on the brightness component of the subject captured in the image data. Ambient environment information that can be acquired by the face camera 201 or the fingerprint camera 202 is, for example, the brightness of the area around where the multimodal authentication device 10 is located. Note that the acquisition of ambient condition information (e.g., brightness) is not limited to the face camera 201 and the fingerprint camera 202, but may also be performed by sensors including cameras provided in or outside the multimodal authentication device 10.

[0023] Microphone 203 is a microphone device (an example of a sensor) that captures the speech of the person to be authenticated. Microphone 203 acquires sound data based on the speech of the person to be authenticated, which is biometric information necessary for voice authentication, or it acquires feature data (e.g., vector data) from the sound data using the sound data. Microphone 203 outputs the sound data or the feature data based on the sound data to processor 22. Note that the calculation of feature data based on the sound data may be performed by processor 22.

[0024] Microphone 203 may acquire ambient condition information of the multimodal authentication device 10 by calculating it based on the signal components contained in the sound acquisition data. Ambient environment information that microphone 203 can acquire includes, for example, the sound pressure of noise signals other than the speech voice of the person being authenticated in the vicinity of the person being authenticated. Note that the acquisition of ambient environment information (e.g., noise signals) is not limited to microphone 203, but may also be performed by sensors including microphones provided in or outside the multimodal authentication device 10.

[0025] The display 21 (an example of a display unit) is a display device configured using, for example, an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence). The display 21 displays the authentication result screen obtained from the processor 22 (see, for example, the authentication success screen WD4 shown in Figure 8, or the authentication failure screen WD5 shown in Figure 9).

[0026] The processor 22 consists of a semiconductor chip on which at least one electronic device such as a CPU (Central Processing Unit), DSP (Digital Signal Processor), GPU (Graphical Processing Unit), or FPGA (Field Programmable Gate Array) is implemented. The processor 22 functions as a controller that oversees the overall operation of the multimodal authentication device 10, performing control processing to coordinate the operation of each part of the multimodal authentication device 10, data input / output processing between each part of the multimodal authentication device 10, data calculation processing, and data storage processing.

[0027] The processor 22 uses programs and control data stored in the ROM (Read Only Memory) of the memory 23 to implement the functions of the overall control unit 50 (see Figure 2) and the modal biometric authentication units 601, ..., 60n (see Figure 2). During operation, the processor 22 uses the RAM (Random Access Memory) of the memory 23 to temporarily store data or information generated or acquired by each unit functionally executed by the processor 22 in the RAM of the memory 23.

[0028] Memory 23 includes, for example, a ROM that stores a program defining various processes performed by the processor 22 and control data used during the execution of that program, and RAM as work memory used when executing various processes performed by the processor 22. The ROM contains a program defining various processes performed by the processor 22 and control data used during the execution of that program. The RAM temporarily stores data or information generated or acquired by the processor 22 (for example, biometric information acquired by each sensor device of the sensor group 20, or the score and confidence level of biometric authentication performed by the modal biometric authentication unit). Note that the ROM may be replaced with, for example, flash memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or an external server device, and the data may be read when the device starts up. Note that the data or information stored in RAM is not limited to these.

[0029] The communication IF24 is a network interface circuit that communicates wirelessly or via wired connection between the multimodal authentication device 10 and the user information database 30 and the drive device 40, respectively. The communication methods used for communication at the communication IF24 include, for example, mobile communications such as WAN (Wide Area Network), LAN (Local Area Network), Wi-Fi (registered trademark), LTE (Long Term Evolution), LTE-Advanced, and 5G, as well as power line communications, short-range wireless communications (e.g., Bluetooth (registered trademark) communications), and communications for mobile phones.

[0030] The input device 25 is a device that receives input from the person to be authenticated. The input device 25 is, for example, a touch panel display or a sensor that can authenticate an access card or employee ID. The input device 25 receives input from the person to be authenticated, such as name, gender, age, ID, password, biometric authentication usage conditions information (see above) or user information (see above) from the multimodal authentication device 10. The input device 25 sends the various types of input information to the processor 22. The input device 25 may be omitted from the configuration of the multimodal authentication device 10.

[0031] The user information database 30 is configured using a storage medium (e.g., flash memory, HDD (Hard Disk Drive), or SSD (Solid State Drive)). The user information database 30 stores previously registered biometric information data and various other information (name, age, gender, facial photograph, ID, password, preferences, proficiency level, etc.) of the person being authenticated used in modal biometric authentication. Biometric information includes, for example, facial image data, feature data of feature points on the facial image data, finger (fingerprint) image data, feature data of feature points on the fingerprint image data, spoken voice data, and feature data of the spoken voice. However, biometric information is not limited to these. The user information database 30 may be stored on, for example, a cloud server (not shown) and connected to the multimodal authentication device 10 via a network for communication.

[0032] The drive unit 40 is a device that performs various processes based on the results of biometric authentication by the multimodal authentication device 10. These various processes include, for example, controlling the opening and closing of an authentication gate (not shown) installed near the multimodal authentication device 10, sounding a buzzer from a speaker mounted on the multimodal authentication device 10, and turning on or flashing a light (for example, an LED (Light Emission Diode) light) mounted on the multimodal authentication device 10. However, the operations performed by the drive unit 40 are not limited to these.

[0033] <Internal configuration of the processor in a multimodal authentication device> Next, with reference to Figure 2, an example of the internal configuration of the processor 22 of the multimodal authentication device 10 will be described. Figure 2 is a block diagram showing an example of the internal configuration of the processor 22 of the multimodal authentication device 10.

[0034] The processor 22 implements the functions of the overall control unit 50 and the modal biometric authentication units 601 to 60n (n: a predetermined integer of 2 or more). The overall control unit 50 and the modal biometric authentication units 601 to 60n may be implemented by a single processor as shown in Figure 2, or they may be implemented as functions of separate processors. The number of modal biometric authentication units is not limited to two, and may be a number that matches the number of different types of biometric authentication that the multimodal authentication device 10 can perform. For example, if the multimodal authentication device 10 can perform three types of modal biometric authentication, namely facial recognition, fingerprint recognition, and voice recognition, then a total of three modal biometric authentication units are implemented by the processor 22.

[0035] The overall control unit 50 performs various processes related to biometric authentication for verifying the identity of the person to be authenticated, and determines and displays a guide method that presents the execution order of multiple types of modal biometric authentication to be preferentially executed for the person to be authenticated. The overall control unit 50 also obtains the processing results of each modal biometric authentication (e.g., the score and confidence level of the modal biometric authentication) from each of the modal biometric authentication units 601 to 60n, determines from the obtained processing results whether the person to be authenticated matches a person registered in the user information database 30 (i.e., whether identity verification has been successfully performed), displays the determination result on the display 21 (see Figure 8 or Figure 9), and / or performs operations such as opening and closing doors via the drive device 40. Functionally, the overall control unit 50 includes a system condition setting unit 51, an environment condition setting unit 52, an application condition setting unit 53, a user selection setting unit 54, a guide method determination unit 55, and a guide display unit 56.

[0036] The system condition setting unit 51 (an example of an acquisition unit) has built-in memory such as flash memory and sets the system condition information of the biometric authentication system 1 in its built-in memory during the initial setup of the biometric authentication system 1. After the initial setup, the system condition setting unit 51 retains the system condition information of the biometric authentication system 1 in its built-in memory. Once the system condition information is set during the initial setup of the biometric authentication system 1 and operation begins, no additions, changes, or deletions of the system condition information are made during operation. The system condition settings may also be stored in, for example, flash memory, HDD, SSD, or an external server device, and the system condition settings may be read when the device starts up.

[0037] The environmental condition setting unit 52 (an example of an acquisition unit) has built-in memory such as flash memory and acquires environmental condition information, which is information indicating the environment around the person being authenticated when biometric information is acquired (for example, brightness, sound pressure of noise signals), directly from the output of the sensor group 20, or calculates it based on that output and sets it in the built-in memory. The environmental condition setting unit 52 acquires or calculates environmental condition information each time biometric information is acquired in order to perform biometric authentication. Note that the environmental condition setting unit 52 may acquire environmental condition information separately from the acquisition of biometric information.

[0038] The usage condition setting unit 53 (an example of an acquisition unit) has built-in memory such as flash memory and acquires usage condition information entered by the user using the input device 25 and sets it in the built-in memory. Whenever the user's use of the biometric authentication system 1 changes, the usage condition setting unit 53 acquires the updated usage condition information from the input device 25 and sets it in the built-in memory.

[0039] The user selection and setting unit 54 (an example of an acquisition unit) has built-in memory such as flash memory and acquires user information entered by the user using the input device 25 and sets it in the built-in memory. In addition, if the user selection and setting unit 54 acquires personal identification information such as the user's ID by the user using the input device 25, it may acquire user information corresponding to that personal identification information from the user information database 30 and set it in the built-in memory. The user selection and setting unit 54 acquires user information and sets it in the built-in memory each time the user's preferences or characteristics change.

[0040] The guide method determination unit 55 determines a guide method that presents an execution order for prioritizing the authentication target from among multiple types of biometric authentication, based on the contents set in the system condition setting unit 51, the environment condition setting unit 52, the usage condition setting unit 53, and the user selection setting unit 54 (an example of ambient environment information). Details of the procedure for determining the guide method by the guide method determination unit 55 will be described later.

[0041] The guide display unit 56 outputs and displays the guide method determined by the guide method determination unit 55 on the display 21. The guide display unit 56 also outputs and displays a screen (see Figure 8 or Figure 9) on the display 21 showing the biometric authentication result determined by the overall control unit 50 based on the processing results of each of the modal biometric authentication units 601 to 60n.

[0042] The multimodal authentication device 10 is equipped with a number of modal biometric authentication units corresponding to the number of biometric authentications that can be performed, in order to support multiple types of biometric authentication. Specifically, the multimodal authentication device 10 is capable of performing each of n (n: a predetermined integer of 2 or more) types of modal biometric authentication, and accordingly has n modal biometric authentication units (specifically, modal biometric authentication units 601 to 60n) to correspond to each modal biometric authentication.

[0043] The modal biometric authentication unit 601 includes an environmental condition determination unit 611 and a score calculation unit 621. Similarly, the modal biometric authentication unit 60n includes an environmental condition determination unit 61n and a score calculation unit 62n. Therefore, although the description of the modal biometric authentication unit uses the modal biometric authentication unit 601 as an example, this description is also applicable to the configuration of other modal biometric authentication units (e.g., modal biometric authentication unit 60n).

[0044] The modal biometric authentication unit 601 performs modal biometric authentication processing using one or more biometric pieces of information acquired from the sensor group 20, and sends the result of the modal biometric authentication processing to the overall control unit 50.

[0045] The environmental condition determination unit 611 acquires environmental condition information (e.g., brightness, sound pressure of noise signals) from the output of the sensor group 20 or the environmental condition setting unit 52. Based on the acquired environmental condition information, the environmental condition determination unit 611 determines the reliability of the modal biometric authentication process performed by the modal biometric authentication unit 601.

[0046] The score calculation unit 621 acquires biometric information necessary for modal biometric authentication performed by the modal biometric authentication unit 601 from the output of the sensor group 20 (for example, facial imaging data or feature data for each feature point in the imaging data), and also reads the same type of biometric information (for example, facial imaging data or feature data for each feature point in the imaging data) registered in the user information database 30. The score calculation unit 621 calculates the result of the modal biometric authentication process (that is, a score indicating the likelihood of whether the person to be authenticated matches a person registered in the user information database 30) by comparing the acquired biometric information with the same type of biometric information registered in the user information database 30.

[0047] Figure 3 shows a table that defines examples of conditions for determining whether or not to use facial recognition. In Figure 3, "static system conditions" correspond to system condition information, "dynamic environment conditions" correspond to environment condition information, and "dynamic user conditions" correspond to user information. The guide method determination unit 55 (see Figure 2) determines whether or not it is appropriate to prioritize facial recognition for the person to be authenticated, based on at least one of the system condition information, environment condition information, and user information, and the contents shown in the table in Figure 3.

[0048] For example, the "static system conditions" specify items such as "camera installation position" and "sensor performance." The "camera installation position" indicates the height and distance of the face camera 201 (i.e., the distance from the person to be authenticated). Since the face camera 201 is installed during the initial setup of the biometric authentication system 1, whether or not face authentication should be prioritized based on the "camera installation position" in the system conditions information is determined during system design or equipment installation (initial setup). In other words, face authentication based on the "camera installation position" is determined to be prioritized if the image data or its feature data from the face camera 201 meets the design requirements for face authentication (e.g., usable distance and angle conditions).

[0049] Furthermore, "sensor performance" indicates the resolution and sensitivity of the face camera 201. Since the face camera 201 is installed during the initial setup of the biometric authentication system 1, whether or not face authentication should be prioritized based on the "sensor performance" in the system condition information is determined during the system design phase. In other words, face authentication based on "sensor performance" is determined to be prioritized if the image data or feature data from the face camera 201 meets the design requirements for face authentication (e.g., required resolution, minimum illumination conditions).

[0050] Next, the "dynamic environmental conditions" define the items "too dark / too bright" and "other lighting conditions." "Too dark / too bright" indicates a state where the brightness based on the luminance component of the image data captured by the face camera 201 is below a predetermined darkness threshold, or the brightness is greater than a predetermined brightness threshold. Whether or not to prioritize face recognition based on the "too dark / too bright" environmental condition information ("dynamic environmental conditions") will be determined periodically or when modal biometric authentication is performed. In other words, the necessity of face recognition due to "too dark / too bright" is determined periodically by referring to the history of brightness based on the image data of faces from past face recognition, or by referring to a table (not shown) that defines the brightness of the face and background parts in the image data captured by the face camera 201, or, if the face camera 201 is equipped with an illuminance sensor (not shown), the determination will be made using the output value of that illuminance sensor.

[0051] Furthermore, "other lighting conditions" refer to side lighting and backlighting around the position captured by the face camera 201. Whether or not to prioritize face recognition based on the "other lighting conditions" in the environmental conditions information ("dynamic environmental conditions") will be determined periodically or when modal biometric authentication is performed. In other words, the necessity of face recognition based on "other lighting conditions" is determined periodically by referring to the history of face lighting conditions during past face recognition, or to a table (not shown) that defines the relationship between face lighting conditions and the background.

[0052] Next, the "dynamic user conditions" specify the following items: "mask, sunglasses, bangs, facial expression, face direction, multiple people," "known to be twins," and "registration quality deficiency." When a person to be authenticated attempts to undergo facial recognition, if any of the following conditions are met in the image data captured by the face camera 201, such as wearing a "mask," wearing sunglasses, having part of the face obscured by "bangs," having a significantly different "facial expression" from normal (e.g., smiling), having a "face direction" that is not facing forward, or having "multiple people" in the image, the accuracy of facial recognition will deteriorate. For this reason, it is preferable to determine whether or not to prioritize facial recognition based on the "mask, sunglasses, bangs, facial expression, face direction, multiple people" in the user information ("dynamic user conditions") when modal biometric authentication is performed, based on the image data captured by the face camera 201. Furthermore, if facial recognition is performed when any or more of the following conditions are met in the image data captured by the face camera 201: the person is wearing a mask, sunglasses, part of their face is obscured by bangs, their facial expression is significantly different from their normal expression (e.g., smiling), their face is not facing forward, or multiple people are in the image, it is preferable to use not only the score but also the result of the facial recognition confidence assessment to determine whether or not facial recognition is necessary in subsequent instances.

[0053] Furthermore, if the condition "it is known that they are twins" is met by referring to the user information database 30, it becomes difficult to determine whether the person being authenticated is the older or younger brother, for example, in the case of male twins, and the accuracy of facial recognition deteriorates. The same applies to female twins. For this reason, it is preferable to determine whether or not to prioritize facial recognition based on the "it is known that they are twins" condition in the user information ("dynamic user conditions") when performing modal biometric authentication (especially when referring to the user information database 30) based on the image data captured by the face camera 201. In addition, if facial recognition is performed after the condition "it is known that they are twins" is met, it is preferable that the output also includes information indicating that the existence of twins is known.

[0054] Furthermore, if the "registration quality deficiency" condition is met by referencing the user information database 30, the accuracy of facial recognition will deteriorate due to the poor quality (reliability) of the registered information or data. For this reason, it is preferable to determine whether or not to prioritize facial recognition based on the "registration quality deficiency" of the user information ("dynamic user conditions") when performing modal biometric authentication (especially when referencing the user information database 30) based on the image data captured by the face camera 201. In addition, if facial recognition is performed after the "registration quality deficiency" condition is met, it is preferable that the output also includes information indicating that the quality of the facial image data or its feature data at the time of registration is insufficient.

[0055] Furthermore, if the multimodal authentication device 10 is positioned at a height where the person to be authenticated can naturally view the face camera 201 (i.e., without tilting their gaze upwards or downwards beyond a certain angle) at the location where the device is situated, or if the person to be authenticated is located a short distance away from the multimodal authentication device 10, the multimodal authentication device 10 may prioritize guiding facial authentication based on system condition information that includes the height information of the face camera 201. In this case, use cases include, but are not limited to, access control, reception in hospitals, roll call, and identity verification at events.

[0056] Figure 4 shows a table that defines examples of conditions for determining whether to use voice authentication. In Figure 4, "static system conditions" correspond to system condition information, "dynamic environment conditions" correspond to environment condition information, and "dynamic user conditions" correspond to user information. The guide method determination unit 55 (see Figure 2) determines whether it is appropriate to prioritize voice authentication for the person to be authenticated, based on at least one of the system condition information, environment condition information, and user information, and the contents shown in the table in Figure 4.

[0057] For example, the "static system conditions" specify items such as "microphone position," "sensor performance," and "acoustic environment." "Microphone position" indicates the distance of microphone 203 (i.e., the distance from the person to be authenticated). Since microphone 203 is installed during the initial setup of biometric authentication system 1, whether or not voice authentication should be prioritized based on the "microphone position" in the system conditions information is determined during system design or equipment installation (initial setup). In other words, voice authentication based on "microphone position" is determined to be prioritized if the sound data or its feature data from microphone 203 meets the design requirements for voice authentication (e.g., conditions such as usable distance).

[0058] Furthermore, "sensor performance" indicates information such as whether microphone 203 is a microphone array or a single microphone, sensitivity, and transmission conditions. Since microphone 203 is installed during the initial setup of biometric authentication system 1, whether or not voice authentication should be prioritized based on the "sensor performance" information of the system conditions will be determined during system design. In other words, voice authentication based on "sensor performance" should be prioritized if the sound data or its feature data from microphone 203 meets the design requirements for voice authentication (e.g., required sensitivity).

[0059] Furthermore, "acoustic environment" refers to the size of the space including the location of the person to be authenticated, or the conditions of reverberation around that location. Since microphone 203 is installed during the initial setup of biometric authentication system 1, whether or not voice authentication should be prioritized based on the "acoustic environment" in the system condition information is determined at the time of microphone 203 installation. In other words, voice authentication based on "acoustic environment" is determined to be prioritized if the sound data or its feature data from microphone 203 meets the design requirements for voice authentication (for example, the size of the space including the location of the person to be authenticated, or the conditions of reverberation around that location).

[0060] Next, the "dynamic environmental conditions" include an item called "background noise." "Background noise" indicates the sound pressure of the noise components (e.g., noise signals) around microphone 203 when microphone 203 picks up the voice spoken by the person being authenticated. Whether or not voice authentication should be prioritized based on the "background noise" in the environmental conditions information ("dynamic environmental conditions") will be determined periodically or when modal biometric authentication is performed. In other words, the necessity of voice authentication based on "background noise" is determined periodically by referring to a table (not shown) that defines the background noise components (e.g., sound pressure levels) when modal biometric authentication (voice authentication) is not being performed and when it is being performed.

[0061] Next, the "dynamic user conditions" specify the following items: "wearing a mask, etc.," "speech quality," and "knowing that the person is a twin." When a person to be authenticated wears a mask, etc. when attempting to undergo voice authentication, the voice spoken by the person to be authenticated is more easily attenuated by the mask, etc., and the level of sound pressure, etc., picked up by microphone 203 decreases, thereby degrading the accuracy of voice authentication. For this reason, it is preferable to determine whether or not to prioritize voice authentication based on the "mask, etc." in the user information ("dynamic user conditions") when modal biometric authentication is performed, based on the sound data picked up by microphone 203. Furthermore, if voice authentication is performed while "mask, etc." is being worn in the sound data picked up by microphone 203, it is preferable to use not only the score but also the user conditions (e.g., the result of the voice authentication confidence assessment, the likelihood of obtaining a score) to determine whether or not voice authentication is necessary in the next time.

[0062] Furthermore, "speech quality" refers to factors that affect the quality of speech, such as time, complexity, and the presence of simultaneous speakers. If the "speech quality," including these exemplified factors, is poor, the sound data picked up by microphone 203 will also be similarly negatively affected, resulting in a deterioration of the accuracy of voice authentication. For this reason, it is preferable to determine whether or not to prioritize voice authentication based on the "speech quality" of the user information ("dynamic user conditions") when performing modal biometric authentication, based on the sound data picked up by microphone 203. In addition, if voice authentication is performed using the sound data picked up by microphone 203, it is preferable to use not only the score but also the user conditions (e.g., the result of the voice authentication confidence assessment, the likelihood of obtaining a score) to determine whether or not voice authentication is necessary in subsequent instances.

[0063] Furthermore, if the condition "it is known that they are twins" is met by referring to the user information database 30, it becomes difficult to determine whether the person being authenticated is the older or younger brother, for example, in the case of male twins, and the accuracy of voice authentication deteriorates. The same applies to female twins. For this reason, it is preferable to determine whether or not to prioritize voice authentication based on the "it is known that they are twins" condition in the user information ("dynamic user conditions") when performing modal biometric authentication (especially when referring to the user information database 30) based on the sound data picked up by the microphone 203. In addition, if voice authentication is performed after the condition "it is known that they are twins" is met, it is preferable that the output also includes information indicating that the existence of twins is known.

[0064] Furthermore, if the person being authenticated is not looking in any direction towards the face camera 201, fingerprint camera 202, or microphone 203, or if biometric information is acquired in a way that covers a wide area centered on the location of the multimodal authentication device 10, the multimodal authentication device 10 may prioritize voice authentication based on the characteristics of the microphone 203, which only needs to capture the voice of the person being authenticated. In this case, use cases include, but are not limited to, identifying workers in factories, identifying speakers in meetings, and managing usage history such as entry to sports gyms.

[0065] Figure 5 shows a table that defines examples of conditions for determining whether to use fingerprint authentication. In Figure 5, "static system conditions" correspond to system condition information, "dynamic environment conditions" correspond to environment condition information, and "dynamic user conditions" correspond to user information. The guide method determination unit 55 (see Figure 2) determines whether it is appropriate to prioritize fingerprint authentication for the person to be authenticated, based on at least one of the system condition information, environment condition information, and user information, and the contents shown in the table in Figure 4.

[0066] For example, the "static system conditions" specify items such as "sensor installation location" and "feedback conditions." The "sensor installation location" indicates, for example, the height and distance (i.e., distance) of the fingerprint camera 202. Since the fingerprint camera 202 is installed during the initial setup of the biometric authentication system 1, whether or not fingerprint authentication should be prioritized based on the "sensor installation location" in the system conditions information is determined during system design or equipment installation (initial setup). In other words, fingerprint authentication based on the "sensor installation location" is determined to be prioritized if the image data or its feature data from the fingerprint camera 202 meets the design requirements for fingerprint authentication (e.g., usable distance and angle conditions).

[0067] Furthermore, the "feedback conditions" indicate whether or not to provide feedback (e.g., on-screen or audio notification) to the person being authenticated regarding the results of the fingerprint authentication process, and the content of that feedback. It can be difficult for the person being authenticated to know whether the fingerprint camera 202 has correctly captured their finger, and providing feedback in this case is expected to improve the accuracy of fingerprint authentication. Whether or not to prioritize fingerprint authentication based on the "feedback conditions" in the system condition information will be determined during system design. In other words, fingerprint authentication based on the "feedback conditions" will be determined to be prioritized if the presence or absence of feedback and notification of its content to the person being authenticated (see above) are met.

[0068] Next, the "dynamic environmental conditions" include the item "too dark / too bright". "Too dark / too bright" indicates a state where the brightness based on the luminance component of the image data captured by the fingerprint camera 202 is below a predetermined darkness threshold, or the brightness is greater than a predetermined brightness threshold. Whether or not fingerprint authentication should be prioritized based on the "too dark / too bright" environmental condition information ("dynamic environmental conditions") will be determined periodically or when modal biometric authentication is performed. In other words, the necessity of fingerprint authentication due to "too dark / too bright" is determined periodically by referring to the brightness history based on the image data of fingers during past fingerprint authentication, or by referring to a table (not shown) that defines the brightness of the finger portion and background portion in the image data captured by the fingerprint camera 202, or, if the fingerprint camera 202 is equipped with an illuminance sensor (not shown), the determination will be made using the output value of that illuminance sensor.

[0069] Next, the "dynamic user conditions" specify the following items: "shooting distance / focus," "skin surface condition of the fingers," and "number of fingers used." When a person to be authenticated attempts to undergo fingerprint authentication, it is necessary for the fingerprint camera 202 to maintain the correct shooting distance and to perform proper focusing in order to improve the accuracy of fingerprint authentication. For this reason, it is preferable to determine whether or not to prioritize fingerprint authentication based on the "shooting distance / focus" of the user information ("dynamic user conditions") when modal biometric authentication is performed, based on the image data captured by the fingerprint camera 202. Furthermore, if fingerprint authentication is performed using the image data captured by the fingerprint camera 202, it is preferable to determine whether or not subsequent fingerprint authentication is necessary not only based on the score but also on the fingerprint image quality conditions (e.g., finger size, resolution, focus level).

[0070] When a person to be authenticated attempts to undergo fingerprint authentication, if the condition of the skin surface of their fingers (e.g., moisture level, presence of dirt, presence of scratches) is not good, the accuracy of fingerprint authentication will deteriorate. Moisture level indicates whether the fingers are dry or wet. For this reason, it is preferable to determine whether or not to prioritize fingerprint authentication based on the "skin surface condition of fingers" in the user information ("dynamic user conditions") when performing modal biometric authentication, based on the image data captured by the fingerprint camera 202. Furthermore, if fingerprint authentication is performed using the image data captured by the fingerprint camera 202, it is preferable to determine whether or not subsequent fingerprint authentication is necessary not only based on the score but also on fingerprint image quality conditions such as moisture level, dirt, and scratches.

[0071] It is known that when a person to be authenticated attempts to undergo fingerprint authentication, there are algorithms that use one finger and algorithms that use multiple fingers. Therefore, it is preferable to determine whether or not to prioritize fingerprint authentication based on the "number of fingers used" in the user information ("dynamic user conditions") when performing modal biometric authentication based on the image data from the fingerprint camera 202, or when performing modal biometric authentication based on the image data from the fingerprint camera 202 (especially when referring to the user information database 30). When performing fingerprint authentication using the image data from the fingerprint camera 202, it is preferable to determine whether to use one finger or multiple fingers, depending on the algorithm used.

[0072] Furthermore, if the person being authenticated is not looking in any direction towards the face camera 201, fingerprint camera 202, or microphone 203, or if the sensor group 20 is installed at a relatively low position relative to the person being authenticated, the multimodal authentication device 10 may prioritize fingerprint authentication based on the characteristics of the fingerprint camera 202, which only needs to capture the person's finger. In this case, use cases include, but are not limited to, driver authentication, hotel check-in / check-out / entry / exit management, locker locking, and ATM use.

[0073] Figure 6 shows a first display example of a modal biometric authentication guide method that transitions over time. Figure 7 shows a second display example of a modal biometric authentication guide method that transitions over time. The guide method determination unit 55 (see Figure 2) determines a guide method that presents an execution order for prioritizing the authentication target from among multiple types of biometric authentication, based on the contents set in the system condition setting unit 51, the environment condition setting unit 52, the usage condition setting unit 53, and the user selection setting unit 54 (an example of ambient environment information). The guide display unit 56 outputs and displays the guide method determined by the guide method determination unit 55 on the display 21.

[0074] The first example of the guidance method, as a sequence to prioritize execution for the person being authenticated, guides (directs) in the following order, for example, as shown in Figure 6: Guide screen WD1 specialized for facial recognition → Guide screen WD2 specialized for voice recognition → Guide screen WD3 specialized for fingerprint recognition. The screen transitions (switches) between guide screens WD1, WD2, and WD3 are performed by the guide display unit 56 when predetermined switching conditions are met, as will be described later. It goes without saying that the execution order is not limited to the example in Figure 6, but is merely one example.

[0075] The facial recognition guide screen WD1 displays, for example, a portion of a full-body image showing a person's face, the name of the modal biometric authentication (specifically the words "Facial Recognition"), and the message to the person to be authenticated, "Please hold your face up." Note that the message to the person to be authenticated is not limited to "Please hold your face up"; any message instructing the person undergoing facial recognition to position their face so that it is visible to the facial camera 201 is acceptable, and so on.

[0076] The voice authentication guide screen WD2 displays, for example, an image showing a person speaking, the name of the modal biometric authentication (specifically the words "Voice Authentication"), and a message to the person being authenticated: "Please say the following words: 'aiueo'." Note that the phrase to be spoken by the person being authenticated is not limited to "aiueo," and the same applies to the following.

[0077] The fingerprint authentication guide screen WD3 displays, for example, an illustration of a fingerprint camera 202 capturing a person's finger, the name of the modal biometric authentication (specifically the words "Fingerprint Authentication"), and a message to the user being authenticated: "Turn your palm upwards and place your finger over the guideline. Please wait." Note that the message to the user being authenticated is not limited to the above and the same applies below.

[0078] The second example of the guidance method, as an order to prioritize execution for the person being authenticated, guides the user in the following order, for example, as shown in Figure 7: Guide screen WD1H, which highlights only the facial recognition portion of the three types of modal biometric authentication (facial recognition, voice recognition, and fingerprint recognition) and grays out the other two modal biometric authentication portions; Guide screen WD2H, which highlights only the voice recognition portion of the same three types of modal biometric authentication and grays out the other two modal biometric authentication portions; and Guide screen WD3H, which highlights only the fingerprint recognition portion of the same three types of modal biometric authentication and grays out the other two modal biometric authentication portions. The screen transitions (switches) between guide screens WD1H, WD2H, and WD3H are performed by the guide display unit 56 when predetermined switching conditions are met, as will be described later. It goes without saying that the execution order is not limited to the example in Figure 7, but is merely one example.

[0079] The WD1H guide screen highlights, for example, the name of the modal biometric authentication (specifically, the text "Facial Recognition") and the message to the user, "Please hold your face up," while graying out the name of the modal biometric authentication and the message to the user corresponding to voice recognition and fingerprint recognition.

[0080] The WD2 guide screen highlights, for example, the name of the modal biometric authentication (specifically, the text "Voice Authentication") and the message to the user being authenticated, "Please say the following words: 'aiueo'," while graying out the name of the modal biometric authentication and the message to the user corresponding to facial recognition and fingerprint authentication, respectively.

[0081] The WD3 guide screen highlights, for example, the name of the modal biometric authentication method (specifically, the words "fingerprint authentication") and the message to the user being authenticated, "Turn your palm upwards and place your finger over the guideline. Please wait." It grays out the names of the modal biometric authentication methods and the messages to the user corresponding to facial recognition and voice recognition, respectively.

[0082] Figure 8 shows an example of a display screen for successful authentication. Figure 9 shows an example of a display screen for failed authentication. The guide display unit 56 outputs and displays a screen (see Figure 8 or Figure 9) on the display 21 showing the result of biometric authentication determined by the overall control unit 50 based on the processing results of each of the modal biometric authentication units 601 to 60n.

[0083] The authentication success screen WD4 in Figure 8 displays a message indicating that biometric authentication was successful (e.g., "OK") based on the processing results of each of the modal biometric authentication units 601 to 60n, as determined by the overall control unit 50; the name of the person to be authenticated (e.g., "A Maru B Taro") identified by referencing the user information database 30; and the type of modal biometric authentication actually used (MD1). Here, the type of modal biometric authentication actually used (MD1) is exemplified by the two types highlighted: facial recognition and fingerprint recognition, but it is not limited to these. The grayed-out display indicates that voice recognition was not used.

[0084] On the other hand, the authentication failure screen WD5 in Figure 9 displays a message indicating that biometric authentication failed based on the processing results of each of the modal biometric authentication units 601 to 60n, a message indicating that the name of the person to be authenticated could not be identified by referring to the user information database 30, and the type of modal biometric authentication actually used, MD1. Here, the type of modal biometric authentication actually used, MD1, is illustrated by the two types highlighted, face recognition and fingerprint recognition, but is not limited to these. The grayed-out display indicates that voice recognition was not used.

[0085] <System Operating Procedure> Next, with reference to Figures 10 to 14, an example of the operation procedure of the multimodal authentication device according to this embodiment will be described. Figure 10 is a flowchart showing an example of the overall operation procedure of the multimodal authentication device 10. Figure 11 is a flowchart showing an example of the operation procedure of the first subroutine for determining the guide method. Figure 12 is a flowchart showing an example of the operation procedure of the second subroutine for determining the guide method. Figure 13 is a flowchart showing an example of the operation procedure of the subroutine for switching the guide method. Figure 14 is a flowchart showing an example of the operation procedure of the subroutine for determining the priority of modal biometric authentication. Each process (step) in Figures 10 to 14 is mainly executed by the processor 22 of the multimodal authentication device 10 (see Figure 2).

[0086] In Figure 10, the processor 22 acquires system condition information for the biometric authentication system 1 during initial setup or startup, and temporarily stores it in memory 23, etc. (Step St1). The processor 22 acquires environmental condition information (e.g., brightness, sound pressure of noise signals) indicating the environment around the person being authenticated, either directly from the output of the sensor group 20 or calculated based on that output, and temporarily stores it in memory 23, etc. (Step St2). The processor 22 acquires usage condition information (e.g., information indicating cash withdrawal, deposit balance inquiry, or cash deposit) entered by the person being authenticated using the input device 25, and temporarily stores it in memory 23, etc. (Step St3). The processor 22 also acquires user information (e.g., user ID) entered by the person being authenticated using the input device 25, and temporarily stores it in memory 23, etc. (Step St3).

[0087] Based on the various information set in each of steps St1 to St3, the processor 22 determines a guide method for presenting an execution order among multiple types of biometric authentication to be prioritized for the person being authenticated (step St4). Detailed examples of the operation procedure in step St4 will be described later with reference to Figures 11 and 12, respectively.

[0088] The processor 22 outputs the guidance method determined in step St4 (for example, the priority and display order of each modal biometric authentication) to the display 21 and displays it, thereby initiating guidance on which modal biometric authentication should be prioritized for the person being authenticated (step St5A). In parallel with step St5A, the processor 22 also executes the processing of each modal biometric authentication for the person being authenticated according to the guidance method determined in step St4 (step St5B). Using the processing results of each modal biometric authentication executed in step St5B, the processor 22 makes a determination (overall determination) as to whether or not the person being authenticated is a person who has been previously registered in the user information database 30 (step St6B).

[0089] After starting the guide in step St5A, the processor 22, if it determines that a predetermined switching condition is met based on the processing result of modal biometric authentication performed in step St5B, for example, switches the guide method (i.e., in what order modal biometric authentication to prioritize for the person being authenticated at this time) and outputs and displays it on the display 21 (step St6A). Step St6A is not executed unless the predetermined switching condition is met, and the display of the current guide method is maintained. The processor 22 determines whether the overall judgment (see step St6B) has been completed (step St7). If it is determined that the overall judgment (see step St6B) has not been completed (step St7, NO), the processor 22 executes the process in step St5A.

[0090] If the processor 22 determines that the overall judgment (step St6B) is complete (step St7, YES), it terminates the display on the display 21 of the guide method determined in step St4 or the guide method switched in step St6A (step St8). Upon termination of the guide method display, the processor 22 generates a screen (see Figure 8 or Figure 9) showing the authentication result for the person to be authenticated determined in step St6B, and outputs and displays it on the display 21.

[0091] Specifically, if the processor 22 determines that the person to be authenticated matches a person registered in the user information database 30 (i.e., the person to be authenticated is the real person), it displays the authentication success screen WD4 shown in Figure 8 on the display 21 (step St9). On the other hand, if the processor 22 determines that the person to be authenticated does not match a person registered in the user information database 30 (i.e., the person to be authenticated is not the real person), it displays the authentication failure screen WD5 shown in Figure 9 on the display 21 (step St10). This completes the overall operation procedure of the multimodal authentication device 10.

[0092] Here, we will describe in detail the first subroutine that illustrates the process of determining the guiding method in step St4, with reference to Figure 11.

[0093] In Figure 11, the processor 22 obtains information about the surrounding environment of the person to be authenticated (for example, the sound pressure of noise signals around the person to be authenticated, ambient brightness, the current time, or information about the installation location of the multimodal authentication device 10) by referring to memory 23 or the like (step St11). Based on the surrounding environment information obtained in step St11, the processor 22 determines the priority of each modal biometric authentication (step St12). Details of step St12 will be described later with reference to Figure 14, but for example, if the surrounding environment is bright, facial recognition may be prioritized, or if the environment is dark, voice recognition may be prioritized. Also, if the surrounding environment is noisy, voice recognition may be given a lower priority, or if the installation location requires fast processing of modal biometric authentication, facial recognition may be given a higher priority. The processor 22 determines the display order of the guides for each modal biometric authentication based on the priority of each modal biometric authentication determined in step St12 (for example, displaying each modal biometric authentication in order of priority as shown in Figure 6, or displaying all guides but highlighting the high-priority guides and graying out the low-priority guides as shown in Figure 7) (step St13).

[0094] The processor 22 determines and sets switching conditions (step St14) for switching (changing) the modal biometric authentication guide currently displayed as having priority execution (for example, any of the single modal biometric authentication guide screens WD1, WD2, WD3 in Figure 6, or any of the highlighted modal biometric authentication guide screens WD1H, WD2H, WD3H in Figure 7) to another modal biometric authentication guide. Examples of switching conditions include maintaining the display of the currently displayed modal biometric authentication guide until the biometric information of the person being authenticated is acquired, or switching to the next modal biometric authentication guide after a certain period of time has elapsed, regardless of whether the biometric information of the person being authenticated has been acquired or not. It goes without saying that the switching conditions are not limited to these.

[0095] Here, a second subroutine that illustrates in detail the process of determining the guiding method in step St4 will be described with reference to Figure 12.

[0096] In Figure 12, the processor 22 obtains user information of the person to be authenticated (for example, the user ID which is the identification number of the person to be authenticated) or usage condition information (for example, the service that the person to be authenticated wants to use, such as cash transfer, cash deposit, or deposit balance inquiry) by referring to memory 23, etc. (step St21). Based on the user information or usage condition information obtained in step St21, the processor 22 determines the priority of each modal biometric authentication (step St22). For example, if the usage condition information is cash transfer, it is determined that fingerprint authentication of the person to be authenticated is mandatory, and other modal biometric authentication (facial recognition, voice recognition) is not necessarily mandatory. Based on the priority of each modal biometric authentication determined in step St22, the processor 22 determines the display order of the guides for each modal biometric authentication (for example, displaying each modal biometric authentication in order of decreasing priority as in Figure 6, or displaying all guides but highlighting the guides with decreasing priority and graying them out as in Figure 7) (step St23). Furthermore, in step St23, the processor 22 may decide to display the modal biometric authentication required in step St22 at the beginning of the guide method.

[0097] The processor 22 determines and sets switching conditions (step St24) for switching (changing) the modal biometric authentication guide currently displayed as being prioritized for execution (for example, any of the single modal biometric authentication guide screens WD1, WD2, WD3 in Figure 6, or any of the highlighted modal biometric authentication guide screens WD1H, WD2H, WD3H in Figure 7) to another modal biometric authentication guide. Examples of switching conditions include maintaining the display of a modal biometric authentication guide determined to be mandatory until the corresponding biometric information of the person being authenticated is obtained. It goes without saying that the switching conditions are not limited to these.

[0098] Here, a subroutine that illustrates the process of switching the guiding method in step St6A will be explained in detail with reference to Figure 13.

[0099] In Figure 13, the processor 22 determines, by referring to memory 23, etc., whether or not each of the two or more modal biometric authentication units has performed the corresponding modal biometric authentication process (step St31). If it is determined that each of the two or more modal biometric authentication units has not performed the corresponding modal biometric authentication process (step St31, NO), the processor 22 repeats the process in step St31 until it is determined that each of the two or more modal biometric authentication units has performed the corresponding modal biometric authentication process.

[0100] If the processor 22 determines that each of the two or more modal biometric authentication units has performed the corresponding modal biometric authentication process (step St31, YES), it obtains the score for each modal biometric authentication (step St32). Based on the score for each modal biometric authentication obtained in step St32, the processor 22 determines the display order of the guides for each modal biometric authentication (for example, if the score is lower than a predetermined NG threshold, it displays a prompt to obtain biometric information corresponding to that modal biometric authentication again, or if the score is higher than a predetermined OK threshold, it displays the guides for other modal biometric authentications) (step St33).

[0101] The processor 22 determines and sets switching conditions (step St34) for switching (changing) the modal biometric authentication guide currently displayed as the priority to be executed (for example, any of the single modal biometric authentication guide screens WD1, WD2, WD3 in Figure 6, or any of the highlighted modal biometric authentication guide screens WD1H, WD2H, WD3H in Figure 7) to another modal biometric authentication guide. Examples of switching conditions include maintaining the display of a modal biometric authentication guide determined to be mandatory until the corresponding biometric information of the person being authenticated is obtained. It goes without saying that the switching conditions are not limited to these.

[0102] Here, the details of the priority determination of each modal biometric authentication based on the surrounding environment information in step St12 will be explained with reference to Figure 14. Note that the priority determination of each modal biometric authentication based on the surrounding environment information in step St12 is not limited to what is shown in Figure 14.

[0103] In Figure 14, the processor 22 determines whether or not the usage condition information has been set in memory 23 or the like (step St41). If the processor 22 determines that the usage condition information has not been set (step St41, NO), it sets the display order of the modal biometric authentication guide to a predetermined default, or a guide display order determined based on other conditions (e.g., system condition information, environmental condition information, user information) (step St42). After step St42, the process in Figure 14 ends.

[0104] On the other hand, if the processor 22 determines that usage condition information has been set (step St41, YES), it determines whether the primary (i.e., the first to guide) modal biometric authentication guide has been set by referring to memory 23, etc. (step St43).

[0105] If the processor 22 determines that the primary modal biometric authentication guide has not been set (step St43, NO), it decides to set the primary modal biometric authentication guide according to environmental condition information, for example (step St44). For example, if the brightness around the person to be authenticated is within an appropriate range stored in memory 23, etc. (for example, from the darkness threshold (see above) to the brightness threshold (see above)), the processor 22 decides to use the facial recognition guide as the primary modal biometric authentication guide (step St45). For example, if the sound pressure of the noise signal around the person to be authenticated (background noise) is less than a predetermined sound pressure threshold stored in memory 23, etc., the processor 22 decides to use the voice recognition guide as the primary modal biometric authentication guide (step St46). Furthermore, if, for example, the brightness around the person to be authenticated is not within a predetermined range indicating an appropriate range, and the sound pressure of the noise signal around the person to be authenticated (background noise) is greater than a predetermined sound pressure threshold, the processor 22 decides to use the fingerprint authentication guide as the primary modal biometric authentication guide (step St47).

[0106] On the other hand, if the processor 22 determines that the primary modal biometric authentication guide has been set (step St43, YES), it determines whether the primary modal biometric authentication is facial recognition, voice recognition, or fingerprint recognition (step St48).

[0107] If the processor 22 determines that the primary modal biometric authentication is facial recognition (step St48, face), and the sound pressure of the ambient noise surrounding the person to be authenticated is less than a predetermined sound pressure threshold stored in memory 23, etc., it decides to use the voice authentication guide as the guide for the secondary (i.e., the second after the primary) modal biometric authentication (step St49). If the processor 22 determines that the primary modal biometric authentication is facial recognition (step St48, face), and the sound pressure of the ambient noise surrounding the person to be authenticated is greater than a predetermined sound pressure threshold stored in memory 23, etc., it decides to use the fingerprint authentication guide as the guide for the secondary (i.e., the second after the primary) modal biometric authentication (step St50).

[0108] If the processor 22 determines that the primary modal biometric authentication is voice authentication (step St48, voice), and the ambient light around the person being authenticated is within an appropriate range stored in memory 23, etc. (for example, from the darkness threshold (see above) to the brightness threshold (see above)), then it decides to use the face authentication guide as the guide for the secondary (i.e., the second after the primary) modal biometric authentication (step St51). If the processor 22 determines that the primary modal biometric authentication is voice authentication (step St48, face), and the ambient light around the person being authenticated is not within an appropriate range stored in memory 23, etc. (for example, from the darkness threshold (see above) to the brightness threshold (see above)), then it decides to use the fingerprint authentication guide as the guide for the secondary (i.e., the second after the primary) modal biometric authentication (step St52).

[0109] If the processor 22 determines that the primary modal biometric authentication is fingerprint authentication (step St48, fingerprint), and the ambient light around the person being authenticated is within an appropriate range stored in memory 23, etc. (for example, from the darkness threshold (see above) to the brightness threshold (see above)), then it decides to use the facial recognition guide as the guide for the secondary (i.e., the second after the primary) modal biometric authentication (step St53). If the processor 22 determines that the primary modal biometric authentication is fingerprint authentication (step St48, fingerprint), and the sound pressure of the ambient noise signal around the person being authenticated (background noise) is less than a predetermined sound pressure threshold stored in memory 23, etc., then it decides to use the voice recognition guide as the guide for the secondary (i.e., the second after the primary) modal biometric authentication (step St54).

[0110] As described above, the authentication guide device according to this embodiment (for example, the multimodal authentication device 10) comprises one or more acquisition units (for example, a system condition setting unit 51, an environmental condition setting unit 52, and an application condition setting unit 53) that acquire ambient condition information when acquiring one or more biometric information of a person (for example, the person to be authenticated) used for multiple types of biometric authentication (modal biometric authentication); a guide determination unit (for example, a guide method determination unit 55) that determines a guide method for presenting an execution order to be preferentially executed for the person among multiple types of biometric authentication based on the ambient condition information; and a guide display unit 56 that displays the guide method on a display unit (for example, a display 21). As a result, the multimodal authentication device 10 can adaptively present to the user an effective authentication guide appropriate to the ambient condition information according to the usage status of authentication (for example, modal biometric authentication) (for example, ambient condition information of the person to be authenticated), and can support the efficiency of authentication.

[0111] Furthermore, the surrounding condition information includes system condition information regarding the placement of the authentication guide device (e.g., multimodal authentication device 10), environmental condition information around the person (e.g., person to be authenticated), and biometric authentication application condition information. The guide determination unit (e.g., guide method determination unit 55) determines the guide method based on at least one of the system condition information, environmental condition information, and application condition information. As a result, the multimodal authentication device 10 can present the person to perform modal biometric authentication that is suitable for the usage conditions around the person to be authenticated, in accordance with at least one of the system condition information, which is difficult to change midway through the process, the environmental condition information, which is easy to change each time measurements are taken by the sensor group 20, and the application condition information, which is likely to differ in purpose for each person to be authenticated.

[0112] Furthermore, the surrounding conditions information also includes user information that reflects the person's preferences or characteristics. This allows the multimodal authentication device 10 to present the person being authenticated with a modal biometric authentication method that is suitable for that person, in accordance with the user information that differs for each person being authenticated in terms of their individual preferences or characteristics.

[0113] Furthermore, the guide determination unit (for example, the guide method determination unit 55) determines the guide method based on at least two of the following: system condition information regarding the placement of the authentication guide device (for example, the multimodal authentication device 10), environmental condition information surrounding the person (for example, the person to be authenticated), biometric authentication usage condition information, and user information that describes the person's preferences or characteristics. As a result, the multimodal authentication device 10 can present the person to be authenticated with the execution of modal biometric authentication that is suitable for the usage situation, combining at least two of the following: system condition information whose contents are difficult to change midway through the process, environmental condition information that can be easily changed each time measurements are taken by the sensor group 20, usage condition information whose purpose tends to differ for each person to be authenticated, and user information whose personal preferences or characteristics differ for each person to be authenticated.

[0114] Furthermore, environmental condition information has the characteristic of fluctuating over time and is a measured value of environmental parameters around the authentication guide device (e.g., the multimodal authentication device 10). As a result, the multimodal authentication device 10 can appropriately guide the authentication process to perform biometric authentication smoothly and with minimal burden on the user, by taking into account the environmental condition information, which tends to fluctuate each time the user is acquiring biometric information for biometric authentication.

[0115] Furthermore, system condition information has the characteristic of not changing over time and is fixed equipment condition information determined when the authentication guide device (e.g., multimodal authentication device 10) is installed. As a result, the multimodal authentication device 10 can appropriately guide the user to perform smooth biometric authentication with minimal burden on the user by taking into account the fixed equipment condition information determined when the multimodal authentication device 10 is installed (e.g., the position or performance of the face camera 201 and fingerprint camera 202, the position or performance of the microphone 203).

[0116] Furthermore, the usage condition information is information indicating the intended use of biometric authentication by the authentication guide device (e.g., the multimodal authentication device 10). This allows the multimodal authentication device 10 to appropriately guide the user in performing smooth biometric authentication with minimal burden on the user, taking into account the intended use of the authentication.

[0117] Furthermore, the multiple types of biometric authentication include at least two of the following: facial recognition, fingerprint recognition, and voice recognition of the person (e.g., the person being authenticated). The biometric information includes the person's facial image or features based on that facial image corresponding to facial recognition, the person's fingerprint image or features based on that fingerprint image corresponding to fingerprint recognition, and the person's voice or features based on that voice corresponding to voice recognition. As a result, the multimodal authentication device 10 can perform authentication using at least two of the person being authenticated's face, fingerprints, and voice in a combined manner, making it less susceptible to damage from impersonation by others.

[0118] Furthermore, the guide determination unit (e.g., the guide method determination unit 55) determines switching conditions for switching the execution order within the guide method. When the switching conditions are met, the guide display unit 56 switches the execution order within the guide method that is displayed on the display unit (e.g., the display 21) at the time the switching conditions are met and displays it on the display unit. As a result, the multimodal authentication device 10 does not always display the same modal biometric authentication, but rather presents the person being authenticated with the next modal biometric authentication to be performed each time the switching conditions are met.

[0119] Furthermore, the authentication guide device (e.g., multimodal authentication device 10) further includes one or more authentication units (e.g., modal biometric authentication units 601-60n) that perform at least one of multiple types of biometric authentication while the guide method is being displayed on the display unit (e.g., display 21). This allows the multimodal authentication device 10 to perform modal biometric authentication in parallel after acquiring the biometric information of the person to be authenticated, while presenting a guide on the execution order of modal biometric authentication that is tailored to the usage environment around the person to be authenticated.

[0120] Furthermore, the guide determination unit (for example, the guide method determination unit 55) changes the execution order based on the biometric authentication score. This allows the multimodal authentication device 10 to present the user with a guide for modal biometric authentication that should be prioritized based on the score of the modal biometric authentication performed, in real time.

[0121] Furthermore, the guide display unit 56 displays the final results of multiple types of biometric authentication of a person (e.g., the person being authenticated) on the display unit (e.g., display 21) along with the names of the multiple types of biometric authentication. This allows the multimodal authentication device 10 to visually present the final result of modal biometric authentication (i.e., whether authentication was successful or unsuccessful) to the person being authenticated, and to provide appropriate feedback to the person being authenticated.

[0122] While embodiments have been described above with reference to the attached drawings, this disclosure is not limited to such examples. It is clear to those skilled in the art that various modifications, alterations, substitutions, additions, deletions, and equivalents can be conceived within the scope of the claims, and these are also understood to fall within the technical scope of this disclosure. Furthermore, the components of the embodiments described above can be combined in any way without departing from the spirit of the invention. [Industrial applicability]

[0123] This disclosure is useful as an authentication guide device and authentication guide method that adaptively presents effective authentication guides to users according to their authentication usage, thereby supporting the efficiency of authentication. [Explanation of symbols]

[0124] 1. Biometric authentication system 10. Multimodal Authentication Device 20 sensor group 21 displays 22 processors 23 memory 24 Communication IF 25 Input Devices 30. User Information Database 40 Drive unit 50 Overall control unit 51 System Condition Setting Section 52 Environmental Condition Setting Unit 53. Usage Condition Setting Section 54 User Selection Setting Section 55 Guide Method Determination Section 56 Guide display section 201 Face Camera 202 Fingerprint Camera 203 Mike 601, 60n Modal Processing Unit 611, 61n Environmental condition determination section 621, 62n score calculation unit

Claims

1. One or more acquisition units that acquire information on the surrounding environmental conditions when acquiring one or more biometric information of a person used for multiple types of biometric authentication, A guide determination unit that determines a guide method for presenting the person with a priority execution order for performing the multiple types of biometric authentication based on the aforementioned environmental condition information, A guide display unit that displays the aforementioned guide method on a display unit, The system includes an integrated determination unit that performs authentication determination of the person using the processing results of the multiple types of biometric authentication performed according to the execution order of the displayed guide method, The guide determination unit determines the switching conditions for switching the execution order in the guide method, When the switching condition is met, the guide display unit switches the execution order of the guide method displayed on the display unit when the switching condition is met and displays it on the display unit. Authentication guide device.

2. The guide determination unit determines the guiding method based on at least two of the following: system condition information relating to the arrangement of the authentication guide device, environmental condition information surrounding the person, application condition information for the biometric authentication, and user information relating to the person's preferences or characteristics. The authentication guide device according to claim 1.

3. The aforementioned environmental condition information has characteristics that change over time and is a measured value of environmental parameters around the authentication guide device. The authentication guide device according to claim 1.

4. The aforementioned system condition information has the characteristic of not changing over time and is fixed equipment condition information determined when the authentication guide device is installed. The authentication guide device according to claim 2.

5. The aforementioned usage condition information is information indicating the use of biometric authentication by the authentication guide device. The authentication guide device according to claim 2.

6. The aforementioned multiple types of biometric authentication include at least two of the following: facial recognition, fingerprint recognition, and voice recognition of the person. The biometric information includes a facial image of the person corresponding to the facial recognition or features based on that facial image, a fingerprint image of the person corresponding to the fingerprint recognition or features based on that fingerprint image, and a voice of the person corresponding to the voice recognition or features based on that voice. The authentication guide device according to claim 1.

7. The system further comprises one or more authentication units that perform at least one of the multiple types of biometric authentication while the guiding method is being displayed on the display unit, The authentication guide device according to claim 1.

8. The guide determination unit changes the execution order based on the biometric authentication score. The authentication guide device according to claim 7.

9. The guide display unit displays the final result of the multiple types of biometric authentication of the person, along with the names of the multiple types of biometric authentication, on the display unit. The authentication guide device according to claim 7.

10. An authentication guiding method performed by an authentication guiding device, A step of acquiring information on the surrounding environmental conditions when acquiring one or more biometric data of a person to be used for multiple types of biometric authentication, A step of determining a guide method that, based on the aforementioned environmental condition information, presents the person with an execution order for prioritizing the execution of the multiple types of biometric authentication; The steps include displaying the aforementioned guide method on the display unit, The process includes a step of performing an authentication determination of the person using the processing results of the multiple types of biometric authentication performed according to the execution order of the displayed guide method, In determining the guiding method, the switching conditions for switching the execution order in the guiding method are determined. In the display of the guide method, when the switching condition is satisfied, the execution order in the guide method displayed on the display unit when the switching condition is satisfied is switched and displayed on the display unit. Authentication guide and method.

Images