Face authentication device, face authentication method, face authentication program, and security system

Abstract

To automatically detect occurrence of an abnormal security situation for an authentication object person when confirming that the authentication object person has predetermined authority based on facial authentication results.SOLUTION: A face authentication device for acquiring information about an authentication object person includes: a sensor unit for capturing images around the face authentication device; an authentication unit for executing facial authentication processing of a person captured by the sensor unit to authenticate an authentication permission person having predetermined authority; and an abnormal situation detection unit for executing abnormal situation detection processing for analyzing the image captured by the sensor unit and automatically detecting that the face-authenticated authentication permission person is in an abnormal situation.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a face recognition device, a face recognition method, and a face recognition program that execute face recognition processing to check whether or not a specific authority exists, and a security system that uses face recognition results for security purposes. [Background technology]

[0002] Conventionally, doors that can be locked and unlocked based on the results of face authentication have been used. For example, Patent Document 1 discloses a system that performs face authentication by comparing a face image of a user captured by a camera with a preregistered face image that is permitted to enter and exit, and unlocks the electronic lock of the door when the authentication is successful.

[0003] When a facility that has doors that are unlocked according to the results of face recognition is to be secured, it is necessary to prevent malicious people from illegally unlocking the doors and entering the facility. For example, Patent Document 2 discloses a system that guides the person to be authenticated to move their gaze randomly during authentication processing, and requires the person to follow the guidance as an authentication condition. If the person does not move their gaze as instructed, the face is not recognized and the door is not unlocked, so that malicious people can be prevented from unlocking the door through unauthorized authentication using photographs or printed matter. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2014-234637 A [Patent Document 2] JP 2006-85226 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the above conventional technology, if a malicious person threatens a user who is permitted to enter and exit through a door to have his / her face authenticated, the user may move his / her eyes as instructed, be authenticated, and the door may be unlocked. For example, a method may be considered in which the user is instructed in advance to make a motion or eye movement that signals an abnormality during face authentication so that the occurrence of an abnormality can be detected by a face authentication camera, but there is a possibility that the user who feels in danger may not be able to calmly respond.

[0006] The present disclosure has been made in consideration of the conventional technologies including the above-mentioned problems, and one of its objectives is to provide a face recognition device, a face recognition method, a face recognition program, and a security system that can automatically detect the occurrence of an abnormality when checking the presence or absence of a specified authority based on the results of face recognition. [Means for solving the problem]

[0007] The face recognition device according to the present disclosure is a face recognition device that obtains information about a person to be authenticated, and includes a sensor unit that captures an image of the surroundings of the face recognition device, an authentication unit that performs face recognition processing of the person captured by the sensor unit and authenticates an authorized person who has specified authority, and an abnormal situation detection unit that analyzes the image captured by the sensor unit and performs abnormal situation detection processing to automatically detect whether the authorized person whose face has been authenticated is in an abnormal situation.

[0008] In the above configuration, the abnormal situation detection unit may execute a process to detect a specified object regarded as a weapon, the arm of another person, and another person in close contact from the captured image, and determine that the authenticated person is in an abnormal situation based on the detection results.

[0009] In the aforementioned configuration, the abnormal situation detection section may execute the detection process on a predetermined area around a face of the authentication authorized person appearing in the captured image.

[0010] In the above configuration, the abnormal situation detection unit may execute a process of detecting a predetermined posture, a predetermined movement, and a predetermined facial expression of the authentication authorized person from the captured image, and determine that the authentication authorized person is in an abnormal situation based on the detection results.

[0011] In the above configuration, the predetermined posture may include the authentication authorized person's face being tilted, the predetermined movement may include the authentication authorized person's eyes wandering, and the predetermined facial expression may include the authentication authorized person's facial expression of fear.

[0012] In the above configuration, the image processing device may further include an abnormal situation response processing unit that executes an abnormal situation response process including a process of reporting an abnormal situation to a specified external device when the abnormal situation detection unit detects an abnormal situation and a process of transmitting information about a person appearing in the captured image to the external device.

[0013] In the aforementioned configuration, the information transmitted to the external device in the emergency response process may include a captured image of a person appearing in the captured image together with the authentication authorized person, and information regarding a current location of the person.

[0014] The facial recognition method disclosed herein is a facial recognition method executed by a facial recognition device that acquires information about a person to be authenticated, and includes a step of executing a facial recognition process that detects a face from an image captured around the facial recognition device and determines whether the face is the face of an authorized person who has specified authority, and a step of executing an abnormal situation detection process that analyzes the captured image and automatically detects whether the authorized person whose face has been authenticated is in an abnormal situation.

[0015] The facial recognition program disclosed herein is a facial recognition program for obtaining information about a person to be authenticated, and causes a computer to execute a facial recognition process that detects a face from an image captured around a facial recognition device and determines whether the face is the face of an authorized person who has specified authority, and a process that analyzes the captured image and executes an abnormal situation detection process that automatically detects whether the authorized person whose face has been authenticated is in an abnormal situation.

[0016] The security system according to the present disclosure includes a facial recognition device having the above-described configuration, and a mechanical security sensor used for mechanical security performed when the facility in which the facial recognition device is installed is unoccupied, and is a security system that deactivates the mechanical security based on the authentication result of the facial recognition device, and when the abnormality detection unit detects an abnormality, the mechanical security is seemingly deactivated while continuing to use the mechanical security sensor to obtain information regarding the person captured in the captured image.

[0017] In the above configuration, the system may further include an alarm device that issues an alarm when a specified event is detected by the machine security sensor while the machine security is being performed, and when the abnormality detection unit detects an abnormality, the alarm device may be controlled so as not to issue an alarm even if the specified event is detected by the machine security sensor. Effect of the Invention

[0018] According to the facial recognition device, facial recognition method, facial recognition program, and security system disclosed herein, when authenticating a user through facial recognition processing, it is possible to automatically detect whether an abnormality in security has occurred to the user. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a schematic diagram for explaining an overview of a security system including a face recognition device according to this embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of a security system. [Diagram 3] FIG. 3 is a schematic diagram for explaining the abnormal situation response process. [Figure 4] FIG. 4 is a diagram for explaining the abnormal situation determination conditions. [Diagram 5] FIG. 5 is a flowchart showing an example of the security process. [Figure 6] FIG. 6 is a flowchart showing an example of a process executed by the security system during machine security. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, with reference to the attached drawings, embodiments of a face recognition device, a face recognition method, a face recognition program, and a security system according to the present disclosure will be described. The place where the face recognition device is used is not particularly limited, but in the present embodiment, a case where the face recognition device is used for locking and unlocking control of a door serving as an entrance to a facility guarded by a security system will be described as an example.

[0021] 1 is a schematic diagram for explaining an overview of a security system 1 including a face recognition device according to this embodiment. The security system 1 guards a facility to be secured, which has a door 200 that is locked and unlocked by a locking / unlocking unit 210. A face recognition terminal device 100 having an operation / display unit 130 and a sensor unit 140 is installed near the door 200. The terminal device 100 can accept information input from a user 11 of the security system 1 and display information to be notified to the user 11 via the operation / display unit 130, which is, for example, a touch panel type liquid crystal display device.

[0022] The terminal device 100 is installed outside a door 200 that is an entrance to a facility that is the object of security by the security system 1. The security system 1 controls locking and unlocking of the door 200 based on an authentication result by face authentication processing. For example, a user 11 who comes to the door 200 to enter the facility has his / her face captured by a camera (image sensor) of the sensor unit 140 according to a guide displayed on the operation display unit 130. When face authentication processing is executed based on the captured image and the user 11 is authenticated, the door 200 that was locked by the locking / unlocking unit 210 is unlocked. The user 11 can enter the facility through the unlocked door 200. That is, face authentication processing is executed to confirm that the user 11 is an authorized person who has the authority to enter the facility through the door 200, and the locking and unlocking of the door 200 is controlled based on the authentication result.

[0023] In addition to the conventional door lock / unlock control, the security system 1 can automatically detect when the user 11 is in an abnormal security situation. Furthermore, when the security system 1 detects an abnormal situation, it can execute a process to respond to the abnormal situation. An abnormal security situation is a situation in which the security company needs to check the status of the facility to be guarded or the facility user. The following explanation will be given by taking as an example a case in which the user 11, who is permitted to enter and exit through the door 200, is forced by a suspicious person 12 to unlock the door 200.

[0024] The security system 1 uses the sensor unit 140 of the terminal device 100 to perform face recognition processing of a user 11 entering a facility through a door 200 (A). The sensor unit 140 includes at least an imaging sensor for acquiring a face image to be used in the face recognition processing. The security system 1 uses an image captured by the imaging sensor to perform face recognition processing of the user 11. The face recognition processing may be performed by the terminal device 100, or may be performed by an external device communicably connected to the terminal device 100. The terminal device 100 that performs the face recognition processing, or the terminal device 100 and the external device that is connected to the terminal device 100 and performs the face recognition processing, constitute a face recognition device, and the face recognition device performs face recognition processing using a captured image obtained by capturing an image of the surroundings of the device by the sensor unit 140.

[0025] Since the method of face recognition processing is conventionally known, detailed description will be omitted. In the face recognition processing, a person's face portion is detected on a captured image obtained by the terminal device 100, and it is determined whether or not the detected face is the face of a person permitted to enter the facility through the door 200. For example, face images of each of a plurality of people permitted to enter through the door 200 are registered in a database in advance. Based on a comparison result between the registered face image in the database and the face image of the user 11 detected from the captured image of the terminal device 100, it is determined whether or not the user 11 is a person permitted to enter through the door 200. However, the method of face recognition processing is not particularly limited. For example, in addition to a mode in which the face image itself is registered in the database, a feature amount extracted from the face image may be registered in the database, and the face recognition processing may be performed based on a comparison result between the feature amount and the feature amount extracted from the face image detected by the terminal device 100.

[0026] The security system 1 executes an abnormal situation detection process (B) to automatically detect that an abnormal situation in terms of security has occurred with respect to the user 11 entering through the door 200. The abnormal situation detection process may be executed by the terminal device 100, or may be executed by an external device communicatively connected to the terminal device 100. The abnormal situation detection process is executed by the terminal device 100, or the terminal device 100 and an external device connected to the terminal device 100, that is, by a face authentication device.

[0027] As shown in FIG. 1, the security system 1 performs image analysis on a captured image 600 of the user 11 obtained by the sensor unit 140, and determines whether or not an abnormal situation has occurred with respect to the user 11.

[0028] For example, the security system 1 analyzes the captured image 600 and executes a process of detecting a predetermined detection target such as a suspicious person 12 in close contact, an object considered to be a weapon 20, or the arm of the suspicious person 12, and determines that an abnormal situation has occurred when at least one of these is detected. The detection process may be executed on a predetermined area around the face of the user 11 captured in the captured image 600. For example, the contour of the face of the user 11 may be detected, and the detection process may be executed on a strip-shaped area of ​​a predetermined width set outside the contour. Whether or not a person detected in the captured image 600 is a suspicious person 12 may be determined by, for example, face recognition processing. Specifically, face recognition processing may be executed on a person 12 captured other than the user 11, and if it cannot be confirmed that this person 12 is a person permitted to enter through the door 200, the person 12 may be determined to be a suspicious person 12.

[0029] For example, the security system 1 analyzes the captured image 600 and executes a process of detecting a predetermined posture, a predetermined movement, and a predetermined facial expression of the user, and determines that an abnormal state exists when at least one of these is detected. For example, the predetermined posture includes a tilted face of the user 11. The predetermined movement includes the eyes of the user 11 drifting. The predetermined facial expression includes a facial expression of fear of the user 11. A method of detecting an abnormal situation will be described later.

[0030] The security system 1 unlocks the door 200 based on the result of the face authentication process (C). Specifically, after the user 11 is authenticated by the face authentication process and it is confirmed that the user 11 is a person permitted to enter the facility through the door 200, the terminal device 100 controls the locking / unlocking unit 210 to unlock the door 200. For example, information indicating that the door 200 has been unlocked is displayed on the screen of the operation display unit 130 of the terminal device 100. After confirming the unlocking information on the operation display unit 130, the user 11 can open the door 200 and enter the facility.

[0031] If an abnormality is detected when the user 11 enters the facility, the security system 1 executes an abnormality response process (D). For example, the security system 1 executes a process of reporting the occurrence of an abnormality to a security company. For example, the security system 1 executes a process of tracking the user 11 and suspicious person 12 who enter the facility through the door 200 using security equipment such as security cameras and motion sensors installed in the facility, and providing the obtained information to the security company. The report to the security company and the provision of information may be targeted to a specific server device used by the security company, or may be targeted to a specific communication terminal carried by a security officer.

[0032] For the purpose of ensuring the safety of the user 11, the security system 1 executes an abnormal situation response process so that the suspicious person 12 does not notice that an abnormal situation has been automatically detected. Even when an abnormal situation is detected from the presence of the suspicious person 12, the security system 1 displays unlocking information for the door 200 on the operation display unit 130 of the terminal device 100 and unlocks the door 200, as in a normal state in which no abnormal situation is detected. Note that the security system 1 may be configured to notify the user 11 of the automatic detection of an abnormal situation while preventing the suspicious person 12 from notifying the user. For example, the security system 1 may display a predetermined message, symbol, or the like indicating that an abnormal situation has been automatically detected on the operation display unit 130 of the terminal device 100 so that only the user 11 can understand.

[0033] For example, if an attempt is made to break the locking / unlocking unit 210 in order to enter the facility through the door 200, the security system 1 can detect this as the occurrence of a predetermined event and issue an alarm using an alarm device. The occurrence of a predetermined event is detected using security equipment such as the terminal device 100, a security camera, and a motion sensor. The security system 1 can activate the alarm device to scare off the intruder, and can issue an alarm sound, warn by voice, or turn on or blink an alarm lamp. On the other hand, if an abnormality is automatically detected during the face authentication process of the user 11, the security system 1 controls the alarm device so that it does not activate even if the predetermined event is detected, and executes an abnormality response process while operating in the same manner as when the predetermined event is not detected.

[0034] A security company that receives a report of an abnormal situation from the security system 1 can rush to the facility to be secured without being noticed by the suspicious person 12. Based on the information provided by the security system 1, the security company can identify the current locations of the user 11 and the suspicious person 12 within the facility and respond accordingly.

[0035] In this way, the security system 1 can automatically detect that an abnormal situation has occurred with the user 11 whose face has been authenticated by the terminal device 100, based on information obtained by a face authentication device including the terminal device 100. The detection of an abnormal situation is performed using an image captured around the face authentication device. An abnormal situation is automatically detected based on the person 12 and the weapon 20 captured in a captured image 600 capturing an image of the user 11, and the posture, movement, facial expression, etc. of the user 11 recognized from the captured image 600.

[0036] The security system 1 is set to execute an abnormal situation response process when the user 11 performs a predetermined action set in advance. For example, by determining a predetermined sign in advance, such as blinking slowly three times toward the sensor unit 140 of the terminal device 100, or holding the entrance pass hanging from the neck with both hands over the sensor unit 140, the security system 1 detects the predetermined sign and executes an abnormal situation response process. In addition, the security system 1 can automatically detect an abnormal situation as described above even when the user 11 is in a situation where he or she cannot perform a predetermined sign to notify an abnormal situation. This makes it possible to accurately detect an abnormal situation and respond to it regardless of the situation of the user 11.

[0037] Next, a specific configuration example of the security system 1 will be described. Fig. 2 is a block diagram showing a configuration example of the security system 1. Fig. 2 shows a security server 400 provided in a security company in addition to the security system 1 provided in the facility to be guarded. The security server 400 may be included in the security system 1.

[0038] 2, the security system 1 includes, in addition to the terminal device 100 and the door 200 described in FIG 1, a management device 300 communicatively connected to the terminal device 100. In the example shown in FIG 2, the terminal device 100 and the management device 300 constitute a face authentication device. Although FIG 2 shows only one terminal device 100 and a door 200 connected to the terminal device 100, if there are multiple doors 200, a terminal device 100 may be provided corresponding to each door 200 and each terminal device 100 may be communicatively connected to the management device 300.

[0039] The security system 1 includes a surveillance camera 510, a human presence sensor 520, and an alarm device 530 used for guarding a facility. Fig. 2 shows one each of the surveillance camera 510, the human presence sensor 520, and the alarm device 530, but the surveillance cameras 510 and the human presence sensors 520 are installed at multiple locations inside and outside the facility to be secured so as to track the suspicious person 12. Similarly, multiple alarm devices 530 are installed inside and outside the facility as necessary.

[0040] A predetermined event indicating an unauthorized intrusion into the facility or a sign of such an intrusion is detected by the surveillance camera 510 and the human presence sensor 520. When a predetermined event is detected, the alarm device 530 issues a warning to the intruder and a notice to users in the facility by emitting sound, voice, light, etc. However, as described above, when it is automatically detected that the user 11 is in an abnormal situation, the alarm device 530 is controlled not to operate from the viewpoint of the safety of the user 11.

[0041] The terminal device 100 includes a control unit 110 and a communication unit 120 in addition to the operation display unit 130 and the sensor unit 140 described in Fig. 1. The communication unit 120 is used to communicate with external devices such as a management device 300 via a network 2.

[0042] The sensor unit 140 includes an image sensor 141 for acquiring a face image used in face authentication processing. The sensor unit 140 may include a depth sensor for three-dimensionally recognizing an object in an imaging range, and depth information obtained by the depth sensor may be used to analyze an image obtained by the image sensor 141. For example, the distance between the user 11 and the suspicious person 12 shown in the captured image 600 shown in FIG. 1 may be identified based on the depth information and used to determine the degree of closeness between the two people. For example, the three-dimensional shape of an object shown in the captured image 600 may be identified based on the depth information and used to detect a predetermined object, etc.

[0043] The control unit 110 includes an information acquisition unit 111 and a door control unit 112. The information acquisition unit 111 controls the sensor unit 140 to acquire information necessary for face authentication processing and abnormal situation detection processing. The door control unit 112 controls the locking / unlocking unit 210 to lock / unlock the door 200. For example, a program corresponding to the control unit 110 is pre-stored in a dedicated storage device, and the function and operation of the control unit 110, i.e., the information acquisition unit 111 and the door control unit 112, are realized by executing this program by hardware such as a CPU.

[0044] The door 200 includes the locking / unlocking unit 210 described in Fig. 1. The locking / unlocking unit 210 is communicatively connected to the terminal device 100. Fig. 2 shows an example in which the terminal device 100 and the door 200 are directly communicatively connected, but the door 200 may include a communication unit and be communicatively connected to the terminal device 100 via the network 2.

[0045] The locking / unlocking unit 210 locks and unlocks the door 200. The door 200 is usually in a locked state locked by the locking / unlocking unit 210 and cannot be opened. As described above, when the user 11 is face-recognized and enters the facility, the locking / unlocking unit 210 is controlled by the door control unit 112 of the terminal device 100 to unlock the door 200. When a predetermined time has elapsed since the door 200 was unlocked, the locking / unlocking unit 210 locks the door 200 again. As long as the door 200 can be locked and unlocked by the control from the terminal device 100, the type of the locking / unlocking unit 210 is not particularly limited. For example, a mechanism that has been conventionally called an electric lock, an electromagnetic lock, an electronic lock, or the like may be used as the locking / unlocking unit 210.

[0046] The management device 300 includes a control unit 310, a communication unit 320, and a storage unit 330. For example, a computer device equipped with a touch panel type liquid crystal display as an operation display unit is used as the management device 300. The communication unit 320 is used to communicate with external devices such as the terminal device 100 and the security server 400 via the network 2.

[0047] The storage unit 330 is a non-volatile storage device. The storage unit 330 stores authentication information 331 and abnormal situation determination conditions 332. The authentication information 331 stores information such as a registered face image used in face authentication processing. The abnormal situation determination conditions 332 store determination conditions for detecting an abnormal situation in the abnormal situation detection processing.

[0048] The control unit 310 includes an authentication unit 311, an abnormal situation detection unit 312, an unlocking determination unit 313, and an abnormal situation response processing unit 314. The authentication unit 311 acquires an image of a user from the terminal device 100 and executes a face authentication process. The authentication unit 311 may be capable of executing multiple types of authentication processes, such as an authentication process based on a user ID and a password input to the operation display unit 130 of the terminal device 100, in addition to the face authentication process.

[0049] The authentication information 331 in the memory unit 330 includes registered face images used for face authentication processing of each of multiple users permitted to enter and exit through the door 200, the user ID of each user, and a password that each user can use to enter and exit.

[0050] The authentication unit 311 receives the captured image acquired by the information acquisition unit 111 of the terminal device 100 from the imaging sensor 141 of the sensor unit 140 via the network 2, and executes face authentication processing using the captured image. If the captured image contains a human face, the authentication unit 311 compares and collates it with each registered face image registered in the authentication information 331. If the authentication information 331 contains a registered face image that matches the face of a person captured by the terminal device 100, the person is authenticated and determined to be a user who is permitted to enter through the door 200.

[0051] The abnormal situation detection unit 312 receives captured images acquired by the information acquisition unit 111 of the terminal device 100 from the imaging sensor 141 of the sensor unit 140 via the network 2, and executes an abnormal situation detection process using the captured images. The terminal device 100 continuously captures images using the imaging sensor 141. The abnormal situation detection unit 312 can execute the abnormal situation detection process in real time using still images continuously captured by the imaging sensor 141 or moving images captured by the imaging sensor 141. The abnormal situation detection process is performed based on an abnormal situation determination condition 332, which will be described later.

[0052] The unlocking determination unit 313 determines whether or not to unlock the door 200 based on the authentication result by the authentication unit 311. When it is confirmed that the person is authenticated by the face authentication process and is permitted to enter and exit through the door 200, the unlocking determination unit 313 determines to unlock the door 200.

[0053] The unlocking determination unit 313, which has determined that the door 200 should be unlocked, transmits an unlocking command to the terminal device 100. In the terminal device 100 which has received the unlocking command via the network 2, the door control unit 112 controls the locking / unlocking unit 210 of the door 200, thereby unlocking the door 200.

[0054] The emergency situation response processing unit 314 executes an emergency situation response process when the emergency situation detection unit 312 detects an emergency situation. Fig. 3 is a schematic diagram for explaining the emergency situation response process. As shown in Fig. 3, the emergency situation response processing unit 314, i.e., the management device 300, reports the occurrence of an emergency situation to a security server 400 used by a security company via the network 2. The security company that receives the report can instruct a security officer to rush to the facility to be guarded that is the source of the report.

[0055] As shown in FIG. 3, the emergency response processing unit 314 can track users 11 and suspicious persons 12 entering the facility through the door 200 by using surveillance cameras 510 and human presence sensors 520 installed at multiple locations inside and outside the facility.

[0056] The emergency response processing unit 314 provides information on the user 11 and the suspicious person 12 to the security server 400 via the network 2. For example, images of the user 11 and the suspicious person 12 captured by the terminal device 100 are transmitted to the security server 400. For example, information that can be used to identify the current location of the user 11 and the current location of the suspicious person 12 within the facility is transmitted to the security server 400. Security personnel can use a communication terminal such as a smartphone or tablet to check the information received by the security server 400 from the management device 300. The security personnel can respond to an emergency by referring to information such as the images and current locations of the user 11 and the suspicious person 12.

[0057] Next, the abnormal situation detection process will be described. The abnormal situation detection process is executed by the abnormal situation detection unit 312 of the management device 300, using information obtained by the sensor unit 140 of the terminal device 100, based on the abnormal situation determination conditions 332 stored in the storage unit 330. When the analysis result of the captured image by the sensor unit 140 of the terminal device 100 satisfies the determination conditions registered in the abnormal situation determination conditions 332, it is determined that an abnormal situation has occurred.

[0058] Fig. 4 is a diagram for explaining the abnormal situation determination condition 332. As shown in Fig. 4, the abnormal situation detection unit 312 determines that an abnormal situation exists on the condition that a detection target registered for each of a plurality of detection types is detected from an image captured by the terminal device 100.

[0059] 4, detection of a sign by which the user notifies himself of an abnormal situation is registered as a judgment condition in abnormal situation judgment conditions 332. As described above, predetermined actions such as blinking eye movement and hand movement touching the entry pass are set as signs. The abnormal situation detection unit 312 recognizes the eye movement and hand movement of the user by analyzing still images or moving images continuously captured by the image sensor 141. If the recognized movement matches the sign notifying an abnormal situation, the abnormal situation detection unit 312 judges that the user is in an abnormal situation.

[0060] The detection of a predetermined object is registered as a judgment condition in abnormal situation judgment conditions 332. The predetermined object includes weapons such as a gun, a knife, a stun gun, and an ice pick. The abnormal situation detection unit 312 searches for the object registered in the abnormal situation judgment conditions 332 on the image captured by the image sensor 141. When the predetermined object is captured in the captured image, the abnormal situation detection unit 312 judges that the user is in an abnormal situation.

[0061] The detection of a person other than the person to be authenticated, i.e., a person other than the user whose face has been authenticated by the authentication unit 311, is registered as a judgment condition in the abnormal situation judgment condition 332. The abnormal situation detection unit 312 searches for a person other than the user whose face has been authenticated who appears in an image captured by the imaging sensor 141, and judges that the user is in an abnormal situation if the person's face is not authenticated by the authentication unit 311.

[0062] However, if it is known that a user always enters alone, it may be determined that the user is in an abnormal situation only based on the fact that a person other than the user is captured in the captured image, without performing face authentication processing of the person other than the user. For example, if the users are required to perform face authentication one by one when entering the building, and the terminal device 100 is installed to capture only the area around the face of one user, it is sufficient to determine that an abnormal situation is occurring only based on the fact that a person other than the user is detected around the face of the user. For example, information indicating that the user always enters alone may be registered in the authentication information 331 in association with some user IDs, and it may be determined that an abnormal situation is occurring only based on the fact that a person other than the user is detected during face authentication processing of the user.

[0063] A characteristic for determining an abnormal situation for a person other than the user appearing in the captured image can be added to the determination conditions. For example, a condition for determining an abnormal situation can be added that a person other than the user is in close contact with the user. For example, a condition for determining an abnormal situation can be added that a person other than the user is hiding his / her face with a balaclava or a mask. Even if a person other than the user is detected, if a predetermined characteristic such as a mask is not detected for this person, it can be set not to determine an abnormal situation. In other words, if a person other than the user having a predetermined characteristic is detected, it can be set to determine an abnormal situation.

[0064] 4, the posture of the user whose face has been authenticated by the authentication unit 311 is registered as a judgment condition in the abnormal situation judgment condition 332. For example, when the user shown in the image captured by the image sensor 141 is in a posture of being pinned down, the abnormal situation detection unit 312 judges that the user is in an abnormal situation. For example, when the arm of a person other than the user is wrapped around the neck of the user, it is judged that an abnormal situation has occurred.

[0065] Specifically, for example, the abnormal situation detection unit 312 determines that the user is in an abnormal situation based on the inclination of the user's face captured in an image captured by the image sensor 141. When a suspicious person is pointing a weapon at the user or is holding the user in a bear hug, the user's face is inclined compared to normal. When the state in which the user's face is inclined beyond a predetermined angle continues for a predetermined period of time or more, the user is determined to be in an abnormal situation. The inclination of the user's face may be recognized directly from the captured image, or may be recognized based on the angle between a straight line connecting the positions of the left and right eyes and the horizontal line by detecting the positions of the left and right eyes from the captured image. Instead of the inclination of the face, the inclination of the eyes, i.e., the angle between the straight line connecting the positions of the left and right eyes of the user and the horizontal line, may be directly used, and when the state in which this angle exceeds a predetermined angle continues for a predetermined period of time or more, the user may be determined to be in an abnormal situation.

[0066] The behavior of the user whose face has been authenticated by the authentication unit 311 is registered as a judgment condition in the abnormal situation judgment condition 332. The abnormal situation detection unit 312 judges that the user is in an abnormal situation when a behavior such as the user's eyes wandering, the user's grimacing, or the user's trembling is recognized from the image captured by the image sensor 141.

[0067] For example, a predetermined facial expression such as a facial expression of fear or pain of the user is registered as a judgment condition in the abnormal situation judgment condition 332. When a predetermined facial expression is recognized from an image captured by the image sensor 141, the abnormal situation detection unit 312 judges that the user is in an abnormal situation.

[0068] 4 is detected, the abnormal situation may be determined to be occurring. Alternatively, the abnormal situation may be determined to be occurring when two or more of the detection targets are detected. The determination conditions, including the detection targets for detecting the abnormal situation and the thresholds for determining that the abnormal situation is occurring, may be set appropriately according to the facility or environment in which the security system 1 is used.

[0069] Conventionally known image analysis and recognition technologies may be used to recognize a person's face, posture, movement, facial expression, etc. Similarly, conventionally known image analysis and recognition technologies may be used to recognize an object such as a murder weapon.

[0070] For example, facial expression recognition may be performed using AI (Artificial Intelligence) that has been trained on various human facial expressions. If a numerical value indicating a correlation between the user's face obtained by the terminal device 100 and a face showing fear that has been trained on by machine learning exceeds a predetermined threshold, it may be determined that the user shows a fearful expression. Similarly, detection of a specific object including a murder weapon may be performed by image analysis and image recognition using AI that has been trained on specific objects that are considered to be murder weapons.

[0071] When the sensor unit 140 includes a depth sensor, the depth information may be used to recognize a person's face, posture, movement, facial expression, object, etc. For example, a three-dimensional shape or a person's skeleton recognized by the depth sensor may be used to detect a person's posture, movement, object, etc.

[0072] Next, an example of security processing executed by the security system 1 shown in Fig. 2 will be described. Fig. 5 is a flowchart showing an example of the security processing. The security system 1 monitors the area around the door 200 using a management device 300 and a terminal device 100 installed near the door 200.

[0073] Specifically, the authentication unit 311 of the management device 300 continues to monitor whether or not a recognizable face is included in images continuously captured by the image sensor 141 of the terminal device 100 around the door 200. When a recognizable face is detected in the captured images obtained by the terminal device 100, the process shown in Fig. 5 is started, where the authentication unit 311 executes face recognition processing (step S1) and the abnormal situation detection unit 312 executes abnormal situation detection processing (step S2).

[0074] After the face of a user is authenticated, the abnormal situation detection process can be repeatedly executed using images continuously captured by the sensor unit 140. For example, the abnormal situation detection process continues until the image of the face-authenticated user can no longer be captured by the image sensor 141. For example, the abnormal situation detection process may be repeatedly executed at predetermined time intervals, such as every second, or may be repeatedly executed every time the image captured by the sensor unit 140 changes.

[0075] If the detected face is not authenticated by the face recognition process (step S3; No), that is, if the face detected from the image captured by the terminal device 100 is not the face of a user permitted to enter through the door 200, the processing related to the detected face is terminated.

[0076] If the detected face is authenticated (step S3; Yes), that is, if the face detected from the captured image is the face of a user permitted to enter through the door 200, the unlocking determination unit 313 determines to unlock the door 200 and transmits an unlocking command to the terminal device 100. In the terminal device 100 that has received the unlocking command, the door control unit 112 controls the locking / unlocking unit 210 of the door 200 to unlock the door 200 (step S4).

[0077] If no abnormality is detected in the abnormality detection process (step S5; No), the process ends. On the other hand, if an abnormality is detected in the abnormality detection process (step S5; Yes), the abnormality response processor 314 executes an abnormality response process (step S6).

[0078] In the emergency response process, predetermined information for reporting an emergency to a security company is transmitted to the security server 400. In addition, information on the user 11 and the suspicious person 12 obtained using the surveillance camera 510 and the human presence sensor 520 is transmitted to the security server 400.

[0079] In response to the report sent to the security server 400, a security officer from the security company rushes to the facility to be secured. The emergency response process continues unless the security officer performs a predetermined release operation on the management device 300 or the terminal device 100 (step S7; No). When the security officer finishes responding to the emergency and performs a predetermined release operation (step S7; Yes), the process ends.

[0080] Next, mechanical security will be described. In a facility to be guarded, mechanical security is carried out after all facility users have left the facility. For example, some of the surveillance cameras 510 and human sensors 520 installed inside and outside the facility are provided for mechanical security and do not operate while employees are present in the facility. Some of these surveillance cameras 510 and human sensors 520 start operating when mechanical security begins. Some of the alarm devices 530 also start operating at the same time as mechanical security begins.

[0081] After starting mechanical security, the management device 300 monitors for intruders into the facility using the surveillance camera 510 and the human sensor 520. If an unauthorized intrusion into the facility or a sign of such an intrusion is detected, the management device 300 warns the intruder by emitting sound and light from the alarm device 530, and also notifies the security company.

[0082] During mechanical security, if a user who is permitted to enter the facility executes face recognition processing on the terminal device 100 at the door 200 and enters normally, the mechanical security is lifted. When mechanical security is lifted, the surveillance camera 510 and the human sensor 520 installed for mechanical security stop operating. Similarly, some of the alarm devices 530 also stop operating at the same time as the end of mechanical security. If an abnormality is detected when face recognition processing is executed during mechanical security, the security system 1 executes processing that is different from normal.

[0083] Fig. 6 is a flowchart showing an example of processing executed by the security system 1 during machine security. During machine security, when a face that can be recognized is detected in a captured image obtained by the terminal device 100, the processing shown in Fig. 6 is started. The processing from steps S11 to S15 shown in Fig. 6 is performed in the same manner as steps S1 to S5 shown in Fig. 5.

[0084] If the abnormality detection process in step S12 does not detect an abnormality in the user entering through the door 200 (step S15; No), the management device 300 executes a process to notify the user that mechanical security has been released (step S16). For example, information indicating that mechanical security has been released is displayed on the operation display unit 130 of the terminal device 100. For example, the display on the operation display unit 130 of the terminal device 100 is changed from a display indicating mechanical security to a display indicating normal security after mechanical security has been released.

[0085] Next, the management device 300 executes a mechanical security termination process to stop the operation of the surveillance camera 510, the human sensor 520, and the alarm device 530 provided for mechanical security (step S17), and the process related to the user's entry into the building is completed.

[0086] On the other hand, if an abnormal situation of the face-authenticated user is detected in the abnormal situation detection process (step S15; Yes), the abnormal situation response process is executed by the abnormal situation response processing unit 314 of the management device 300. The abnormal situation response process includes the processes of steps S18 to S20 shown in FIG.

[0087] If an abnormality is detected, the process of notifying the cancellation of mechanical security is executed in the same manner as if no abnormality was detected (step S18). That is, in step S18, the same process as in step S16 is executed.

[0088] The emergency response processing unit 314 reports the occurrence of an emergency by transmitting predetermined information to the security server 400 of the security company (step S19). The emergency response processing unit 314 reports the cancellation of mechanical security in step S18, and continues mechanical security while seemingly canceling the mechanical security (step S20). As a result, the surveillance camera 510 and the human sensor 520 for mechanical security continue to operate, and the emergency response processing unit 314 can use the surveillance camera 510 and the human sensor 520 to track users and suspicious people who have entered the building within the building. However, in the mechanical security that continues after the detection of an emergency, the management device 300 controls the alarm device 530 not to operate, so the alarm device 530 does not issue an alarm.

[0089] Tracking of users and suspicious people continues unless a security officer from the security company who arrives at the facility to be secured upon receiving the report performs a specified release operation on the management device 300 or terminal device 100 (step S21; No). When the security officer finishes dealing with the abnormal situation and performs a specified release operation (step S21; Yes), the management device 300 executes mechanical security termination processing (step S22). As in step S17, the operation of the surveillance camera 510 and human sensor 520 for mechanical security is stopped with the end of mechanical security, and processing related to the entry of the user is terminated.

[0090] In this way, even if a suspicious person threatens the user and causes face authentication by the terminal device 100, the door 200 is unlocked and the unlocking of the mechanical security is notified. Although the mechanical security appears to be released, the mechanical security actually continues, and the user and the suspicious person are tracked inside and outside the facility, and information on their current locations is sent to the security server 400. This allows the security company to rush to the facility to be guarded and respond to the abnormal situation without the suspicious person knowing. For example, the security company can confirm, based on the information obtained from the management device 300, whether the suspicious person is still in the facility, where he or she is in the facility, or whether he or she has already left the facility, and take different measures depending on the confirmation results.

[0091] In addition, the security sensors such as the surveillance camera 510 and the motion sensor 520 provided for security are not limited to being stopped after the security is completed, but may continue to operate in a mode different from that during security. Similarly, the alarm device 530 may continue to operate in a mode different from that during security even after the security is completed.

[0092] The configuration of the security system 1 shown in FIG. 2 of this embodiment is a functional schematic, and the configuration of the security system 1 is not limited to this physical configuration. In the example shown in FIG. 2, the terminal device 100 may realize part or all of the functions and operations of the management device 300 described above. For example, the terminal device 100 may execute at least one of the face authentication process and the abnormal situation detection process. The management device 300 may realize part of the functions and operations of the terminal device 100 described above. For example, the door 200 may be connected to the network 2, and the management device 300 may directly control the locking and unlocking of the door 200 via the network 2. The face authentication device that executes the face authentication process and the abnormal situation detection process may be realized by either the terminal device 100 or the management device 300, or may be realized by the terminal device 100 and the management device 300. The form of distribution and integration of each device is not limited to the above example, and all or a part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0093] In the present embodiment, an example has been described in which the face authentication process for authenticating an authorized person having a predetermined authority is executed for a user who is seeking to enter or exit through the door 200, but the purpose of the face authentication process is not limited to locking and unlocking control of the door 200. The face authentication process may be executed to control at least one of the start and end of a predetermined process such as mechanical security. For example, a user may be authenticated by the face authentication process, and mechanical security may be started or released based on the authentication result. The face authentication process may be executed to restrict the use of a predetermined device such as a computer device. For example, a user may be authenticated by the face authentication process, and the lock of the device may be released so that the computer device can be used based on the authentication result. In these cases, too, it is possible to execute an abnormal situation detection process for the authorized person during the execution of the face authentication process, and respond to an abnormal situation that occurs in the authorized person based on the detection result.

[0094] As described above, according to the security system of the present embodiment, for example, when a face recognition process of a user who is seeking to enter a facility is executed to unlock the door, it is determined whether the user is in an abnormal situation, and if so, this is automatically detected. The face recognition device executes an abnormal situation detection process in addition to the face recognition process to detect an abnormal situation. For example, in the case where a malicious person threatens a user who is permitted to enter the facility to have his / her face recognized, the security system can detect that the user is in an abnormal situation by the face recognition device. When an abnormal situation is detected, the security system can report the abnormal situation to a security company, track an intruder into the facility, and the like, while apparently operating in the same way as when no abnormal situation is detected. The security company that receives the report can respond to the abnormal situation by using the tracking information of the intruder obtained from the security system. [Industrial Applicability]

[0095] As described above, the face recognition device, face recognition method, face recognition program, and security system disclosed herein are useful for automatically detecting the occurrence of an abnormality when checking the presence or absence of a specified authority based on the results of face recognition. [Explanation of symbols]

[0096] 1. Security System 2 Network 100 Terminal Equipment 110, 310 Control unit 111 Information Acquisition Department 112 Door control section 120, 320 Communications Department 130 Operation display section 140 Sensor section 141 Image Sensor 200 Doors 210 Locking / unlocking section 300 Management device 311 Authentication Department 312 Abnormality Detection Unit 313 Unlocking Judgment Unit 314 Emergency Situation Response Department 330 Storage section 400 Security Server 510 Surveillance Camera 520 Human Sensor 530 Alarm device

Claims

1. A face recognition device that acquires information about a person to be authenticated, A sensor unit that captures an image of the surroundings of the face authentication device; an authentication unit that performs face authentication processing on a person captured by the sensor unit to authenticate an authorized person having a predetermined authority; an abnormal situation detection unit that executes an abnormal situation detection process by analyzing an image captured by the sensor unit and automatically detecting whether the authorized person who has been face-authenticated is in an abnormal situation; A face recognition device comprising:

2. The face recognition device according to claim 1, characterized in that the abnormal situation detection unit executes a process of detecting a predetermined object regarded as a weapon, an arm of another person, and another person in close contact from the captured image, and determines that the authenticated person is in an abnormal situation based on the detection results.

3. 3. The face authentication device according to claim 2, wherein the abnormality detection unit executes the detection process on a predetermined area around the face of the authenticated person appearing in the captured image.

4. The face recognition device according to claim 1, characterized in that the abnormal situation detection unit executes a process of detecting a predetermined posture, a predetermined movement, and a predetermined facial expression of the authenticated person from the captured image, and determines that the authenticated person is in an abnormal situation based on the detection results.

5. the predetermined posture includes a face of the authorized person being tilted; the predetermined motion includes the authorized person's eyes moving, The predetermined facial expression includes a facial expression of fear of the authorized person. The face authentication device according to claim 4 .

6. an abnormality response processing unit that executes an abnormality response process including a process of reporting an abnormality to a predetermined external device when the abnormality detection unit detects an abnormality and a process of transmitting information about a person captured in the captured image to the external device; The face recognition device according to any one of claims 1 to 5, further comprising:

7. The face recognition device according to claim 6, characterized in that the information transmitted to the external device in the emergency response process includes a captured image of a person appearing in the captured image together with the authorized person, and information regarding the current location of the person.

8. A face recognition method executed by a face recognition device that acquires information about a person to be authenticated, comprising: A step of executing a face authentication process for detecting a face from a captured image of the periphery of the face authentication device and determining whether or not the face is a face of an authorized person having a predetermined authority; a step of executing an abnormal situation detection process for automatically detecting whether the authorized person whose face has been authenticated is in an abnormal situation by analyzing the captured image; A face recognition method comprising:

9. A face recognition program for acquiring information about a person to be authenticated, A step of executing a face authentication process in which a face is detected from a captured image of the periphery of the face authentication device and whether or not the face is a face of an authorized person having a predetermined authority; a step of executing an abnormal situation detection process for automatically detecting whether the authorized person whose face has been authenticated is in an abnormal situation by analyzing the captured image; A face recognition program that causes a computer to execute the above.

10. A face recognition device according to any one of claims 1 to 5, a sensor for mechanical security used for mechanical security performed when the facility in which the face recognition device is installed is unoccupied; Including, A security system that releases the mechanical security based on the authentication result of the face authentication device, When the abnormality detection unit detects an abnormality, the mechanical security is seemingly cancelled, but the use of the mechanical security sensor is continued to acquire information about a person captured in the captured image. A security system characterized by:

11. An alarm device that issues an alarm when a predetermined event is detected by the machine security sensor during the machine security operation. Further equipped with When the abnormality detection unit detects an abnormality, the alarm device is controlled so as not to issue an alarm even if the predetermined event is detected by the machine security sensor.

11. The security system according to claim 10.

Images