Face authentication device, face authentication method, and face authentication program

Abstract

To detect persons entering and exiting from a door and control locking / unlock of the door.SOLUTION: A face authentication device for executing face authentication processing for confirming the right to use a facility is constituted to include: a sensor unit including at least one sensor for acquiring an image in which the surrounding of the facility is imaged and information relating to the bearing of the body of a person included in the image; an authentication unit for detecting the face of the person from the image obtained by the sensor unit and determining whether or not it is the face of a person having the right to use the facility; a determination unit for identifying the bearing of the body of the person imaged by the sensor unit from the information obtained by the sensor unit and determining whether or not the person seen in the image uses the facility; and a control unit for exercising control pertaining to the enablement of use of the facility on the basis of the results of face authentication processing and determination processing, thus control is exercised for the enablement of use of the facility by the person having been determined to have the right to use the facility by face authentication processing, on condition that it has been determined by determination processing that the person uses the facility.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a face recognition device, a face recognition method, and a face recognition program that perform face recognition processing to confirm the authority to use a specified facility. [Background technology]

[0002] Conventionally, face recognition technology has been used to confirm the use authority of various facilities, such as facility doors and devices installed in the facility. For example, Patent Document 1 discloses a system that performs face recognition by comparing a face image of a user captured by a camera with a face image registered in advance that is permitted to enter and exit, and unlocks an electric lock on the door when the recognition is successful. The electric lock on the door is automatically locked and unlocked so that only users whose face images have been registered in advance can enter and exit, improving security and convenience for users. Face recognition technology is used in various fields, and for example, Patent Document 2 discloses a technology that controls the availability of an image forming device based on the face recognition result of a user approaching the device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-234637 A [Patent Document 2] Patent Publication No. 2021-024093 Summary of the Invention [Problem to be solved by the invention]

[0004] However, as the accuracy of face recognition has improved and face recognition of users at a distance from the camera has become possible, for example, when the electric lock of a door is locked or unlocked based on the result of face recognition of the user, the user may be face recognized and the electric lock of the door may be unlocked simply by passing in front of the door without entering or exiting. By using the technology of Patent Document 2, the electric lock can be unlocked based on the user's approach to the door, but even when the user passes in front of the door, the user may move away from the door after approaching it, which may cause the electric lock to be unlocked. When the anti-passback function is used, a first user passing in front of the door may be unintentionally face recognized and the electric lock may be unlocked, and a second user may enter or exit through the unlocked door, and the anti-passback function may be activated and the first user may not be able to enter or exit.

[0005] The present disclosure has been made in consideration of the conventional technologies including the above-mentioned problems, and one of its objectives is to provide a face recognition device, a face recognition method, and a face recognition program capable of detecting persons who intend to use equipment. [Means for solving the problem]

[0006] The facial recognition device according to the present disclosure is a facial recognition device that performs facial recognition processing to confirm authority to use a facility, and includes a sensor unit including at least one sensor that acquires an image captured around the facility and information regarding the orientation of the torso of a person contained in the image; an authentication unit that performs facial recognition processing to detect a person's face from the image obtained by the sensor unit and determine whether or not the face is the face of a person who has authority to use the facility; a determination unit that identifies the orientation of the torso of the person captured by the sensor unit from the information obtained by the sensor unit and performs a determination process to determine whether or not the person will use the facility based on the orientation of the torso; and a control unit that performs control related to whether or not the facility can be used based on the results of the facial recognition processing and the determination processing, and executes control to allow the person who has been determined in the facial recognition processing to have authority to use the facility to use the facility, on the condition that the person who has been determined in the determination processing to use the facility will use the facility.

[0007] In the above configuration, the sensor unit includes a first sensor that captures an image of the periphery of the equipment and a second sensor that acquires information identifying the left and right shoulders of the person imaged by the first sensor, and the authentication unit detects a face from the image captured by the first sensor and executes the face authentication process, and the determination unit identifies the left and right shoulders of the person whose face has been authenticated from the information acquired by the second sensor, and identifies the orientation of the torso based on the left and right shoulders to execute the determination process.

[0008] In the above configuration, the determination unit may determine the distance and direction from a predetermined origin position set in three-dimensional space to the right shoulder and the distance and direction to the left shoulder of the person whose face has been authenticated, and determine the orientation of the person's torso based on the positions of the right shoulder and the left shoulder.

[0009] In the above configuration, the sensor unit includes a first sensor that captures an image of the periphery of the equipment, and the authentication unit detects a face from an image captured by the first sensor and performs the face authentication process, and the determination unit calculates, from the shoulder width of the person whose face has been authenticated, a shoulder width of the person that would appear in the image captured by the first sensor if the person were facing the first sensor directly, and identifies the orientation of the person's torso based on the calculated shoulder width and the shoulder width of the person captured in the captured image, and performs the determination process.

[0010] In the above configuration, the determination unit may determine whether or not the person will use the facility based on the direction in which the facility is located as viewed from the person and the orientation of the torso, assuming that the orientation of the torso is a direction perpendicular to a straight line connecting the left and right shoulders of the person whose face has been authenticated.

[0011] In the aforementioned configuration, the determination unit may execute the determination process for a person whose face has been authenticated until a predetermined time has elapsed since the face authentication.

[0012] In the above configuration, the determination unit may identify a position of a person who has been facially authenticated, calculate a time required for the person to travel from the identified position to the facility, and execute the determination process targeting the person until the time has elapsed.

[0013] The face recognition device according to the present disclosure includes a sensor unit including at least one sensor that acquires an image captured around a door locked by a locking / unlocking unit and information regarding the orientation of a person's torso contained in the image; an authentication unit that executes a face recognition process that detects a person's face from the image acquired by the sensor unit and determines whether or not the face is the face of a person authorized to enter and exit through the door; a determination unit that identifies the orientation of the torso of the person imaged by the sensor unit from the information acquired by the sensor unit and executes a determination process that determines whether or not the person will enter or exit through the door based on the orientation of the torso; a door control unit that controls the locking / unlocking unit to unlock the door on condition that a person determined in the face recognition process to be a person authorized to enter and exit through the door is determined in the determination process to enter or exit through the door; and an entry / exit management unit that manages entry / exit information of a person whose face has been authenticated through the door based on whether the door has been opened or closed after the door has been unlocked, and performs anti-passback control of the door based on the entry / exit information managed by the entry / exit management unit. The anti-passback control referred to here includes at least one of the following: when the door is an entrance / exit to a specified space such as inside a building or a room, control that limits the people who can enter the specified space through the door to only those whose entry / exit information indicates that they are outside the specified space, and control that limits the people who can exit the specified space through the door to only those whose entry / exit information indicates that they are inside the specified space.

[0014] The facial recognition method according to the present disclosure is a facial recognition method executed by a facial recognition device to confirm authority to use a facility, and includes the steps of: executing a facial recognition process to detect a person's face from a captured image around the facility and determine whether or not the face is the face of a person who has authority to use the facility; acquiring information regarding the orientation of the person's torso included in the captured image, identifying the orientation of the person's torso, and executing a determination process to determine whether or not the person will use the facility based on the orientation of the torso; and executing control to allow the person who has been determined in the facial recognition process to have authority to use the facility to use the facility, on condition that the determination process has determined that the person will use the facility.

[0015] The facial recognition program of the present disclosure is a facial recognition program for confirming authority to use a facility, and causes a computer to execute the following steps: executing a facial recognition process of detecting a face from a captured image of the vicinity of the facility and determining whether or not the face is the face of a person who has authority to use the facility; acquiring information regarding the orientation of the person's torso included in the captured image, identifying the orientation of the person's torso, and executing a determination process of determining whether or not the person will use the facility based on the orientation of the torso; and executing control to allow the person who has been determined in the facial recognition process to have authority to use the facility to use the facility, on the condition that the determination process determines that the person will use the facility. Effect of the Invention

[0016] According to the facial recognition device, facial recognition method, and facial recognition program disclosed herein, for example, facial recognition processing can be performed on equipment such as facility doors and devices installed in the facility, for users who wish to use the equipment, and control over whether or not the equipment can be used can be performed based on the authentication results. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram for explaining an overview of a door locking / unlocking system including a face recognition device according to this embodiment. [Diagram 2]FIG. 2 is a schematic diagram of FIG. 1 viewed from above. [Diagram 3] FIG. 3 is a block diagram showing a configuration example of a door locking / unlocking system. [Figure 4] FIG. 4 is a diagram for explaining the entry / exit determination process performed by using the image sensor of the terminal device. [Diagram 5] FIG. 5 is a diagram for explaining the entry / exit determination process performed by utilizing the depth sensor of the terminal device. [Figure 6] FIG. 6 is a diagram for explaining a method of calculating an angle indicating the orientation of a user's torso. [Figure 7] FIG. 7 is a flowchart showing an example of the door locking / unlocking process. [Figure 8] FIG. 8 is a flowchart showing a case where the process shown in FIG. 7 is executed in a setting that utilizes the anti-passback function. [Figure 9] FIG. 9 is a flowchart showing a process flow when the door locking / unlocking system executes an entry / exit determination process while tracking a user. [Figure 10] FIG. 10 is a flowchart showing a process flow when the door locking / unlocking system determines the entry / exit judgment and the time limit for door unlocking for each user who has undergone face authentication. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, an embodiment of a face recognition device, a face recognition method, and a face recognition program according to the present disclosure will be described with reference to the accompanying drawings. The place where the face recognition device is used is not particularly limited, but in this embodiment, a case where the face recognition device is used for locking and unlocking control of a door that is an entrance to a facility will be described as an example.

[0019] FIG. 1 is a schematic diagram for explaining an overview of a door locking / unlocking system 1 including a face authentication device according to this embodiment. The door locking / unlocking system 1 includes a terminal device 100 having an operation display unit 130 and a sensor unit 140, and a door 200 that is locked and unlocked by a locking / unlocking unit 210. The terminal device 100 is installed near the door 200. The terminal device 100 can accept information input from users 11 and 12 of the door locking / unlocking system 1 and display information to be notified to the users 11 and 12 via the operation display unit 130, which is, for example, a touch panel type liquid crystal display device. As shown in FIG. 1, the diagram showing the configuration of the door locking / unlocking system 1 shows orthogonal coordinates so that the relationship between the configurations shown in each diagram can be understood. The Z-axis direction of the orthogonal coordinates indicates the up-down direction, and the X-axis direction indicates the horizontal direction along the wall surface on which the terminal device 100 is provided and the surface of the closed door 200.

[0020] The door locking / unlocking system 1 controls the locking / unlocking of the door 200 based on the authentication result from the face authentication process. For example, the users 11 and 12 who come to the door 200 have their faces captured by the camera (image sensor) of the sensor unit 140, following the guidance displayed on the operation display unit 130. When the face authentication process is executed using the captured image and the users 11 and 12 are authenticated, the door 200 that was locked by the locking / unlocking unit 210 is unlocked. The users 11 and 12 can enter and exit by opening and closing the unlocked door 200.

[0021] In addition, the door locking / unlocking system 1 detects the users 11 and 12 entering and exiting through the door 200 before the users 11 and 12 approach the door 200, performs imaging and face authentication processing, and controls the locking and unlocking of the door 200 based on the authentication result. When the users 11 and 12 arrive in front of the door 200, their faces have already been authenticated and the door 200 has been unlocked, so the users 11 and 12 can enter and exit through the door 200 without checking the display on the operation display unit 130 or holding their faces over the sensor unit 140. For example, the door locking / unlocking system 1 can authenticate the users 11 and 12 who are several meters away from the door 200 and lock and unlock the door 200.

[0022] The door locking / unlocking system 1 unlocks the door 200 when the users 11 and 12 enter or exit through the door 200, but does not unlock the door 200 when the users 11 and 12 simply pass in front of the door 200. Below, the explanation will be continued taking as an example a case where the user 11 enters or exits through the door 200, but the user 12 does not enter or exit through the door 200.

[0023] The door locking / unlocking system 1 executes face authentication processing of the users 11 and 12 by using the sensor unit 140 of the terminal device 100 (A). The sensor unit 140 includes at least an imaging sensor for acquiring a face image to be used in the face authentication processing. The door locking / unlocking system 1 detects the faces of the users 11 and 12 from an image captured by the imaging sensor and executes the face authentication processing. The face authentication processing may be executed by the terminal device 100 or an external device communicably connected to the terminal device 100. The terminal device 100 that executes the face authentication processing, or the terminal device 100 and the external device that is connected to the terminal device 100 and executes the face authentication processing, constitute a face authentication device, and the face authentication device executes the face authentication processing by using a captured image obtained by capturing an image of the door 200, i.e., the periphery of the facility.

[0024] Since the method of face recognition processing is conventionally known, detailed description will be omitted. In the face recognition processing, a person's face portion is detected on a captured image obtained by the terminal device 100, and it is determined whether or not the detected face is the face of a person permitted to enter and exit through the door 200. For example, face images of each of a plurality of people permitted to enter and exit through the door 200 are registered in a database in advance. Based on a comparison result between the registered face images in the database and the face images of the users 11 and 12 detected from the captured image of the terminal device 100, it is determined whether or not the users 11 and 12 are persons permitted to enter and exit through the door 200. However, the method of face recognition processing is not particularly limited. For example, in addition to a mode in which the face image itself is registered in the database, a feature amount extracted from the face image may be registered in the database, and the face recognition processing may be performed based on a comparison result between the feature amount and the feature amount extracted from the face image detected by the terminal device 100.

[0025] The door locking / unlocking system 1 executes an entry / exit determination process for determining whether or not the users 11, 12 enter or exit through the door 200 (B). The entry / exit determination process may be executed by the terminal device 100, or may be executed by an external device communicably connected to the terminal device 100. The entry / exit determination process for determining whether or not the users enter or exit through the door 200 is a determination process for determining whether or not the facility is to be used. The determination process is executed by the terminal device 100, or by the terminal device 100 and an external device connected to the terminal device 100, i.e., by a face authentication device.

[0026] The door locking / unlocking system 1 judges whether the users 11 and 12 enter or exit through the door 200 based on the orientation of the torso of the users 11 and 12. Specifically, the door locking / unlocking system 1 judges that the users 11 and 12 enter or exit through the door 200 when the orientation of the torso of the users 11 and 12 coincides with the direction of the door 200 and the terminal device 100 as seen from the users 11 and 12 within a predetermined error range. Here, the orientation of the torso in this embodiment refers to the direction in the horizontal plane (XY plane) in which the front part of the torso part above the waist of the users 11 and 12, where the navel is located, is facing. The solid arrows 21b and 22b shown in FIG. 1 indicate the orientation of the torso of each user 11 and 12.

[0027] The door locking / unlocking system 1 detects the left and right shoulders of each user 11, 12 by using the sensor unit 140 of the terminal device 100. The door locking / unlocking system 1 identifies the orientation of the torso based on the positional relationship between the right and left shoulders. For example, the left and right shoulders of each user 11, 12 may be detected by analyzing an image captured by an image sensor of the terminal device 100.

[0028] The left and right shoulders may be detected using other sensors included in the sensor unit 140 of the terminal device 100. For example, the sensor unit 140 may include a stereo vision type imaging sensor capable of identifying the distance to an object. If the distance from the sensor unit 140 to the left and right shoulders is known, the orientation of the torso can be identified based on the distance to the left shoulder and the distance to the right shoulder.

[0029] The sensor unit 140 may include a depth sensor capable of measuring the distance to the shoulders of the users 11 and 12. The depth sensor may be a sensor that irradiates an electromagnetic wave such as a laser beam on an object and detects the object from the time required for the electromagnetic wave to travel back and forth. The depth sensor may be a sensor that irradiates an object with a predetermined pattern of electromagnetic waves such as an infrared laser beam composed of a plurality of spot lights, analyzes the obtained projected image of the predetermined pattern, and detects the users 11 and 12. A sensor called a TOF sensor, a LiDAR sensor, a distance sensor, a 3D sensor, or the like may be used as the depth sensor. The depth sensor can be used to detect the position of the shoulders of the users 11 and 12 in a three-dimensional space, or to measure the direction (angle) and distance of the shoulders from a predetermined origin position, thereby identifying the orientation of the torso. A method for identifying the orientation of the torso of the users 11 and 12 will be described later.

[0030] Fig. 2 is a schematic diagram of Fig. 1 viewed from above. Fig. 2 shows only the faces and torsos of users 11 and 12. Point O shown in Fig. 2 indicates the position that is the origin of the distance and direction measured by the sensor unit 140 of the terminal device 100. The setting position of the origin is not particularly limited, and the origin may be set on the door 200, for example, but the explanation will continue with an example in which the origin is set on the terminal device 100.

[0031] The terminal device 100 continuously captures images of the periphery of the door 200 using the image sensor of the sensor unit 140. The door locking / unlocking system 1 determines whether or not a recognizable face is included in the continuously captured images, and executes face recognition processing when a recognizable face is found. For example, as shown in Fig. 2, the faces of users 11 and 12 are oriented as indicated by dashed arrows 21a and 22a, and face recognition processing is executed.

[0032] In this way, the face recognition process is executed on the condition that a face image that can be recognized is acquired, regardless of the orientation of the torso or the direction of travel of the users 11 and 12. For this reason, face recognition process may be executed not only for the user 11 entering or exiting through the door 200, but also for the user 12 simply passing in front of the door 200.

[0033] The door locking / unlocking system 1 judges whether the users 11 and 12 will enter or exit through the door 200 based on the orientation of the torsos of the users 11 and 12 who have undergone face authentication. The orientation of the torso of the user 11, indicated by the solid line arrow 21b, coincides with the direction in which the terminal device 100 is located as viewed from the user 11 within a predetermined error range. For this reason, it is judged that the user 11 will enter or exit through the door 200. On the other hand, the orientation of the torso of the user 12, indicated by the solid line arrow 22b, differs from the direction in which the terminal device 100 is located as viewed from the user 12 by more than a predetermined error range. For this reason, it is judged that the user 12 will not enter or exit through the door 200. In other words, it is judged whether the users 11 and 12 will enter or exit through the door 200 based on whether the orientation of the torsos of the users 11 and 12 is facing the door 200.

[0034] For example, the door locking / unlocking system 1 detects the right shoulder R1 and the left shoulder L1 of the user 11 by the sensor unit 140 of the terminal device 100 as shown in FIG. 2. The door locking / unlocking system 1 determines that the direction of a straight line passing through the center point C1 of the line segment connecting the right shoulder R1 and the left shoulder L1 and perpendicular to this line segment is the direction of the torso of the user 11 as shown by the solid arrow 21b. The door locking / unlocking system 1 determines that the user 11 enters or exits through the door 200 based on the fact that the angle α1 between the solid arrow 21b and the line connecting the point C1 and the origin O is equal to or less than a preset threshold value Th. Among the directions of the straight line perpendicular to the line segment connecting the left and right shoulders, the direction of the torso is the direction in which the front part of the torso with the navel is facing in the horizontal plane.

[0035] Similarly, the right shoulder R2 and the left shoulder L2 of the user 12 are detected, and the direction of the straight line that passes through the center point C2 of the line segment connecting the right shoulder R2 and the left shoulder L2 and is perpendicular to this line segment becomes the solid arrow 22b indicating the orientation of the torso. The door locking / unlocking system 1 determines that the user 12 will not enter or exit through the door 200 based on the fact that the angle α2 between the solid arrow 22b and the line connecting the point C2 and the origin O exceeds the threshold value Th.

[0036] In this embodiment, for simplicity, the orientation of the torso is projected onto a horizontal plane (XY plane) for explanation, but the door locking / unlocking system 1 may recognize distance, direction (angle), position, etc. in three-dimensional space and execute related processing.

[0037] 2, a method for identifying the orientation of the torso by detecting both shoulders of users 11 and 12 has been described, but the orientation of the torso may be identified by detecting the chest or the waist. When detecting the chest or waist, the orientation of the torso may be identified by detecting the left and right ends of the chest or waist, similar to the shoulders.

[0038] Also, the method of determining whether or not the user 11, 12 will enter or exit based on the deviation between the direction of the torso of the user 11, 12 and the direction in which the door 200 and the terminal device 100 are located has been described, but the determination may be made based on the deviation between the direction of the torso and the direction of the face. The direction of the face may be identified by analyzing a captured image in which a face that has been face-recognized is obtained. The analysis may be performed using a conventional technique that detects parts such as the eyes, nose, and mouth from a face image. If the direction of the face and the direction of the torso are known, it is possible to determine that the user 11, whose angle between the dashed arrow 21a indicating the direction of the face and the solid arrow 21b indicating the direction of the torso is equal to or smaller than a predetermined threshold, will enter or exit through the door 200, as shown in FIG. 2, for example. On the other hand, in the case of the user 12, in the case where the angle between the dashed arrow 22a indicating the direction of the face and the solid arrow 22b indicating the direction of the torso is greater than a predetermined threshold, it is possible to determine that the user will not enter or exit through the door 200, even if only the face is directed toward the terminal device 100.

[0039] As shown in Fig. 1, the door locking / unlocking system 1 judges whether to lock or unlock the door 200 based on the result of the face recognition process and the result of the entry / exit judgment process (C). The door locking / unlocking system 1 controls the locking / unlocking of the door 200 based on the judgment result (D).

[0040] The door locking / unlocking system 1 unlocks the door 200 on condition that the face authentication process determines that the user 11 is permitted to enter and exit through the door 200, and the entry / exit determination process determines that the user 11 will enter and exit through the door 200. The door locking / unlocking unit 210, which normally locks the door 200 so that it cannot be opened or closed, is controlled by the door locking / unlocking system 1 to unlock the door 200, thereby allowing the door 200 to be opened and closed.

[0041] For example, after the door 200 is unlocked, information indicating that the door 200 is unlocked is displayed on the operation display unit 130 of the terminal device 100. Information indicating the user 11, such as the user ID (identification information) of the user 11, may be displayed on the operation display unit 130 so that the user 11 can know that entry and exit are permitted. The user 11 recognizes from the display on the terminal device 100 that the door 200 has been unlocked, and can enter and exit through the door 200.

[0042] On the other hand, even if a user 12 is determined by face authentication processing to be a person permitted to enter and exit through the door 200 when passing in front of the door 200, the entry / exit determination processing determines that the user 12 will not enter or exit through the door 200, so the door locking / unlocking system 1 does not unlock the door 200.

[0043] When walking toward the door 200 to enter or exit through the door 200, even if the face direction may be changed to various directions as the gaze is shifted to surrounding people or objects, the direction of the torso is highly likely to be directed toward the door 200, which is the destination. The door locking / unlocking system 1 utilizes this to control the locking / unlocking of the door 200 by the locking / unlocking unit 210. This makes it possible to prevent the door 200 from being unlocked when there is no need to unlock it. It is also possible to avoid a situation in which the door 200 is unlocked through face authentication of the user 12, and another user enters or exits through the door 200, causing the anti-passback function to work and preventing the user 12 from entering or exiting through the door 200.

[0044] In this way, the face authentication device including the terminal device 100 executes face authentication processing for authenticating the users 11, 12 who have the authority to enter and exit through the door 200, and a determination processing for determining whether or not the users 11, 12 intend to enter and exit through the door 200. The face authentication device confirms that the face-authenticated user 11 intends to enter and exit through the door 200, and unlocks the door 200. In other words, the face authentication device executes face authentication processing for confirming the authority to use the facility and a determination processing for confirming the intention to use the facility, and can execute control to enable use of the facility based on the results of the face authentication processing and the determination processing.

[0045] Next, a specific configuration example of the door locking / unlocking system 1 will be described. Fig. 3 is a block diagram showing a configuration example of the door locking / unlocking system 1. In the example shown in Fig. 3, the door locking / unlocking system 1 includes, in addition to the terminal device 100 and the door 200 described in Fig. 1 and Fig. 2, a management device 300 communicably connected to the terminal device 100. In the example shown in Fig. 3, the terminal device 100 and the management device 300 constitute a face authentication device.

[0046] When controlling the locking and unlocking of only one door 200, the door locking and unlocking system 1 may be configured to include the terminal device 100 and the door 200 without including the management device 300, as described in Fig. 1 and Fig. 2. When the door locking and unlocking system 1 controls the locking and unlocking of multiple doors 200, it is preferable to use the management device 300 as shown in Fig. 3. Fig. 3 shows only one terminal device 100 and the door 200 connected to the terminal device 100, but when controlling the locking and unlocking of multiple doors 200, a terminal device 100 may be provided corresponding to each door 200 and each terminal device 100 may be connected to the management device 300 so as to be able to communicate with the management device 300.

[0047] The terminal device 100 includes a control unit 110 and a communication unit 120 in addition to the operation display unit 130 and the sensor unit 140 described in Fig. 1 and Fig. 2. The communication unit 120 is used to communicate with external devices such as a management device 300 via a network 2.

[0048] The sensor unit 140 includes an image sensor 141 for acquiring a face image used in face authentication processing, and a depth sensor 142 for acquiring information capable of identifying the orientation of the user's torso used in the entry / exit determination processing. However, the sensor unit 140 may include only the image sensor 141, or may include other sensors in addition to the image sensor 141 and the depth sensor 142.

[0049] The control unit 110 includes an information acquisition unit 111 and a door control unit 112. The information acquisition unit 111 controls the sensor unit 140 to acquire information necessary for face authentication processing and entrance / exit determination processing. The information acquisition unit 111 also acquires information regarding locking / unlocking and opening / closing of the door 200. The door control unit 112 controls the locking / unlocking unit 210 to lock / unlock the door 200. For example, a program corresponding to the control unit 110 is stored in advance in a dedicated storage device, and the program is executed by hardware such as a CPU to realize the functions and operations of the control unit 110, i.e., the information acquisition unit 111 and the door control unit 112.

[0050] 1 and 2, the door 200 includes an opening / closing sensor 220 that detects the opening and closing of the door 200. However, the locking / unlocking unit 210 may include the opening / closing sensor 220. The locking / unlocking unit 210 and the opening / closing sensor 220 are communicatively connected to the terminal device 100. FIG. 3 shows an example in which the terminal device 100 and the door 200 are directly communicatively connected, but the door 200 may include a communication unit and be communicatively connected to the terminal device 100 via the network 2.

[0051] The locking / unlocking unit 210 locks and unlocks the door 200. The door 200 is usually in a locked state locked by the locking / unlocking unit 210 and cannot be opened. As described above, when it is determined to unlock the door 200 based on the results of the face authentication process and the entry / exit determination process, the locking / unlocking unit 210 is controlled by the door control unit 112 of the terminal device 100 to unlock the door 200 and open the door 200. When a predetermined condition is satisfied, such as after a predetermined time has elapsed, the locking / unlocking unit 210 locks the door 200 again, which will be described later. As long as the door 200 can be locked and unlocked by the control from the terminal device 100, the type of the locking / unlocking unit 210 is not particularly limited. For example, a mechanism that has been conventionally called an electric lock, an electromagnetic lock, an electronic lock, or the like may be used as the locking / unlocking unit 210.

[0052] The management device 300 includes a control unit 310, a communication unit 320, and a storage unit 330. For example, a computer device equipped with a touch panel type liquid crystal display or the like as an operation display unit is used as the management device 300. The communication unit 320 is used to communicate with external devices such as the terminal device 100 via the network 2.

[0053] The storage unit 330 is a non-volatile storage device. The storage unit 330 stores authentication information 331 in which information such as a face image used in face authentication processing is registered, and entry / exit information 332 in which status information of users entering and exiting through the door 200 is registered. The status information is information that indicates whether the user is inside or outside the room when the door 200 is an entrance / exit for a room, for example.

[0054] The control unit 310 includes an authentication unit 311, an entry / exit determination unit 312, an unlocking determination unit 313, and an entry / exit management unit 314. The authentication unit 311 acquires an image of a user from the terminal device 100 and executes a face authentication process. The authentication unit 311 may be capable of executing multiple types of authentication processes, such as an authentication process based on a user ID and a password input to the operation display unit 130 of the terminal device 100, in addition to the face authentication process.

[0055] The authentication information 331 in the storage unit 330 includes a registered face image used in the face authentication process of each of multiple users permitted to enter and exit through the door 200, the user ID of each user, a password that each user can use for entering and exiting, etc. Information used in the entry / exit determination process, such as the shoulder width of each user, may also be registered in the authentication information 331, which will be described later.

[0056] The authentication unit 311 receives the captured image acquired by the information acquisition unit 111 of the terminal device 100 from the imaging sensor 141 of the sensor unit 140 via the network 2, and executes face authentication processing using the captured image. If the captured image contains a human face, the authentication unit 311 compares and collates it with each registered face image registered in the authentication information 331. If the authentication information 331 contains a registered face image that matches the face of a person captured by the terminal device 100, the person is authenticated and determined to be a user who is permitted to enter and exit through the door 200.

[0057] The entrance / exit determination unit 312 receives information for identifying the orientation of the torso, acquired by the information acquisition unit 111 of the terminal device 100 from the sensor unit 140, via the network 2, and executes an entrance / exit determination process using the information. The entrance / exit determination unit 312 is a determination unit that executes a determination process for determining whether or not a user enters or exits through the door 200, i.e., whether or not the user uses the facility. The entrance / exit determination process is performed as described in Fig. 1 and Fig. 2, and the determination method will be described later.

[0058] The unlocking determination unit 313 determines whether or not to unlock the door 200 based on the authentication result by the authentication unit 311 and the determination result by the entry / exit determination unit 312. When a user who has been authenticated by the face authentication process and confirmed to be a person permitted to enter and exit through the door 200 is determined to enter and exit through the door 200 in the entry / exit determination process, the unlocking determination unit 313 determines to unlock the door 200. If there is no person authenticated by the face authentication process, no determination is made by the unlocking determination unit 313. Even if a user is authenticated by the face authentication process, when it is determined by the entry / exit determination process that this user will not enter or exit through the door 200, the unlocking determination unit 313 determines not to unlock the door 200.

[0059] The unlocking determination unit 313, which has determined to unlock the door 200, transmits an unlocking command to the terminal device 100. In the terminal device 100 which has received the unlocking command via the network 2, the door control unit 112 controls the locking / unlocking unit 210 of the door 200, thereby unlocking the door 200. The door control unit 112 is a control unit which controls the locking / unlocking of the door 200, i.e., whether or not the facility can be used.

[0060] The entrance / exit management unit 314 registers and manages the status information of users who enter and exit through the door 200 in the entrance / exit information 332 in the storage unit 330. When a user enters and exits by opening and closing the unlocked door 200, the information acquisition unit 111 of the terminal device 100 detects this via the opening / closing sensor 220 of the door 200. The information acquisition unit 111 transmits information indicating that the door 200 has been opened and closed, i.e., that the user has entered and exited, to the management device 300. The entrance / exit management unit 314 of the management device 300 receives this information via the network 2 and updates the user's status information registered in the entrance / exit information 332.

[0061] Specifically, if the door 200 is an entrance / exit to a room, when a user opens and closes the door 200 to enter the room, the user's information is updated from "exit" to "enter". When a user opens and closes the door 200 to leave the room, the user's information is updated from "enter" to "exit". Using this, the management device 300 can also be set to execute anti-passback control. Note that the anti-passback control referred to in this embodiment refers to control that limits users who can enter through the door 200 to only users whose entry / exit information is in the "exit" state, and limits users who can exit through the door 200 to only users whose entry / exit information is in the "enter" state.

[0062] In the case where the anti-passback control is set, for example, the unlocking determination unit 313 determines whether or not to unlock the door 200 based on the result of the face authentication process and the result of the entry / exit determination process, as well as the status information of the user registered in the entry / exit information 332. For example, when it is determined by the face authentication process and the entry / exit determination process that the user is face authenticated and enters the room, the unlocking determination unit 313 decides to unlock the door 200 on the condition that the status information of the user in the entry / exit information 332 is "exited". In other words, when the status information of this user is already "entered", even if the result of the face authentication process and the entry / exit determination process indicates that the door 200 can be unlocked, the unlocking determination unit 313 determines not to unlock the door 200. Note that the anti-passback control may be performed by changing the registered face image used by the authentication unit 311 in the face authentication process. Specifically, anti-passback control can be achieved by performing facial recognition processing of persons entering the room using only registered facial images with status information of "Leaving the room," and performing facial recognition processing of persons leaving the room using only registered facial images with status information of "Entering the room."

[0063] Next, an example of a determination method in the entry / exit determination process will be described. As long as it is possible to identify the orientation of the user's torso and determine whether or not this orientation is in the direction in which the terminal device 100 and the door 200 are located as seen from the user, the method of the entry / exit determination process is not particularly limited, but an example of the determination method will be described below with reference to Figs. 4 to 6.

[0064] Fig. 4 is a diagram for explaining the entrance / exit determination process performed using the image sensor 141 of the terminal device 100. When the entrance / exit determination process is performed by the method shown in Fig. 4, as long as the sensor unit 140 of the terminal device 100 includes the image sensor 141, it does not need to include other sensors.

[0065] When the image sensor 141 of the terminal device 100 obtains an image 400 including a face image capable of executing face recognition processing, the authentication unit 311 of the management device 300 cuts out a face image 410 from the image 400 as shown in Fig. 4 and executes face recognition processing. The image 400 may be one of still images continuously captured by the image sensor 141, or may be a still image extracted from a moving image captured by the image sensor 141.

[0066] The face authentication process is performed using authentication information 331 in the storage unit 330. A registered face image is registered in the authentication information 331 in association with the user ID of each user. When the entry / exit determination process is performed using the image 400, information indicating the shoulder width of each user is registered in advance in the authentication information 331 in association with the user ID.

[0067] A face image 410 obtained by the terminal device 100 installed near the door 200 is compared and matched with a registered face image included in the authentication information 331 to confirm whether or not the person in the face image 410 is a user permitted to enter and exit through the door 200. When the authentication unit 311 performs face authentication on the user and confirms that the user is permitted to enter and exit through the door 200, the shoulder width L0 of the user registered in the authentication information 331 is identified.

[0068] The entry / exit determination unit 312 identifies the orientation of the user's torso based on the shoulder width of the user shown in the image 400 and the shoulder width that would appear in the image 400 if the user were facing the terminal device 100 directly at this imaging position.

[0069] Specifically, since the installation position of the terminal device 100 is fixed, the position of the user can be identified by analyzing the image 400. For example, the position of the user when the image 400 is captured can be identified from information on the walls, floors, objects, etc. around the user in the image 400. In order to make it easier to identify the position of the user from the image 400, markings used for position identification may be applied to the floors and walls around the door 200, and the position of the user may be identified based on the markings in the image 400. If the position of the user is known from the image 400, the shoulder width of the user that appears in the image when the user faces the image sensor 141 at that position can be calculated based on the positional relationship between the terminal device 100 and the user and the shoulder width L0 of the user registered in the authentication information 331.

[0070] For example, as shown in FIG. 4, the entry / exit determination unit 312 calculates the shoulder width P1 that appears in the image when the user is facing directly toward the image sensor 141, i.e., the installation direction of the terminal device 100, as the number of pixels on the image 400.

[0071] The entry / exit determination unit 312 detects the shoulder width P2 of the user actually captured in the image 400 by the number of pixels. In the image 400, the shoulder width, which should be P1 pixels if the user faces the direction of the image sensor 141, is actually captured by P2 pixels, so that the angle α can be calculated as shown in FIG. 4. The entry / exit determination unit 312 calculates the angle α by using an inverse trigonometric function from the pixel numbers P1 and P2 indicating the shoulder width. The entry / exit determination unit 312 specifies the orientation of the torso based on the calculated angle α as described above and executes the entry / exit determination process. It goes without saying that the angle α may be calculated by using the dimension indicating the shoulder width instead of the number of pixels.

[0072] 5 is a diagram for explaining the entry / exit determination process performed using the depth sensor 142 of the terminal device 100. When the image sensor 141 of the terminal device 100 obtains an image 400 including a face image on which face recognition processing can be performed, the authentication unit 311 of the management device 300 performs face recognition processing in the same manner as described in FIG.

[0073] The entry / exit determination unit 312 uses the depth sensor 142 to measure the distance d1 to the right shoulder R11 of the user 421 who has passed face authentication and the distance d2 to the left shoulder L11, as shown in FIG. 5(a). When the user 421 is facing directly in the installation direction of the terminal device 100 as seen from the user 421, as shown in FIG. 5(a), the distance d1 to the right shoulder R11 is equal to the distance d2 to the left shoulder L11 (d1 = d2).

[0074] Specifically, when the user 421 is facing the terminal device 100, a perpendicular line passing through the center point C of the line segment connecting the right shoulder R11 and the left shoulder L22 passes through the origin O set in the terminal device 100, and the origin O, the right shoulder R11, and the left shoulder L11 form an isosceles triangle relationship.

[0075] On the other hand, when the body of the user 421 faces left from the facing state shown in FIG. 5(a), as shown in FIG. 5(b), the distance d2 to the left shoulder L11 is greater than the distance d1 from the origin O to the right shoulder R11 (d1 < d2). Similarly, when the body of the user 421 faces right from the facing state, as shown in FIG. 5(c), the distance d1 is greater than the distance d2 (d1 > d2).

[0076] The entry / exit determination unit 312 determines whether the user 421 enters or exits from the door 200 based on the positional relationship between the right shoulder R11 and the left shoulder L11 of the user 421 and the origin O shown in FIGS. 5(a) to 5(c). Specifically, according to the position of the user 421, the relationship between the distance d1 and the distance d2 for determining that the user 421 enters or exits from the door 200 is set in advance as a determination condition, and the entry / exit determination is executed based on this determination condition.

[0077] For example, in the example shown in FIG. 5, based on the positional relationship shown in FIG. 5(a), when the distance d1 and the distance d2 match within a predetermined error range, the user 421 is determined to be a person entering or exiting from the door 200.

[0078] As shown in FIG. 5(c), when distance d1 is greater than distance d2, if the difference between distances d1 and d2 exceeds a predetermined threshold, entry / exit determination unit 312 determines that user 421's torso is not facing door 200 and user 421 will not enter or exit through door 200, i.e., is a non-entering or exiting person.

[0079] Even when the distance d1 is smaller than the distance d2 as shown in Fig. 5(b), if the difference between the distances d1 and d2 exceeds a predetermined threshold, the entry / exit determination unit 312 determines that the user 421 is a non-enterer, but a threshold different from that when the distance d1 is larger than the distance d2 as shown in Fig. 5(c) is set. Specifically, even if the distance d1 is smaller than the distance d2, the threshold is set in consideration of the range in which the user 421 can be considered to be heading toward the door 200 as shown in Fig. 5(b).

[0080] Even if the absolute value of the difference between the distance d1 and the distance d2 is approximately the same, depending on which of the distances d1 and d2 is larger, the user 421 may be determined as an entering or exiting person as shown in Fig. 5(b) or as a non-entering or exiting person as shown in Fig. 5(c). For this reason, depending on the positional relationship between the user 421 and the door 200 and depending on which of the distance d1 to the right shoulder R11 and the distance d2 to the left shoulder L11 is larger, the relationship between d1 and d2 for determining the user 421 as an entering or exiting person is set as a determination condition. Note that the method of determination based on the distances d1 and d2 is not limited to the method of using the difference, and may be, for example, a mode of determination based on the ratio of the distances d1 and d2.

[0081] The entry / exit determination unit 312 may use the distances d1 and d2 to calculate an angle indicating the orientation of the torso of the user 421, and then execute the entry / exit determination process. FIG. 6 is a diagram for explaining a method for calculating an angle indicating the orientation of the torso of the user 421. FIG. 6 shows a state in which the torso of the user 421 is facing rightward from the front facing state shown in FIG. 5(a). The right shoulder R12 and left shoulder L12 of the user 421 shown in FIG. 6 are located at positions obtained by rotating the right shoulder R11 and left shoulder L11 in a clockwise direction around the center point C of the line segment between the right shoulder R11 and left shoulder L11 shown in FIG. 5(a).

[0082] As in the case of Fig. 5, the entry / exit determination unit 312 acquires the distance d1 to the right shoulder R12 and the distance d2 to the left shoulder L12 of the user using the depth sensor 142 of the terminal device 100, and specifies angles β1 and β2 in the directions in which the right shoulder R12 and the left shoulder L12 are located as viewed from the origin O, as shown in Fig. 6. In Fig. 6, the positive side of the Y axis perpendicular to the front surface of the terminal device 100 in the horizontal plane is set to 0 degrees, and the angles from this point are β1 and β2.

[0083] As shown in Figure 6, if the position of the right shoulder R11, which is at a distance d1 from point O in the direction of angle β1, and the position of the left shoulder L12, which is at a distance d2 from point O in the direction of angle β2, can be identified, the perpendicular line Lp passing through the center point C of the line segment connecting these two points can be identified.

[0084] For example, the entry / exit determination unit 312 executes the entry / exit determination process using the angle α between the line Lc that passes through points O and C and the line Lp as the orientation of the torso of the user 421. When the angle α is included within a predetermined angle range, the entry / exit determination unit 312 determines that the user 421 is an entry / exit person heading toward the door 200. In other words, when the angle α is not included within the predetermined angle range, the entry / exit determination unit 312 determines that the user 421 is not heading toward the door 200 and is a non-entering / exiting person.

[0085] Depending on the position of the user 421, that is, based on the positional relationship between the user 421 and the door 200, an angle range for determining the user 421 as an entering or exiting person may be set in advance as a determination condition within a range of 360 degrees in a horizontal plane centered on point C, and the entering or exiting determination process may be performed based on the determination condition. It goes without saying that the determination may be performed using the angle change from when the line segment connecting both shoulders is directly facing each other, specifically, the angle between the line segment connecting points R11 and L11 and the line segment connecting points R12 and L12.

[0086] In this way, in the door locking / unlocking system 1, the orientation of the torso of the user 421 when the user 421 is authenticated by the face authentication process is identified, and based on this orientation, it is determined whether the user 421 is an entering or leaving person. The determination may be performed using the image sensor 141 as described in FIG. 4, or may be performed using the depth sensor 142 as described in FIG. 5 and FIG. 6. When performing the method of FIG. 4, the image sensor 141 may be used. When performing the method of FIG. 5, a sensor capable of measuring the distance to the right shoulder and left shoulder of the user 421 may be used as the depth sensor 142. When performing the method of FIG. 6, a sensor capable of measuring the positions of the right shoulder and left shoulder of the user 421, i.e., the distance and angle to the right shoulder and left shoulder, may be used as the depth sensor 142.

[0087] The entry / exit determination process is not limited to the method described in Fig. 4 to Fig. 6. For example, if the sensor unit 140 includes a depth sensor that can irradiate a user with a laser light of a predetermined pattern consisting of multiple spot lights, detect the user's skeleton, and output information regarding the orientation of the torso, the entry / exit determination process may be performed by directly using the output value of this sensor.

[0088] Next, an example of the door locking / unlocking process executed by the door locking / unlocking system 1 shown in Fig. 3 will be described. Fig. 7 is a flowchart showing an example of the door locking / unlocking process. The door locking / unlocking system 1 monitors the area around the door 200 using a management device 300 and a terminal device 100 installed near the door 200.

[0089] Specifically, the authentication unit 311 of the management device 300 continues to monitor whether or not a recognizable face is included in images continuously captured by the image sensor 141 of the terminal device 100 around the door 200. When a recognizable face is detected in the captured images obtained by the terminal device 100, the process shown in Fig. 7 is started, and the authentication unit 311 executes face recognition processing (step S1).

[0090] If the detected face is not authenticated by the face recognition process (step S2; No), that is, if the face detected from the image captured by the terminal device 100 is not the face of a user permitted to enter or exit through the door 200, the processing related to the detected face is terminated.

[0091] If the detected face is authenticated (step S2; Yes), that is, if the face detected from the captured image is the face of a user who is permitted to enter or exit through the door 200, then the entry / exit determination unit 312 executes an entry / exit determination process (step S3).

[0092] In the entry / exit determination process, the orientation of the user's torso is identified as described above using information acquired by the image sensor 141 and the depth sensor 142, and it is determined whether or not the user is entering or exiting through the door 200 based on the orientation of the torso.

[0093] If the face-authenticated user is determined by the entry / exit determination process to not be a person entering or exiting through the door 200 (step S4; No), the process ends. If the face-authenticated user is determined to be a person entering or exiting through the door 200 (step S4; Yes), the unlocking determination unit 313 determines to unlock the door 200 and transmits an unlocking command to the terminal device 100. The door control unit 112 of the terminal device 100 that has received the unlocking command controls the locking / unlocking unit 210 of the door 200 to unlock the door 200 (step S5).

[0094] When the unlocked door 200 is opened or closed (step S6; Yes), the opening / closing sensor 220 of the door 200 detects this. In response to the detection result of the opening / closing sensor 220, the door control unit 112 controls the locking / unlocking unit 210 of the door 200 to lock the unlocked door 200 again (step S8). In this way, after the door 200 is unlocked, the face-authenticated user opens and closes the door 200 to enter or exit, whereby the door 200 is locked and the process ends.

[0095] If the opening / closing sensor 220 does not detect the opening / closing of the door 200 (Step 6; No), the door 200 remains unlocked and in a standby state until a predetermined time has passed since unlocking (Step S7; No). If the predetermined time has passed since unlocking without the door 200 being opened or closed (Step S7; Yes), the door control unit 112 controls the locking / unlocking unit 210 of the door 200 to lock the door 200 again (Step S8), and the process ends.

[0096] Fig. 8 is a flowchart when the process shown in Fig. 7 is executed with the setting using the anti-passback function. The process shown in Fig. 8 differs from Fig. 7 only in that the process of step S9 related to the anti-passback control is executed when the opening or closing of the door 200 is detected in step S6.

[0097] As shown in FIG. 8, when the anti-passback function is used, when the door 200 is opened or closed after being unlocked (step S6; Yes), entry / exit information indicating this is transmitted from the terminal device 100 to the management device 300. In the management device 300 that has received the entry / exit information, a process related to the anti-passback control is executed (step S9). Specifically, the entry / exit management unit 314 switches the status information of the user registered in the entry / exit information 332 of the storage unit 330 between "enter" and "exit." When a face-authenticated user enters a room through the door 200, the status information of this user is updated from "exit" to "enter," and when the user leaves the room, the status information is updated from "enter" to "exit."

[0098] In addition, in the setting using the anti-passback function, the unlocking of the door 200 is controlled based on the status information of entering and leaving the room. For example, when performing face authentication processing of an entering person in step S1, face authentication processing is performed only on the registered face images of users whose status information is "leaving" among the registered face images registered in the authentication information 331 of the storage unit 330. Similarly, when performing face authentication processing of an leaving person in step S1, face authentication processing is performed only on the registered face images of users whose status information is "entering" among the registered face images registered in the authentication information 331. This realizes the anti-passback function that allows only users whose status information is in the leaving state to enter the room and only users whose status information is in the entering state to leave the room. Also, for example, as described in FIG. 3, even when the anti-passback function is used, face authentication processing may be performed on all registered face images in step S1, and the unlocking determination unit 313 may determine whether or not to unlock the door 200 in step S5 based on the results of the face authentication processing and the entry / exit determination processing and the status information.

[0099] Even when the anti-passback function is used, if a user passes in front of the door 200, the door 200 is not unlocked, thereby preventing a situation in which another user enters or exits through the unintentionally unlocked door 200, preventing the passing user from being able to enter or exit.

[0100] In FIG. 7, an example in which the entrance / exit determination process is executed following the face authentication process has been described, but the entrance / exit determination process may be executed in parallel with the face authentication process. For example, the entrance / exit determination unit 312 may execute the entrance / exit determination process described in FIG. 4 in parallel with the face authentication process by the authentication unit 311, using the captured image obtained by the terminal device 100. For example, in parallel with the image capture by the image sensor 141, the depth sensor 142 may acquire depth information of each person included in the captured image, and the face authentication process by the authentication unit 311 and the entrance / exit determination process by the entrance / exit determination unit 312 described in FIG. 5 or FIG. 6 may be executed in parallel. In this case, if a user who is permitted to enter and exit is found by the face authentication process, the unlocking determination unit 313 may determine whether or not the door 200 can be unlocked based on the result of the entrance / exit determination process performed for this user.

[0101] After a user is authenticated by face authentication processing, an entry / exit determination process may be performed while tracking the user. This will be specifically described with reference to Figures 9 and 10. Figures 9 and 10 show the process when the anti-passback function is used.

[0102] Fig. 9 is a flowchart showing a process flow when the door locking / unlocking system 1 executes the entry / exit determination process while tracking a user. When a face that can be recognized is detected in an image obtained by the terminal device 100, the process shown in Fig. 9 is started, and the authentication unit 311 executes the face recognition process (step S11).

[0103] Since the anti-passback function is set to be used, facial recognition processing is performed using only registered facial images, among those registered in authentication information 331, whose status information in entry / exit information 332 meets the conditions. Facial recognition processing to detect a person permitted to enter through door 200 is performed only on registered facial images of users whose status information is "exited". Similarly, facial recognition processing to detect a person permitted to exit through door 200 is performed only on registered facial images of users whose status information is "entered". Since the processing performed for a user entering the room and the processing performed for a user leaving the room are the same, the following description will be given taking as an example a case where a face-authenticated user enters the room.

[0104] If the face detected from the image captured by the terminal device 100 is not facially authenticated (step S12; No), that is, if the detected face is not the face of a user who is allowed to enter through the door 200, processing related to the detected face is terminated.

[0105] If the detected face is face-authenticated (step S12; Yes), that is, if the detected face is the face of a user who is allowed to enter through door 200, then entry / exit determination processing is executed by entry / exit determination unit 312 (step S13).

[0106] Information acquired by the image sensor 141 and the depth sensor 142 of the terminal device 100 is transmitted to the management device 300, so that the management device 300 can execute face authentication processing and entrance / exit determination processing in real time.

[0107] When a user is authenticated by the face recognition process, tracking of the user is started, and an entry / exit determination process is executed based on information obtained by tracking the user. The entry / exit determination unit 312 can execute the entry / exit determination process for a predetermined time after the user's face is authenticated.

[0108] The image sensor 141 and the depth sensor 142 of the terminal device 100 execute the entry / exit determination process multiple times while tracking the user. For example, the entry / exit determination process may be executed repeatedly at a predetermined time interval, such as every second. For example, the entry / exit determination process may be executed repeatedly by executing the entry / exit determination process when at least one of the image captured by the image sensor 141 and the depth information detected by the depth sensor 142 changes.

[0109] When a user's face is authenticated, even if the entrance / exit determination process determines that the user will not enter through door 200 (step S14; No), the entrance / exit determination process is repeatedly executed until a predetermined time has elapsed since the face is authenticated (step S15; No). If a determination result that the user will enter through door 200 is not obtained within the predetermined time (step S15; Yes), the process ends.

[0110] If a determination result is obtained that the user will enter through door 200 within a predetermined time after face authentication of the user (step S14; Yes), unlocking determination unit 313 determines that the door can be unlocked, and door 200 is unlocked (step S16).

[0111] When a face-authenticated user enters the room through door 200, the opening and closing of the unlocked door 200 is detected (step S17; Yes), and processing related to anti-passback control is executed (step S18). Next, door 200 is locked (step S20), and the processing ends. In the processing of step S18, due to the user's entry, the status of the face-authenticated user is updated from "exit" to "enter" in entry / exit information 332 stored in memory unit 330 of management device 300.

[0112] If the door 200 is not opened or closed (Step S17; No), the door 200 remains unlocked and in a standby state until a predetermined time has passed since unlocking (Step S19; No). If the predetermined time has passed since unlocking without the door 200 being opened or closed (Step S19; Yes), the door 200 is locked again (Step S20), and the process ends. In this case, since the user did not enter the room, the process of Step S18 relating to the anti-passback control is not executed, and the user's status information remains "exited room".

[0113] Although the description has been given taking an example of a person entering through the door 200, the same processing is performed in the case of a person leaving the room. Thus, in the example shown in Fig. 9, while tracking a face-authenticated user, the entry / exit determination processing is performed based on information acquired by the sensor unit 140, and if the user is determined to be an entry / exit person within a predetermined time after face authentication, the door 200 is unlocked. For example, even if the orientation of the user's torso at the time of face authentication is such that the user is determined to be a non-entering / exiting person, if the user moves toward the door 200 and the terminal device 100 within a predetermined time after face authentication, the user is determined to be an entry / exiting person, and the door 200 is unlocked.

[0114] 9, an example in which the anti-passback function is used has been described, but the anti-passback function can also be set not to be used. In this case, the face recognition process in step S11 is performed on the registered face images of all persons permitted to enter and exit the door 200, regardless of the status information in the entry / exit information 332, and the process in step S18 is not performed even if the door 200 is opened or closed after being unlocked.

[0115] 9 shows an example in which entry and exit through the door 200 is possible until a predetermined time has elapsed since the door 200 was unlocked, but the start point of the time limit for entry and exit may be the time point at which face authentication is performed. In this case, the time limit for tracking a face-authenticated user and executing the entry / exit determination process and the time limit for entry and exit through the unlocked door 200 may be calculated individually for each face-authenticated user.

[0116] Fig. 10 is a flowchart showing a process flow when the door locking / unlocking system 1 determines a time limit for each face-authenticated user. In the process shown in Fig. 10, the time T1 at which the entry / exit determination process can be executed and the time T2 at which the door 200 can be unlocked after the face of the user is authenticated are calculated according to the position of the user.

[0117] When a face that can be recognized is detected in a captured image obtained by the terminal device 100, the process shown in Fig. 10 is started, and the authentication unit 311 executes face recognition processing (step S31). Since the anti-passback function is set to be used, face recognition processing is executed using only registered face images that meet the conditions of status information in the entry / exit information 332, among registered face images registered in the authentication information 331 of the management device 300, as in Fig. 9. Below, as in Fig. 9, the explanation will be continued by taking as an example a case where a user whose face has been recognized enters a room.

[0118] If the face detected from the captured image by the terminal device 100 is not face-authenticated (step S32; No), the process for the detected face ends. If the detected face is face-authenticated (step S32; Yes), the entry / exit determination time limit T1 and the unlocking time limit T2 are calculated (step S33).

[0119] Specifically, as described in Fig. 4 to Fig. 6, the entry / exit determination unit 312 uses the image captured by the imaging sensor 141 and the depth information by the depth sensor 142 to identify the position of the user when the face image subjected to face authentication was captured. The entry / exit determination unit 312 identifies the distance from the position of the user to the terminal device 100. The entry / exit determination unit 312 determines the times T1 and T2 based on the identified distance. For example, the time T1 is calculated based on the identified distance and a preset speed, and a predetermined time is added to the time T1 to calculate the time T2.

[0120] Specifically, for example, the entrance / exit determination unit 312 determines T1 as the time it takes for the user to walk the distance from the standing position of the user at the time of face authentication to the terminal device 100 at a predetermined walking speed (e.g., 3 km / h). The entrance / exit determination unit 312 determines T2 as the time T1 plus a predetermined time (e.g., 5 seconds) set in consideration of the time required to open and close the door 200 and pass through.

[0121] After the times T1 and T2 are determined, the process proceeds in the same manner as in Fig. 9, and the entry / exit determination unit 312 executes the entry / exit determination process (step S34). The entry / exit determination process is repeatedly executed until the time elapsed since the user's face was authenticated reaches the previously determined time T1.

[0122] Even if the entrance / exit determination process determines that the face-authenticated user will not enter through door 200 (step S35; No), the entrance / exit determination process is repeatedly executed until time T1 has elapsed since face authentication (step S36; No). If a determination result that the user will enter through door 200 is not obtained before time T1 has elapsed since face authentication (step S36; Yes), the process ends.

[0123] If a determination is made that the user will enter through door 200 before time T1 has elapsed since the user's face was authenticated (step S35; Yes), the unlocking determination unit 313 determines that the door can be unlocked, and the door 200 is unlocked (step S37).

[0124] When a face-authenticated user enters through the door 200, the opening and closing of the unlocked door 200 is detected (step S38; Yes), and processing related to anti-passback control is executed (step S39). Then, the door 200 is locked (step S41), and the processing ends. The processing related to anti-passback control is executed as described in FIG. 9.

[0125] If the door 200 is not opened or closed (step S38; No), the door 200 remains unlocked and in a standby state until the time T2 has elapsed since the face authentication (step S40; No). If the door 200 is not opened or closed and the time T2 has elapsed since the face authentication (step S40; Yes), the door 200 is locked again (step S41), and the process ends.

[0126] Although the above description has been given taking an example of a person entering through door 200, the same processing is performed for a person leaving the room. In the example shown in Fig. 10, as described in Fig. 9, if the anti-passback function is not used, the face recognition processing in step S31 is performed on the registered face images of all persons permitted to enter and exit door 200, regardless of the status information in entry / exit information 332, so that the processing in step S39 is not performed even if the unlocked door 200 is opened or closed.

[0127] 10, the process proceeds based on the time T1 required for the user to move from the face authentication to the location where the terminal device 100 and the door 200 are installed, and the time T2 required for the user to open and close the door 200 and pass through. The door 200 is usually locked for security reasons, and it is not preferable for the unlocked state to continue longer than necessary. The execution time of the entry / exit determination process, which is a condition for unlocking the door 200, is limited based on the time T1 required for the user to reach the door 200 from the face authentication position, and the unlocking time of the door 200 is limited based on the time T2 required for the user to open and close the door 200 and enter and exit, thereby minimizing the time for unlocking the door 200 and maintaining security.

[0128] In the present embodiment, an example has been described in which the origin O used for identifying the orientation of the torso and for judging the orientation of the torso is set in the terminal device 100 in the entry / exit judgment process performed using the depth sensor 142, but the origin O may be set to the door 200. For example, in Figs. 2, 5 and 6, the origin O may be set to the door 200. Depending on the position of the set origin O, a judgment condition for judging whether or not there is an intention to enter or exit through the door 200 may be set, and the judgment process may be performed as described above.

[0129] In the present embodiment, an example has been described in which the terminal device 100 has the sensor unit 140, but some of the sensors included in the sensor unit 140 may be installed at a position away from the terminal device 100. For example, the depth sensor 142 may be prepared as a separate module and connected to the terminal device 100 for use.

[0130] In FIG. 3 of the present embodiment, an example in which the door locking / unlocking system 1 is configured by the management device 300 and the terminal device 100 connected to the door 200 is shown, but this configuration is a functional schematic, and the configuration of the door locking / unlocking system 1 is not limited to this physical configuration. For example, in the example shown in FIG. 3, the terminal device 100 may realize part or all of the functions and operations of the management device 300 described above. The management device 300 may realize part of the functions and operations of the terminal device 100 described above. For example, the door 200 may be connected to a network 2, and the management device 300 may directly control the locking and unlocking of the door 200 via the network 2. The form of distribution / integration of each device is not limited to the above example, and all or a part of them may be functionally or physically distributed / integrated in any unit according to various loads, usage conditions, etc.

[0131] In this embodiment, an example has been described in which a face authentication device executes a face authentication process for confirming the user's entry / exit authority and a judgment process for judging whether the user enters / exits through the door 200, using an example of a user entering / exiting through the door 200, but the use of the face authentication device is not limited to locking / unlocking control of the door 200. The face authentication device may be used to control whether a device such as a computer device can be used. In this embodiment, an example has been described in which the door 200 is locked / unlocked, but if the equipment to be used is a device, the device can be controlled whether it can be used. The control of making the device available or locking the device to make it unavailable can be performed using a conventionally known technology. The face authentication device confirms that the user has the authority to use the device through face authentication processing, and judges whether the face-authenticated user is approaching the device or simply passing in front of the device through a judgment process, and executes control related to the availability of the device based on the judgment result. As a result, it is possible to control whether the device can be used based on the user's authority to use and will to use it, as in the above example.

[0132] In this embodiment, an example has been described in which anti-passback control is performed both when entering a room through the door 200 and when exiting the room through the door 200, but anti-passback control may be performed only when entering the room or only when exiting the room. Furthermore, the door 200 on which anti-passback control is performed is not limited to an example in which the door 200 is an entrance to a room, and the door 200 may be an entrance to a building. That is, the door 200 may be an entrance from the outside to the inside of a building, or an entrance to a specified space called a room, room, facility, equipment, etc. provided inside a building.

[0133] As described above, according to the door locking / unlocking system of the present embodiment, not only is a face authentication process performed to detect a user who is permitted to enter or exit through the door, but an entry / exit determination process is also performed to determine whether or not the face-authenticated user will enter or exit through the door. This makes it possible to control the door so that it is not unlocked when a user who is permitted to enter or exit through the door simply passes in front of the door, and to avoid the door being unlocked at a time unintended by the user. For equipment such as a door or device, the face authentication device performs a face authentication process to confirm whether or not the user has the authority to use the equipment and a determination process to determine whether or not the user has the intention to use the equipment, thereby making it possible to detect a user who has the intention to use the equipment and appropriately control whether or not the user can use the equipment. [Industrial Applicability]

[0134] INDUSTRIAL APPLICABILITY As described above, the face recognition device, face recognition method, and face recognition program according to the present disclosure are useful for detecting users who intend to use a facility and controlling whether or not they can use the facility. [Explanation of symbols]

[0135] 1 Door locking and unlocking system 2 Network 100 Terminal Equipment 110, 310 Control unit 111 Information Acquisition Department 112 Door control section 120, 320 Communications Department 130 Operation display section 140 Sensor section 141 Image Sensor 142 Depth Sensor 200 Doors 210 Locking / unlocking section 220 Open / Close Sensor 300 Management device 311 Authentication Department 312 Entry / exit determination section 313 Unlocking Judgment Unit 314 Entrance / exit management department 330 Storage section

Claims

1. A face recognition device that performs face recognition processing to confirm the use authority of a facility, A sensor unit including at least one sensor that acquires an image of the periphery of the facility and information regarding a direction of a torso of a person included in the image; an authentication unit that executes a face authentication process to detect a person's face from the image obtained by the sensor unit and determine whether or not the face is a face of a person authorized to use the facility; a determination unit that identifies a direction of a torso of the person captured by the sensor unit from the information obtained by the sensor unit, and executes a determination process to determine whether or not the person will use the facility based on the direction of the torso; a control unit that executes control regarding whether or not the facility is available based on results of the face authentication process and the determination process; Including, On condition that a person who has been determined to have the right to use the facility in the face authentication process is determined to use the facility in the determination process, control is executed to allow the person to use the facility. A face recognition device characterized by:

2. The sensor unit includes: A first sensor that captures an image of the surroundings of the facility; a second sensor for acquiring information for identifying the left and right shoulders of the person imaged by the first sensor; Including, The authentication unit detects a face from an image captured by the first sensor and executes the face authentication process, The determination unit identifies the left and right shoulders of the person whose face has been authenticated from the information acquired by the second sensor, and determines the orientation of the torso based on the left and right shoulders to execute the determination process. The face authentication device according to claim 1 .

3. The face recognition device according to claim 2, characterized in that the determination unit determines the distance and direction from a predetermined origin position set in three-dimensional space to the right shoulder and the distance and direction to the left shoulder of the person whose face has been recognized, and determines the orientation of the person's torso based on the positions of the right shoulder and the left shoulder.

4. The sensor unit includes a first sensor that captures an image of the periphery of the facility, the authentication unit detects a face from an image captured by the first sensor and executes the face authentication process; The determination unit calculates, from the shoulder width of the person whose face has been authenticated, a shoulder width of the person that should appear in the image captured by the first sensor when the person faces the first sensor, and specifies the orientation of the person's torso based on the calculated shoulder width and the shoulder width of the person captured in the captured image, and executes the determination process. The face authentication device according to claim 1 .

5. The face recognition device according to any one of claims 1 to 4, characterized in that the determination unit determines whether or not the person who has been face-recognized will use the facility based on the direction in which the facility is located as viewed from the person and the orientation of the torso, with the orientation of the torso being determined to be a direction perpendicular to a straight line connecting the left and right shoulders of the person who has been face-recognized.

6. The face recognition device according to any one of claims 1 to 4, characterized in that the determination unit executes the determination process for a person whose face has been recognized until a predetermined time has elapsed since the face recognition.

7. The face recognition device according to claim 6, characterized in that the determination unit identifies a position of a person whose face has been recognized, calculates a time required for the person to travel from the identified position to the facility, and executes the determination process for the person until the time has elapsed.

8. a sensor unit including at least one sensor that acquires an image of the periphery of the door locked by the locking / unlocking unit and information regarding the orientation of the torso of a person included in the image; an authentication unit that executes a face authentication process to detect a person's face from the image obtained by the sensor unit and determine whether or not the face is that of a person permitted to enter or exit through the door; a determination unit that identifies a direction of a torso of the person captured by the sensor unit from the information obtained by the sensor unit, and executes a determination process to determine whether or not the person enters or exits through the door based on the direction of the torso; a door control unit that controls the locking / unlocking unit to unlock the door on condition that a person determined in the face authentication process to be a person permitted to enter and exit through the door is determined in the determination process to enter and exit through the door; an entrance / exit management unit that manages entrance / exit information of a person who has been face-recognized through the door based on whether the door has been opened or closed after the door has been unlocked; Including, The entrance / exit management unit performs anti-passback control of the door based on the entrance / exit information managed by the entrance / exit management unit. A face recognition device characterized by:

9. A facial recognition method executed by a facial recognition device to confirm an authority to use a facility, comprising: A step of executing a face recognition process for detecting a person's face from a captured image around the facility and determining whether or not the face is the face of a person who has authority to use the facility; acquiring information about a direction of a torso of the person included in the captured image to identify the direction of the torso of the person, and executing a determination process to determine whether or not the person will use the facility based on the direction of the torso; executing a control to allow a person who has been determined to have the right to use the facility in the face authentication process to use the facility on condition that the person has been determined to use the facility in the determination process; A face recognition method comprising:

10. A facial recognition program for verifying the authority to use equipment, A step of executing a face recognition process for detecting a face from a captured image around the facility and determining whether or not the face is the face of a person who has authority to use the facility; acquiring information about a direction of a torso of the person included in the captured image, identifying the direction of the torso of the person, and executing a determination process for determining whether or not the person will use the facility based on the direction of the torso; executing a control to allow a person who has been determined to have the right to use the facility in the face authentication process to use the facility on condition that the person has been determined to use the facility in the determination process; A face recognition program that causes a computer to execute the above.

Images