Authentication device, authentication method, and program
The authentication device improves face recognition accuracy and security by tracking individuals and employing additional verification methods in areas where face recognition is difficult, effectively preventing unauthorized access.
Patent Information
- Application Number
- JP2023082301
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-18
- Publication Date
- 2024-11-28
AI Technical Summary
In walk-through facial recognition systems, cameras installed above gates capture faces at an angle, making it difficult to match faces accurately, especially near the gate where entry verification is crucial, leading to a high risk of unintended entry due to verification errors.
An authentication device that tracks individuals across video frames, performs face verification in a person identification area where faces are visible, and uses trajectory prediction, behavior recognition, and same person determination in an entry determination area to ensure accurate entry decisions.
Enhances the reliability of face authentication by detecting fraudulent attempts and unintended entries, even in areas where face recognition is challenging, by using multiple verification methods.
Smart Images

Figure 2024165803000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a technique for authenticating a face or the like. [Background technology]
[0002] In recent years, face recognition, which authenticates a person by matching facial images and determining whether they belong to the same person, has been used to manage entry to rooms or areas with restricted access. In order to improve user convenience, a method called walk-through face recognition has been proposed, which manages entry by performing face recognition from images taken by a camera that captures a wide area in front of the gate, without the entrant actively turning their face toward the camera in front of the gate.
[0003] Since walk-through face recognition captures a wide area in front of the gate, areas that are not suitable for recognition may also be captured in the image. Patent Document 1 proposes a method of excluding small faces far from the gate and performing recognition on faces captured at a specified face size. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] Patent No. 6151582 [Non-patent literature]
[0005] [Non-Patent Document 1] Bewley, Alex, et al. "Simple online and realtime tracking." 2016 IEEE international conference on image processing (ICIP). IEEE, 2016. [Non-Patent Document 2] DENG, Jiankang, et al. Arcface: Additive angular margin loss for deep face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019. p. 4690-4699. [Non-Patent Document 3] Cao, Zhe, et al. "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields CVPR." (2017). Summary of the Invention [Problem to be solved by the invention]
[0006] In walk-through face recognition, the camera is sometimes installed near the gate, at the top, to ensure a compact size and capture a wide range of images. In this case, people who are photographed near the gate, close to the camera, are large in size, but because the camera is looking down on them, their faces are photographed at an angle. This makes face matching difficult.
[0007] On the other hand, the area near the gate is close to the restricted entry area beyond the gate, and is also an area where it is most important to be vigilant against people trying to evade authentication and enter. If face recognition is used in such a place to determine whether someone is entering, there is a high risk of unintended entry due to incorrect matching.
[0008] The present invention aims to detect the occurrence of unintended intrusions in areas of an image where face recognition is difficult. [Means for solving the problem]
[0009] An authentication device of one embodiment of the present invention comprises an imaging means for capturing video, a tracking means for detecting a person from the video and tracking the person as a tracked person, a comparison means for comparing the tracked person with registered people, and an entry judgment means for determining whether to allow the tracked person to enter a specified area based on the behavior of the tracked person and the comparison result, wherein the comparison means compares the face of the tracked person in a first area in the video, and the entry judgment means makes a judgment based on the comparison result and to allow the tracked person to enter a second area in the video. Effect of the Invention
[0010] According to the present invention, it is possible to detect the occurrence of unintended intrusions in areas of an image where face authentication is difficult. [Brief description of the drawings]
[0011] [Figure 1] FIG. 2 is a component diagram showing an example of a hardware configuration according to the first embodiment of the present invention. [Diagram 2] 1 is a functional configuration diagram showing an example of a functional configuration of an authentication device 100 in a first embodiment of the present invention. [Diagram 3] 2 is a schematic diagram for explaining an operation example of the authentication device 100, a person identification area, an entry determination area, and a gate line in the first embodiment of the present invention. FIG. [Figure 4] 2 is a flowchart showing a process executed by the authentication device 100 for each frame image of a video captured by the imaging unit 201 in the first embodiment of the present invention. FIG. [Diagram 5] 2 is a schematic diagram illustrating an example of tracked person information created by a tracking unit 202 in the first embodiment of the present invention. FIG. [Figure 6] FIG. 11 is a schematic diagram for explaining a method in which the area setting unit 208 sets a person identification area and an entry determination area in the second embodiment of the present invention. [Figure 7] FIG. 11 is a schematic diagram for explaining an example of the operation of the authentication device 100 in the third embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] Hereinafter, the embodiments of the present invention will be described with reference to the drawings. Note that the following embodiments do not limit the invention according to the claims, and all of the combinations of features described in the embodiments are not necessarily essential to the solution of the invention.
[0013] [First embodiment] 1 shows an example of a hardware configuration of an authentication device 100 according to a first embodiment of the present invention. The authentication device 100 has, as its hardware configuration, a control device 11, a storage device 12, a calculation device 13, an input device 14, an output device 15, and an I / F (Interface) device 16. A camera 101 is connected to the I / F device 16. The authentication device 100 is an example of an information processing device.
[0014] The control device 11 is configured with an MPU (Micro Processing Unit) and the like, and controls the entire authentication device 100. The storage device 12 is configured with a recording medium such as a hard disk and an MPU and the like, and holds programs and data necessary for the operation of the control device 11. The arithmetic device 13 is configured with an MPU and the like, and executes necessary arithmetic processing based on the control from the control device 11.
[0015] The input device 14 is a human interface device or the like, and inputs user operations to the authentication device 100. The output device 15 is a display or the like, and presents the processing results of the authentication device 100 to the user. The I / F device 16 is a wired interface such as a universal serial bus, a local area network, or an optical cable, or a wireless interface such as Wi-Fi (registered trademark) or Bluetooth (registered trademark).
[0016] A camera 101 is connected to the I / F device 16. An image captured by the camera 101 is input to the authentication device 100 by the I / F device 16. The I / F device 16 also has a function of transmitting a processing result obtained by the authentication device 100 to the outside, inputting a program, data, etc. required for the operation of the authentication device 100 to the authentication device 100, etc.
[0017] In addition, the I / F device 16 may be connected to an entrance gate or an electronic lock of a gate, and a signal may be sent to open / close, lock, unlock, etc. the gate based on the processing results obtained by the authentication device 100.
[0018] 2 shows an example of a functional configuration of the authentication device 100 according to this embodiment. The authentication device 100 has, as its functional configuration, an imaging unit 201, a tracking unit 202, a matching unit 203, an entry determination unit 204, a trajectory calculation unit 205, a behavior recognition unit 206, and a same person determination unit 207. The authentication device 100 further has, as its functional configuration, an area setting unit 208, a display unit 209, an operation unit 210, and a registration database 220.
[0019] The registration database 220 is a database constructed in the storage device 12, and stores a plurality of records for each registered person, each of which is associated with a feature quantity indicating the facial features of the person. A unique person ID is assigned to each record.
[0020] The image capturing unit 201 acquires an image including a human face captured by the camera 101 connected to the I / F device 16. The image capturing unit 201 may also acquire image data from the storage device 12 or a communication line connected to the I / F device 16.
[0021] The tracking unit 202 detects and tracks a person from the video input from the shooting unit 201. That is, the tracking unit 202 detects a person from a frame image of the video, and also determines the identity of the people detected in the previous and next frame images and associates them. As a result, even if multiple people are included in the video, the tracking unit 202 can track the same person across frame images of different times. Hereinafter, such associated information is called tracked person information, and a series of person images in a video that are considered to be the same person and are identified by one tracked person information is called a tracked person. In this embodiment, a known method is used to associate and track the same person using the position of a person in different frame images and the features of the image of the person at that position, as in the method shown in Non-Patent Document 1. Note that this is just one example, and other tracking methods may be used. In addition, if there are multiple cameras 101, tracking may be performed across multiple images input from them, or a distance image sensor or a ceiling-down camera 101 that identifies the position of a person may be provided and the information may be used to track the person.
[0022] An example of tracking person information is shown in Fig. 5. Fig. 5(A) is a schematic diagram of an image input from the image capture unit 201. Fig. 5(B) shows an example of tracking person information of a person (tracked person) who has moved from position 401 to position 402 on the image in Fig. 5(A).
[0023] The tracking person information has a frame number. The frame number is the serial number of the frame image. In this example, the frame rate of the camera 101 is assumed to be 10 fps. The tracking person in FIG. 5(A) appears at position 401 in frame number 100. The tracking person in FIG. 5(A) moves over 30 seconds (300 frames), is at position 402 in frame number 399, and moves off the screen in the next frame. At this time, the tracking person information holds information from frame numbers 100 to 399 in order. Note that in FIG. 5(B), frame numbers other than the first and last frame numbers and intermediate numbers 200 and 300 are omitted from the illustration.
[0024] In the tracked person information, the positions of the person's face and body in the frame image corresponding to the time detected by the tracking unit 202 are recorded as rectangles for each frame number. The rectangle is recorded as a set of four values: the X and Y coordinates of the upper left corner of the image, the width, and the height. Note that the face rectangle may not exist if the face cannot be detected.
[0025] The tracked person information also records the result of matching in each frame performed by the matching unit 203 in the process described below. Here, the tracked person information records the person ID of the matching result (there may be no match) and the vector value of the feature amount used for matching (omitted from the figure). Note that, as described below, there are cases where matching by the matching unit 203 is not performed depending on the position of the person, and in this example, the matching result and face feature amount are not recorded in frame number 399.
[0026] The tracking person information is managed in a tracking person information list held in the tracking unit 202. The tracking person information list includes tracking person information for each tracking person currently appearing in the video. Tracking person information is added to the tracking person information list every time a person appears. Tracking person information is deleted from the tracking person information list when a person disappears. A unique tracking ID is assigned to each piece of tracking person information.
[0027] FIG. 5(C) is an example of a tracking person information list. The example of FIG. 5(C) shows that there is only one tracking person of FIG. 5(B) in the video. A tracking ID of 1 is assigned to the tracking person information of FIG. 5(B). Every time a person appears and tracking person information is created, a new tracking ID is issued and added to the tracking person information list. In addition, in the process of step S408 described later with reference to FIG. 4, a person identification result in which the collation unit 203 identifies a person for the entire tracking person information is also recorded. In this person identification result, a person ID (which may be "unidentifiable" or "none") is recorded.
[0028] In this embodiment, the tracking process is performed using all frames of the video captured by the image capture unit 201, but frames may be thinned out depending on the processing load, for example, so that one frame out of every two frames is used. In this case, the frame numbers not used in the tracking process are skipped and recorded in the tracked person information.
[0029] The matching unit 203 matches the face image of the tracked person detected by the tracking unit 202 with person data registered in the registration database 220. That is, the matching unit 203 judges which person data the face image corresponds to, or whether it corresponds to none.
[0030] The entry determination unit 204 determines whether or not the tracked person detected by the tracking unit 202 is allowed to enter an area where entry is restricted. The area where entry is restricted is an example of a predetermined area. When processing this determination, the entry determination unit 204 uses the results of the trajectory calculation unit 205, the behavior recognition unit 206, and the same person determination unit 207, as described later.
[0031] The trajectory calculation unit 205 calculates the past trajectory of the person being tracked and predicts the trajectory in the near future, based on information about the person being tracked detected by the tracking unit 202 .
[0032] The behavior recognition unit 206 recognizes the behavior of a person from the video of the person detected by the tracking unit 202.
[0033] The same person determination unit 207 determines whether the face of the tracked person detected by the tracking unit 202 is the same in each frame image, and determines whether the tracked person is consistently the same person.
[0034] The area setting unit 208 sets a person identification area and an entry determination area, which will be described later, for the video captured by the imaging unit 201. The person identification area is an example of a first area. The entry determination area is an example of a second area.
[0035] The display unit 209 controls information required for the user to use the authentication device 100 and visual information of the user interface (UI) to display on the output device 15. When an entry that the entry determination unit 204 determines to be impermissible is detected, an alarm is displayed on the display unit 209. Note that the alarm may be notified to the user by sound, lighting of a patrol lamp, or the like in addition to displaying the alarm on the display unit 209.
[0036] The operation unit 210 receives input from the user inputted to the input device 14 and transmits control information to each part of the authentication device 100 .
[0037] Next, an example of the operation of the authentication device 100 according to this embodiment will be described with reference to Fig. 3. Fig. 3(A) is a side view of the installation location of the authentication device 100. Gate 301 is a gate provided at the entrance to an area to which entry is restricted. In Fig. 3(A), the left side of gate 301 is the area to which entry is restricted. A pedestrian approaches gate 301 from the right side of Fig. 3(A) and passes over gate 301 to enter the restricted area.
[0038] A camera 101 is installed at installation position 302 above gate 301, and this camera 101 captures images of pedestrians approaching gate 301. Because camera 101 is installed at a high position, such as above gate 301, the image captured by camera 101 is from a perspective looking down on the ground.
[0039] As shown in Fig. 3(A), for people in the range indicated by dashed line 303, the angle between the straight line connecting the face and the camera and the optical axis of camera 101 (hereinafter also referred to as "angle between the face and the camera") is relatively small, and the face can be captured close to a frontal view. On the other hand, for people in the range indicated by dashed line 304, the angle between the face and the camera is large, so the face is captured looking down, resulting in an image in which the nose and mouth are difficult to see. In addition, when a person walks towards gate 301, their apparent speed also increases, making it easy for blurring to occur.
[0040] Fig. 3(B) is a schematic diagram of an image captured by camera 101. The area indicated by dashed line 305 is the area on the image corresponding to the range indicated by dashed line 303 in Fig. 3(A). The area indicated by dashed line 306 is the area on the image corresponding to the range indicated by dashed line 304 in Fig. 3(A). Also, dashed line 307 is a gate line indicating the boundary of the entry restricted area in which gate 301 is provided.
[0041] A person approaching gate 301 appears from the top of the image captured by camera 101, and in the area of dashed line 305, their face is shown from the front, and the downward angle becomes larger as they approach gate 301, i.e., the bottom of the image. Then, in the area of dashed line 306, the downward angle of the image of the person's face makes it difficult to see the nose and mouth, and the image becomes significantly blurred, and the person then passes off at the bottom of the screen.
[0042] When performing facial recognition for entry control on such video, matching can be performed accurately in the area surrounded by dashed line 305 because the facial image is nearly forward-facing, but accuracy is likely to be lower in the area surrounded by dashed line 306 due to the angle and blurring.
[0043] Therefore, in this embodiment, the area surrounded by dashed line 305, which is suitable for face matching, is defined as a person identification area, and the area surrounded by dashed line 306, which is close to gate 301 but where matching accuracy is not desired, is defined as an entry judgment area. In the following, the area surrounded by dashed line 305 is referred to as person identification area 305, and the area surrounded by dashed line 306 is referred to as entry judgment area 306.
[0044] For each person approaching gate 301 in the image captured by camera 101, authentication device 100 first performs face matching while passing through person identification area 305. Specifically, authentication device 100 identifies which person among the multiple person data included in registration database 220 corresponds to the person approaching gate 301.
[0045] Next, while the person passes through the entry judgment area 306, the authentication device 100 judges whether the person can enter the entry restricted area beyond the gate 301, and issues an alarm if the person cannot enter. At this time, the result of person identification in the person identification area 305 is used, and furthermore, whether the person on the image is exhibiting suspicious behavior is simultaneously detected and used to judge whether the person can enter.
[0046] In this embodiment, authentication device 100 judges whether entry is permitted in entry judgment area 306 using roughly three methods.
[0047] First, the authentication device 100 performs trajectory prediction by tracking the human body to infer whether the person is about to pass through the gate 301. The authentication device 100 issues an alarm when a person is about to pass through the gate 301 and the result of identifying the person in the person identification area 305 is not a person who is authorized to pass through. The authentication device 100 also issues an alarm here when an unnatural trajectory is taken, such as a person suddenly appearing without passing through the person identification area 305.
[0048] Secondly, the authentication device 100 recognizes the behavior of a person, and issues an alarm if it detects suspicious behavior that may be mischief or vandalism.
[0049] Thirdly, the authentication device 100 performs a same person determination for the tracked person. This is to compare the face image of the tracked person in the video with face images in frame images at other times to determine whether it is consistently the face of the same person. That is, the authentication device 100 determines whether the face of the person changes to a different person's face while passing through the entry determination area 306, or the face becomes unrecognizable. In this embodiment, a method is used in which the matching unit 203 extracts and compares feature amounts used for face matching to determine whether the face images are the same person. If it is determined that they are not the same person, that is, if the face of the tracked person is not consistent, one possibility is that the person in the video is attempting to impersonate someone by hiding their face or holding up a photo. Or, the tracking may take over another person in the vicinity, and different people may be mixed in one tracked person, which may occur due to cutting in or tailgating. Because of this possibility, the authentication device 100 issues an alarm when it is determined that the face is not consistently the same person.
[0050] The face matching performed in the person identification area 305 is a one-to-many matching method that determines which person it is by comparing with many faces contained in the registration database 220. In contrast, the same person determination is performed by one-to-one comparison of two face images of tracked people to determine whether they are the same person.
[0051] Compared to face matching, which requires detailed facial features, trajectory prediction and behavior recognition are relatively less affected by the angle and image quality of the image because they use the overall shape of the human body. Also, since the same person determination is performed between limited frame images taken at similar times by the same camera, the change in the angle and image quality of the image is relatively small. Therefore, it is considered that the change in the feature amount of the same person is small, and it is expected that the accuracy of the same person determination will be maintained even in the entry determination area 306, which is not suitable for face matching.
[0052] As described above, by using a different method in areas where face matching is difficult, the reliability of the face recognition gate as a whole can be improved.
[0053] As described above, according to this embodiment, in addition to determining whether an entering person is authorized to pass, it is possible to detect any suspicious illegal behavior in the entry determination area 306 in front of the gate 301 .
[0054] In this embodiment, an example has been described in which the camera 101 is mounted at a high position and looks down, but the present invention can be applied to other cases in which the difficulty of matching faces changes depending on the position in the image. In addition to looking down, other conditions that may lead to a decrease in matching accuracy and are unsuitable for matching include, for example, a person facing sideways depending on the direction of travel in the passage, lighting conditions, background, and the presence of obstructions. If such areas exist, the present invention can be similarly applied by setting an entry judgment area in a part close to the gate line and setting a person identification area in a part where matching is easier.
[0055] The person identification area, the entry judgment area, and the gate line are determined in advance by the area setting unit 208 when the authentication device 100 is installed. The area setting unit 208 allows the user to set the gate line on the image using the display unit 209 and the operation unit 210, and stores the position. The area setting unit 208 also acquires the depression angle and height from the ground, which are among the installation conditions of the camera 101, from the camera 101 or by input from the user, and calculates the angle with the subject at a point of an average person's head height, for example, 160 cm, for each image position. Then, the area where the depression angle from the camera 101 is less than, for example, 20 degrees is set as the person identification area, and the rest is set as the entry judgment area. For example, the threshold value of the depression angle for switching between the person identification area and the entry judgment area is set so that the accuracy in the person identification area is sufficiently ensured according to the characteristics of the matching performance of the matching unit 203 for the face angle.
[0056] In order to realize the above-mentioned operations, the procedure of the process performed by the authentication device 100 according to this embodiment will be described below with reference to the flowchart in Fig. 4. The process of this flowchart is repeatedly executed on the video captured by the image capturing unit 201, for example, once every 0.2 seconds.
[0057] First, in step S401, the photographing unit 201 acquires the latest frame image from the camera 101. The photographing unit 201 may accumulate frame images in a queue and acquire the first one.
[0058] Next, in step S402, the tracking unit 202 performs human body detection on the frame image acquired in step S401, and detects the human body rectangle and face rectangle of the person included in the frame image. For example, the center positions of the human body rectangle and the face rectangle are matched by Hungarian matching, and the human body rectangle that does not match is left as it is without a face. Also, for the face rectangle that does not match, the corresponding human body rectangle is estimated and added as, for example, a rectangle vertically extended four times the face rectangle.
[0059] Next, in step S403, the tracking unit 202 estimates which person included in the tracking person information list each human body detected in step S402 corresponds to, and updates the tracking person information list. For example, if the human body rectangle detected in step S402 is estimated to be the same person as one of the currently held tracking people using the method shown in Non-Patent Document 1, the corresponding human body rectangle and face rectangle are added to the end of the tracking person information. If not, new tracking person information is generated as the appearance of a new person and added to the tracking person information list. At this time, the initial value of the person identification result is set to "none". Then, tracking person information for which no corresponding human body exists is deleted from the tracking person information list.
[0060] Next, in step S404, the entry determination unit 204 determines whether or not there is any tracking person information included in the tracking person information list that has not yet been processed in step S405 or later in this flow. If the entry determination unit 204 determines that there is any unprocessed information remaining, the process of step S405 is executed. If the entry determination unit 204 determines that there is no unprocessed information remaining, the process is terminated.
[0061] In step S405, the entry determination unit 204 selects one piece of tracked person information that has not yet been processed in step S405 or later in this flow from the tracked person information included in the tracked person information list, and passes the selection to the subsequent processing.
[0062] Next, in step S406, the entry determination unit 204 determines whether the tracking person selected in step S405 is in a location included in the person identification area in the current frame. If the entry determination unit 204 determines that the person is in the person identification area, the process of step S407 is executed. If the entry determination unit 204 determines that the person is not in the person identification area, the process of step S409 is executed.
[0063] In step S407, the matching unit 203 matches the face image in the current frame of the tracking person information selected in step S405 with the face data included in the registration database 220. The matching unit 203 judges which person the face image corresponds to or does not correspond to anyone by matching. For the matching, for example, using the method shown in Non-Patent Document 2, features are extracted from the face image, and compared with the features of each person included in the registration database 220 to calculate a similarity score. The matching unit 203 finds the person with the highest similarity score, and if the similarity score is greater than a threshold T1, it specifies that the face image corresponds to that person. If the similarity score is equal to or less than the threshold T1, the matching unit 203 determines that there is no match. The similarity score also determines that there is no match if there is no face rectangle. The matching unit 203 records the person ID and features of the matching result at the position of the current frame number of the tracking person information.
[0064] Next, in step S408, the matching unit 203 uses the matching results from the start of tracking to the current frame for the tracking person information selected in step S405 to identify which person is the person in the tracking person information. If a specific person ID exceeds, for example, 90% in the past matching results of the tracking person information, including those recorded in this flow, the person is identified as the person, and if not, the person is deemed "unidentifiable." Then, the person identification result is recorded in the tracking person information list. Here, the processing of the tracking person information in step S405 ends, and the process returns to step S404.
[0065] In step S409, the entry judgment unit 204 judges whether the person to be tracked selected in step S405 is in a location included in the entry judgment area in the current frame. This is judged by whether the center point of the face rectangle is included in the entry judgment area. If the entry judgment unit 204 judges that the person is in the entry judgment area, the process of step S410 is executed. If the entry judgment unit 204 judges that the person is not in the entry judgment area, the process of step S404 is executed.
[0066] In step S410, the same person determination unit 207 determines whether the face of the current person selected in step S405 is the same person as the face of the person in the immediately preceding frame image. The same person determination unit 207 extracts features of the face image from the current frame image using the matching unit 203, and compares it with the face features of, for example, 10 frame numbers at the end of the tracked person information. If the average of the similarity scores is greater than the threshold T2, the same person determination unit 207 determines that it is the "same person". If the average of the similarity scores is equal to or less than the threshold T2, the same person determination unit 207 determines that it is "not the same person". Note that the same person determination unit 207 excludes frame numbers that do not have a face rectangle and takes the average, but if there is no face rectangle in, for example, four or more frames, it determines that it is "not the same person". Here, the threshold T2 is set lower than the threshold T1. This is because the same person determination is performed by comparing the face with a single person, and the accuracy is less likely to decrease even if the threshold is lowered compared to matching that compares with multiple people. This shows that it is possible to maintain accuracy even in the approach judgment area, which has worse conditions such as face angle than the person identification area.
[0067] The same person determination unit 207 may compare the feature amount extracted from the current frame image with the feature amount of the person ID recorded in the person identification result of the registration database 220. In this case, if there is no face rectangle or if the person identification result is "unidentifiable", it is determined here for convenience that it is "the same person". Also, for a person whose person identification result is "none", that is, a person for whom step S408 has never been executed and no matching has been performed, it is determined for convenience that it is "not the same person". This may be because the person ran through the person identification area at high speed or passed through while hiding their face. Or, there may be a case where the tracked person suddenly appeared in the entry judgment area, which may be because they cut in from a blind spot of the screen or hid in the shadow of the person in front of them and aimed to tailgate. In either case, it is suspicious, so it is determined that it is "not the same person" to issue an alarm.
[0068] In addition, although the determination of whether or not the images are the same person is performed using the feature amount extracted by the matching unit 203 here, the determination of whether or not the images are the same person may be performed using a method different from that used by the matching unit 203. For example, a method may be used in which feature amounts extracted using a model different from that used by the matching unit 203 are compared, or face patch images are compared. A method specialized in conditions such as a look-down angle may also be used for determining whether or not the images are the same person.
[0069] Next, in step S411, the entry determination unit 204 determines whether or not it was determined in step S410 that the person is not the same person.
[0070] If the entry determination unit 204 determines in step S411 that the person is not the same person as determined in step S410, the process of step S417 is executed. In step S417, the display unit 209 displays a warning to the user. If the process proceeds from step S411 to step S417, it means that the face of the person being tracked has been replaced with another person. In this case, the display unit 209 issues a warning because it is suspected that the person being tracked has been replaced by cutting in from the side or tailgating, or that the face has been altered in an attempt to impersonate someone else.
[0071] If the entry determination unit 204 determines in step S411 that the person is not the same person as the person in step S410, the process of step S412 is executed. If the person identification result is "unidentifiable", the process of step S412 is also executed without immediately issuing a warning.
[0072] In step S412, the behavior recognition unit 206 performs behavior recognition on the tracking person selected in step S405, and determines whether the person has engaged in suspicious behavior. The behavior recognition unit 206 detects the joints of the human body using, for example, the method shown in Non-Patent Document 3, and determines whether the person is engaging in suspicious behavior, in this case, bringing their hands closer to the camera.
[0073] Other suspicious behaviors that may be detected include violent behavior or running extremely fast. Object detection may also be used in combination to detect the holding of a dangerous object such as a crowbar or a knife. Masks or paper with a face printed on it, which may be used for impersonation, may be detected, and movements such as bringing such objects close to the face or putting them on or taking them off may be detected. Behaviors such as falling to the ground, crouching, or getting lost and wandering around may be detected as suspicious behaviors in a broad sense, and a warning may be issued to urge staff to head to the scene.
[0074] Next, in step S413, the entry determination unit 204 determines whether or not it has been determined in step S412 that suspicious behavior has been performed. If the entry determination unit 204 determines that suspicious behavior has been performed in step S412, the process of step S417 is executed. In step S417, the display unit 209 displays an alarm to the user. Here, regardless of the person identification result, an alarm is issued if suspicious behavior has been performed. Note that the method of determining suspicious behavior may be changed depending on whether the person ID in the person identification result is allowed to enter or not, or whether it is "unidentifiable" or "none". For example, if the person is not allowed to enter, the threshold for determining suspicious behavior may be lowered.
[0075] If the entry determination unit 204 determines in step S413 that the person has not engaged in suspicious behavior in step S412, the process proceeds to step S414. In step S414, the trajectory calculation unit 205 predicts the trajectory of the person to be tracked selected in step S405, and determines whether the person is attempting to enter through the gate.
[0076] Specifically, for the most recent five frames of the tracking person information selected in step S405, the trajectory calculation unit 205 calculates movement vectors starting from the center of the human body frame in the immediately preceding frame and ending at the center of the current human body frame, and obtains the average vector. The trajectory calculation unit 205 predicts this average vector to be the movement vector of the person in the immediate future, and predicts the position of the tracking person 10 frames later to be a point moved 10 times the calculated average vector from the center of the current human body frame. If the predicted position of the tracking person 10 frames later crosses the gate line, the trajectory calculation unit 205 determines that the person is about to cross the gate in the near future.
[0077] Note that the number of frames used to calculate the average vector and the number of frames used to predict the destination are merely examples, and it is desirable to set appropriate values depending on the frame rate (fps) of the video and how often this flow is performed.
[0078] Next, in step S415, the entry determination unit 204 determines whether or not it was determined in step S414 that the person is about to cross the gate line. If the entry determination unit 204 determines that the person is about to cross the gate, the process of step S416 is executed. If the entry determination unit 204 determines that the person is not about to cross the gate, the process of step S404 is executed.
[0079] In step S416, the entry determination unit 204 determines whether or not the person with the person ID in the person identification result is allowed to enter the area where entry is restricted. Specifically, if the person ID in the person identification result is included in a set of person IDs that are the criteria for determining entry, the entry determination unit 204 determines that the person is allowed to enter. If the entry determination unit 204 determines that the person is allowed to enter, the process of step S404 is executed. If the entry determination unit 204 determines that the person is not allowed to enter, the process of step S417 is executed. In step S417, the display unit 209 displays an alarm to the user. Here, the alarm is issued because a person who is not allowed to enter is attempting to pass through the gate. Note that an alarm is also issued when the person identification result is "unidentifiable" or "none."
[0080] In step S415, the entry judgment unit 204 executes the judgment as to whether or not a person is allowed to enter only for a person who is about to cross the gate line. This prevents an alarm from being issued for every person who is not allowed to enter but who does not cross the gate line, for example, who simply enters the entry judgment area and then turns back.
[0081] In step S417, the display unit 209 displays an alarm to the user. Here, the reason for issuing the alarm and accompanying information may be displayed together so that the user can easily understand the cause of the alarm. For example, if it is determined in step S411 that the person is not the same person, the person IDs before and after the alarm are displayed, and if it is determined in step S413 that a suspicious behavior has occurred, the type of the behavior, etc., can be displayed together. In addition, it is also possible to determine a response to an event that is unlikely to become an immediate problem, such as a case where the person passes through a person identification area quickly and the person identification result is "none," or a behavior of getting lost and wandering around is detected. In such a case, the display method may be changed to distinguish it from an alarm as a caution or reference information.
[0082] As described above, according to this embodiment, it is possible to perform face authentication of a person with high accuracy in a person identification area, and then detect the occurrence of an unintended intrusion even in an entry determination area where face authentication is difficult.
[0083] [Second embodiment] In the second embodiment of the present invention, a method for determining the person identification area and the entry judgment area, which is different from that in the first embodiment, will be described. In this embodiment, the person identification area and the entry judgment area are automatically set based on the actual accuracy of the matching unit 203 in the video captured after installation. Hereinafter, a processing method performed by the area setting unit 208 when the authentication device 100 is installed in this embodiment will be described.
[0084] First, an image of a person passing through the image of the installation location is captured by the capturing unit 201. Then, for the entire captured image, face detection is performed and the face of the person appearing is matched by using the tracking unit 202 and the matching unit 203. Then, the center position of the face frame of the person appearing and whether the matching result is correct or incorrect are recorded.
[0085] Regarding the correctness or incorrectness of the matching result, first, all the similarity scores below the threshold T1 are regarded as incorrect. Then, for the rest (those with a similarity score greater than the threshold T1), if the correct person matches the person ID of the matching result of the matching unit 203, it is regarded as correct, otherwise it is regarded as incorrect. This can be determined by various methods such as a method in which a person checks and inputs, a method in which the photographing location is controlled and the person, position and time of passing are recorded in advance and compared. As a simpler method, a matching method different from the matching unit 203 can be performed and the result can be regarded as the correct answer (a method with higher accuracy and slower operation can be used, such as a statistical model larger than the matching unit 203). Alternatively, if it is difficult to make an error of giving a high score to a different person, all the similarity scores exceeding the threshold T1 approximately can be regarded as correct.
[0086] Fig. 6(A) is a schematic diagram showing a state in which the correct or incorrect matching results are recorded for an appropriate length of video. In Fig. 6(A), white circles indicate the appearance positions of correct faces, and black circles indicate the appearance positions of incorrect faces. When a sufficient number of such results are collected, the entire captured image is divided into a grid, and for matching results within the grid that are greater than, for example, 80%, the range of the grid is set as a person identification area. In addition, for matching results within the grid that are less than, for example, 80%, the range between the grid including the gate line and the grid between the gate line and the person identification area is set as an entry judgment area.
[0087] Fig. 6(B) is an example of the person identification area and the entry judgment area in the example of Fig. 6(A). The hatched area with diagonal lines is the person identification area, and the hatched area with vertical lines is the entry judgment area. The person identification area and the entry judgment area set by the area setting unit 208 are displayed on the display unit 209 so that the user can confirm them. The area may be modified by the user using the operation unit 210.
[0088] In this way, the person identification area and the entry judgment area can be appropriately set based on the performance of the matching unit 203 in the actual video. Note that Fig. 6 is a simplified schematic diagram, and the areas can be set more precisely by increasing the number of matching results and making the grid finer.
[0089] [Third embodiment] In the third embodiment of the present invention, a method is described in which a plurality of gate lines are provided and an entry judgment is performed for a plurality of gates using a single camera image. Note that the description of the points common to the first and second embodiments is omitted, and only the differences are described.
[0090] FIG. 7(A) is a schematic diagram of an installation example of the authentication device 100 in this embodiment, seen from above. In the example of FIG. 7(A), there are three gates A, B, and C, and different people are permitted to enter each gate. The camera 101 is installed near the ceiling at a position 704. FIG. 7(B) is a schematic diagram of an image captured by the camera 101. In this example, three gate lines, 711, 712, and 713, are provided corresponding to the gates A, B, and C, respectively. These gate lines 711, 712, and 713 correspond to the lines 701, 702, and 703 in FIG. 7(A). In addition, the authentication device 100 holds in advance a set of person IDs of people who are allowed to enter, which are the criteria for determining entry, corresponding to each of the multiple gate lines. In FIG. 7(B), the area 714 surrounded by a dashed line is a person identification area, and the area 715 surrounded by a dashed line is an entry determination area.
[0091] In the third embodiment, the process in Fig. 4 is also executed. In step S414, the trajectory calculation unit 205 predicts the trajectory of the person to be tracked selected in step S405, and determines whether the person is about to cross any of the gate lines.
[0092] Then, in step S416, the entry determination unit 204 uses the set of person IDs of people who are allowed to enter for each gate line to determine whether the person ID of the person identified in step S408 is allowed to enter. In this way, for example, an alarm can be issued even when a person who is allowed to enter gate A but not gates B or C attempts to enter gate B.
[0093] According to this embodiment, one camera can simultaneously handle multiple gates, not limited to three. As an application, it is also possible to handle gates with complex shapes by providing multiple gate lines with a common set of person IDs that can enter.
[0094] <Other embodiments> Although the embodiment has been described above in detail, the present invention can be embodied as, for example, a system, an apparatus, a method, a program, or a recording medium (storage medium), etc. Specifically, the present invention may be applied to a system composed of multiple devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or may be applied to an apparatus composed of a single device.
[0095] The object of the present invention can also be achieved by the following: That is, a recording medium (or storage medium) on which is recorded a program code (computer program) of software that realizes the functions of the above-mentioned embodiments is supplied to a system or device. The storage medium is a computer-readable storage medium. Then, a computer (or a CPU or MPU) of the system or device reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-mentioned embodiments, and the recording medium on which the program code is recorded constitutes the present invention.
[0096] The present invention can also be realized by supplying a program for implementing one or more functions of the above-described embodiments to a system or device, and having one or more processors in a computer of the system or device read and execute the program. The program may be supplied to the system or device via a network or a storage medium. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more functions.
[0097] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.
[0098] The disclosure of this embodiment includes the following configurations and methods. (Configuration 1) A means for capturing video; A tracking means for detecting a person from the video and tracking the person as a tracking person; A matching means for matching the tracked person with a registered person; and an entry determination means for determining whether to permit entry of the person into a predetermined area based on the behavior of the person being tracked and the result of the comparison, The matching means matches a face of the person to be tracked in a first area in the video, The entry determination means determines whether to permit entry of the tracking person in the second area based on the comparison result and the image. 13. An authentication device comprising: (Configuration 2) Further comprising an area setting means for setting the first area and the second area, the area setting means sets the first area and the second area based on the accuracy characteristics of the matching means; 2. The authentication device according to configuration 1. (Configuration 3) the area setting means sets the first area and the second area based on an angle at which a subject is photographed for each position in the video under an installation condition of the photographing means. 3. The authentication device according to configuration 2. (Configuration 4) the area setting means sets the first area and the second area based on the accuracy of each position on the image of the matching means in the video. 3. The authentication device according to configuration 2. (Configuration 5) A trajectory calculation means for obtaining a trajectory of the tracked person from the result of the tracking is further provided, the entry determination means determines whether or not entry of the tracked person is permitted based on the trajectory; 5. The authentication device according to any one of configurations 1 to 4. (Configuration 6) Further comprising a behavior recognition means for recognizing the behavior of the tracking person, The entry determination means determines whether or not to permit the entry of the person to be tracked based on the behavior. 6. The authentication device according to any one of configurations 1 to 5. (Configuration 7) The face recognition device further includes an identical person determination unit for performing an identical person determination by comparing the faces of the tracked person at different times to determine whether they are the faces of the same person, The entry determination means determines whether to permit the entry of the tracked person based on a result of the same person determination. 7. The authentication device according to any one of configurations 1 to 6. (Configuration 8) the entry determination means holds a plurality of entry determination criteria, and determines entry based on the criterion selected based on the position of the person to be tracked; 8. The authentication device according to any one of configurations 1 to 7. (Method 1) The filming process of shooting the video; a tracking step of detecting a person from the video and tracking the person as a tracking person; a matching step of matching the tracked person with a registered person; and an entry determination step of determining whether to permit entry of the person into a predetermined area based on the behavior of the person to be tracked and the result of the comparison, In the matching step, a face of the person to be tracked in a first area in the video is matched, In the entry determination step, a determination is made as to whether entry is permitted for the tracking person in the second area based on the result of the matching and the image. 13. An authentication method comprising: (Program 1) A program for causing a computer to function as each of the means according to any one of configurations 1 to 8. [Explanation of symbols]
[0099] 100 Authentication Device 11 Control device 12 Storage device 13 Arithmetic unit 14 Input Devices 15 Output Devices 16 I / F device
Claims
1. A means for capturing video; A tracking means for detecting a person from the video and tracking the person as a tracking person; A matching means for matching the tracked person with a registered person; and an entry determination means for determining whether to permit entry of the person into a predetermined area based on the behavior of the person being tracked and the result of the comparison, The matching means matches a face of the person to be tracked in a first area in the video; The entry determination means determines whether to permit entry of the tracking person in the second area based on the comparison result and the image.
13. An authentication device comprising:
2. further comprising an area setting means for setting the first area and the second area; the area setting means sets the first area and the second area based on the accuracy characteristics of the matching means; 2. The authentication device according to claim 1.
3. the area setting means sets the first area and the second area based on an angle at which a subject is photographed for each position in the video under an installation condition of the photographing means.
3. The authentication device according to claim 2.
4. the area setting means sets the first area and the second area based on the accuracy of each position on the image of the matching means in the video.
3. The authentication device according to claim 2.
5. A trajectory calculation means for obtaining a trajectory of the tracked person from the result of the tracking is further provided, the entry determination means determines whether or not entry of the tracked person is permitted based on the trajectory; 2. The authentication device according to claim 1.
6. Further comprising a behavior recognition means for recognizing the behavior of the tracking person, The entry determination means determines whether or not to permit the entry of the person to be tracked based on the behavior.
2. The authentication device according to claim 1.
7. The face recognition device further includes an identical person determination unit for performing an identical person determination by comparing the faces of the tracked person at different times to determine whether they are the faces of the same person, The entry determination means determines whether to permit the entry of the tracked person based on a result of the same person determination.
2. The authentication device according to claim 1.
8. the entry determination means holds a plurality of entry determination criteria, and determines entry based on the criterion selected based on the position of the person to be tracked; 2. The authentication device according to claim 1.
9. The filming process of shooting the video; a tracking step of detecting a person from the video and tracking the person as a tracking person; a matching step of matching the tracked person with a registered person; and an entry determination step of determining whether to permit entry of the person into a predetermined area based on the behavior of the person to be tracked and the result of the comparison, In the matching step, a face of the person to be tracked in a first area in the video is matched, In the entry determination step, a determination is made as to whether entry is permitted for the tracking person in the second area based on the result of the comparison and the image.
13. An authentication method comprising:
10. A program for causing a computer to function as each of the means according to any one of claims 1 to 8.
Citation Information
Patent Citations
Direction searching device
JP1986051582A
Cited By
Notification system and notification method
JP2026145364A
