Authentication system, authentication method and program

Abstract

To enable an improvement in processing speed.SOLUTION: An authentication system includes: a control device that, when a request signal for requesting information on a person who attempts to pass through a gate is received from an external device, transmits a face authentication result based on the face image of the person having undergone the determination as taking a certain action to pass through the gate to the external device; and a face authentication server which executes the face authentication, and which transmits the face authentication result to the control device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an authentication system, an authentication method, and a program. [Background technology]

[0002] In railway networks, when a passenger passes through a ticket gate to enter a station or when a passenger passes through a ticket gate to exit a station, it is determined whether the passenger is permitted to pass through the ticket gate. For example, a ticket gate process is performed in which it is determined whether the passenger is permitted to pass through the ticket gate based on the ticket held by the passenger. In addition, it is being considered to combine a ticket gate process in which it is determined whether a target passenger is permitted to pass through the ticket gate based on face matching technology that identifies the target person by a facial image of the target person, with a ticket-based ticket gate process.

[0003] For example, Patent Document 1 describes an information processing device that includes an interface that requests the reading of a medium when a passerby is identified by comparing detected face data detected from a photographed image of the passerby with first face data based on a database in which first face data acquired by an issuing machine and first ticket information on a medium issued by the issuing machine are registered in association with each other, and a processor that outputs a passerby passage determination result based on the first ticket information when the first ticket information is read from the medium of the passerby identified based on the first face data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-144170 Summary of the Invention [Problem to be solved by the invention]

[0005] However, there is room for further study on improving the processing speed of ticket gate processing by combining ticket gate processing based on face matching technology and ticket gate processing based on passenger tickets.

[0006] Non-limiting examples of the present disclosure contribute to providing an authentication system, authentication method, and program that can improve the processing speed of ticket inspection processing that combines ticket inspection processing based on face matching technology and ticket-based ticket inspection processing. [Means for solving the problem]

[0007] An authentication system according to one embodiment of the present disclosure includes a control device that, when receiving a request signal from an external device requesting information about a person attempting to pass through a gate, transmits to the external device the results of face matching based on a facial image of the person who is determined to have performed a specific action to pass through the gate, and a face authentication server that performs the face matching and transmits the results of the face matching to the control device.

[0008] In an authentication method according to one embodiment of the present disclosure, a control device receives a request signal from an external device requesting information about a person attempting to pass through a gate, and upon receiving the request signal, transmits to the external device the results of face matching based on a facial image of the person who has been determined to have performed a specific action to pass through the gate.

[0009] A program according to one embodiment of the present disclosure causes a control device to receive a request signal from an external device requesting information about a person attempting to pass through a gate, and upon receiving the request signal, transmit to the external device the results of face matching based on a facial image of the person who has been determined to have performed a specific action to pass through the gate.

[0010] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. [Effects of the Invention]

[0011] A non-limiting example of the present disclosure can improve the processing speed of ticket inspection processing that combines ticket inspection processing based on face matching technology and ticket-based ticket inspection processing.

[0012] Further advantages and benefits of an embodiment of the present disclosure will become apparent from the specification and drawings. Such advantages and / or benefits may be provided by some of the embodiments and features described in the specification and drawings, respectively, but not necessarily all of them may be provided to obtain one or more identical features. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating an example of a ticket gate system according to an embodiment. [Figure 2] FIG. 1 is a block diagram illustrating an example of a configuration according to an embodiment. [Figure 3] A diagram showing an example of skeletal information [Figure 4] FIG. 1 shows a first example of a person tracking list. [Figure 5] A diagram showing a first example of holding a device over a smartphone [Figure 6A] A diagram showing a second example of holding a device [Figure 6B] A diagram showing the four cases in Fig. 6A [Figure 7A] A diagram showing a third example of holding a device [Figure 7B] A diagram showing the four cases in Fig. 7A [Figure 8A] A diagram showing the fourth example of holding a device [Figure 8B] A diagram showing the four cases in Fig. 8A [Figure 9A] A diagram showing the fifth example of holding a device [Figure 9B] A diagram showing the four cases in Fig. 9A [Figure 10A] A diagram showing an example of a case where two people are trying to enter a ticket gate [Figure 10B] FIG. 10B shows an example of a person tracking list corresponding to FIG. 10A. [Figure 11A]A flowchart showing an example of processing of a face authentication control device according to an embodiment. [Figure 11B] A flowchart showing an example of the holding determination process in ST15 of FIG. 11A. [Figure 12] Diagram showing variations of hand-hold detection [Figure 13] A diagram showing an example of case 1 of holding a smartphone [Figure 14] A diagram showing an example of case 2 of holding the device over your eyes [Figure 15A] A diagram showing an example of case 3 of holding the device over your head. [Figure 15B] FIG. 15B is a diagram showing an example of changes in the person tracking list in Case 3 shown in FIG. 15A. [Figure 16A] A sequence diagram showing the processing flow of each device in one embodiment. [Figure 16B] A sequence diagram showing the processing flow of each device in one embodiment. [Figure 17A] FIG. 10 is a sequence diagram illustrating an example of the processing flow of each device in a first variation of an embodiment. [Figure 17B] FIG. 10 is a sequence diagram illustrating an example of the processing flow of each device in a first variation of an embodiment. [Figure 18] A diagram showing an example of a person tracking list in Variation 1 [Figure 19] FIG. 10 is a diagram showing an example of the configuration of variation 2 of an embodiment; [Figure 20A] FIG. 10 is a sequence diagram illustrating an example of the processing flow of each device in variation 2 of the embodiment. [Figure 20B] FIG. 10 is a sequence diagram illustrating an example of the processing flow of each device in variation 2 of the embodiment. [Figure 21] A diagram showing an example of a people tracking list for Variation 3 [Figure 22A] FIG. 10 is a sequence diagram illustrating an example of the process flow of each device in variation 3 of the embodiment. [Figure 22B] FIG. 10 is a sequence diagram illustrating an example of the process flow of each device in variation 3 of the embodiment. [Figure 23A]FIG. 10 is a sequence diagram illustrating an example of the processing flow of each device in variation 4 of the embodiment. [Figure 23B] FIG. 10 is a sequence diagram illustrating an example of the processing flow of each device in variation 4 of the embodiment. [Figure 24] FIG. 10 is a diagram showing an example of a ticket gate system according to Variation 5 of an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functions are designated by the same reference numerals, and redundant description will be omitted.

[0015] (One embodiment) <Knowledge that led to this disclosure> In railway networks, railway users hold an IC card containing information about their ticket over a reader in a ticket gate, and a ticket gate determination is performed to determine whether the user is permitted to pass through the gate based on the information read by the reader. This ticket gate determination is sometimes referred to as an IC card ticket gate determination. Note that, while the following description uses the example of holding an IC card over the reader as an action taken by a railway user to pass through a ticket gate, the present disclosure is not limited to this. For example, the action taken by a railway user to pass through a ticket gate may include holding another medium over the reader. Here, the other medium may be, for example, a smartphone, smartwatch, or other device compatible with transportation IC cards. Furthermore, the action taken by a railway user to pass through a ticket gate may include inserting a magnetic ticket purchased from a ticket vending machine or the like into the slot of the ticket gate.

[0016] When the reading unit reads the information on the IC card, it may become necessary to determine whether the person holding the IC card matches the person associated with the information on the IC card. For example, in a case where a discount service for train tickets is offered to elderly people, a non-elderly person who is not eligible for the discount service may unfairly receive the discount service by using an IC card associated with the information of a certain elderly person. In order to prevent such unfair use of IC cards, it becomes necessary to determine whether the person holding the IC card matches the person associated with the information on the IC card.

[0017] To determine whether the person holding the IC card matches the person associated with the information on the IC card, it is being considered to use a facial matching process based on a photographed image of the face of the person holding the IC card and a pre-registered image of the face of the person associated with the information on the IC card.

[0018] However, when multiple people are trying to pass through a ticket gate in succession, the captured image may contain multiple faces. In such cases, if it is not possible to identify the person who swiped the IC card among the multiple people included in the captured image, it is difficult to accurately determine whether the person who swiped the IC card matches the person associated with the information on the IC card based on a facial matching process. For example, even if the person who swiped the IC card matches the person associated with the information on the IC card, if a facial matching process is performed based on a facial image of a person different from the person who swiped the IC card among the multiple people included in the captured image, it may be erroneously determined that the person who swiped the IC card does not match the person associated with the information on the IC card. When an erroneous determination occurs, the authentication processing speed decreases.

[0019] Therefore, in this embodiment, a process for identifying who has held up an IC card in a captured image is performed, and a determination is made based on a facial recognition process as to whether the person holding up the IC card matches the person associated with the information on the IC card. For example, in this embodiment, a skeleton detection technique is used to determine a skeleton including a hand extended toward a reader of a ticket gate, and a facial image corresponding to the skeleton is determined to be the facial image of the person holding up the IC card.

[0020] <System configuration example> Fig. 1 is a diagram showing an example of a ticket gate system 1 according to the present embodiment. As shown in Fig. 1, the ticket gate system includes a face authentication control device 10, a ticket gate control device 20, a camera 30, a face authentication server 40, a management server 50, and a ticket gate determination server 60. The ticket gate system 1 is used, for example, at the ticket gates of stations on a railway network or the like, to perform ticket gate determination for station users.

[0021] The ticket gate determination server 60 accepts registration of user information. The user operates a terminal such as a smartphone to register the user's personal information and facial image in the ticket gate determination server 60. The ticket gate determination server 60 stores the registered user information. The registered user information is referred to as registration information, and the user who registered the information is referred to as a registered user. Some or all of the registration information is provided to the management server 50 and the face authentication server 40. The user's registration information includes information such as the user's age (e.g., whether or not the user is elderly), whether or not the user has a disability, and whether or not the user has a specific qualification. However, the registration information listed here is an example, and the present disclosure is not limited thereto. The user's registration information does not necessarily include the user's age (e.g., whether or not the user is elderly), whether or not the user has a disability, etc., and information not exemplified here may be added to the user's registration information.

[0022] Furthermore, the user purchases a ticket via the ticket examination determination server 60. Information on the purchased ticket is contained in an IC card held by the user.

[0023] For example, if a user wishes to purchase a ticket that applies a discount service offered to seniors, the ticket gate determination server 60 checks whether the user is a senior citizen based on the registration information. If the user is a senior citizen, the ticket gate determination server 60 permits the purchase of the ticket that applies the discount service offered to seniors. Information about the purchased ticket, i.e., the ticket that applies the discount service offered to seniors, is included in the IC card held by the purchaser. In other words, the IC card that contains information about the ticket that applies the discount service offered to seniors is associated with the senior citizen who purchased it.

[0024] The ticket gate control device 20 controls the operation of the ticket gate. For example, if a pedestrian attempting to pass through the ticket gate is a person authorized to pass through the ticket gate, the ticket gate control device 20 performs passage control to allow the pedestrian to pass through the ticket gate. In addition, if a pedestrian attempting to pass through the ticket gate is not a person authorized to pass through the ticket gate, the ticket gate control device 20 performs blocking control to block the pedestrian from passing through the ticket gate.

[0025] The ticket gate control device 20 reads the information on the IC card held by the passerby and performs ticket gate determination to determine whether the ticket included in the IC card is a ticket permitted to pass through the ticket gate. The ticket gate determination control device 20 transmits a ticket gate determination request including the IC card information to the ticket gate determination server 60.

[0026] Based on the information on the IC card, the ticket gate determination server 60 determines whether the ticket included in the IC card is a ticket that is permitted to pass through the ticket gate, and sends a ticket gate determination response including the determination result to the ticket gate determination control device 20.

[0027] Furthermore, when the ticket gate determination server 60 determines that the ticket included in the IC card is a ticket that is permitted to pass through the ticket gate, it determines whether identity verification is required based on the information on the IC card. Identity verification corresponds to confirmation of whether the passerby who swipes the IC card is the actual user associated with the IC card. For example, if the ticket is purchased on the condition that the passerby has a specific qualification, it determines that identity verification is required to determine whether the passerby who swipes the IC card matches the user associated with the IC card. The result of the determination of whether identity verification is required is included in a ticket gate determination response and is transmitted to the ticket gate control device 20.

[0028] If the ticket gate determination response contains a determination result that identity verification is required, the ticket gate control device 20 transmits an identity verification request to the face authentication control device 10. The identity verification request is an example of a request signal that requests information about a person who is about to pass through the ticket gate. The identity verification request may or may not include information about the IC card that is held over the ticket gate.

[0029] If the ticket gate determination response includes a determination result that the ticket included in the IC card is not a ticket permitted to pass through the ticket gate, the ticket gate control device 20 may perform control to prevent passage through the ticket gate. Also, if the ticket gate determination response includes a determination result that personal identification is not required, the ticket gate control device 20 may perform control to permit passage through the ticket gate.

[0030] Note that, in the above example, the ticket gate control device 20 transmits a ticket gate determination request to the ticket gate determination server 60 and receives a ticket gate determination response from the ticket gate determination server 60, but the present disclosure is not limited to this. For example, the ticket gate control device 20 may receive determination information (e.g., ticket reservation information) in advance from the ticket gate determination server 60, and when a ticket gate determination request is generated, the ticket gate control device 20 may determine inside itself whether the ticket included in the IC card is a ticket that is permitted to pass through the ticket gate.

[0031] When the face authentication control device 10 receives an identity verification request from the ticket gate control device 20, it identifies the person who is the subject of the identity verification request in the captured image and transmits a face matching request to the face authentication server 40, requesting face matching for the identified person. The face matching request includes information indicating the identified person. When the face authentication control device 10 transmits a face matching request, there are two cases: one is to transmit the face matching request together with the face image of the person who is the subject of the identity verification request in the captured image and IC card information, and the other is to transmit the face matching request including only the face image of the person who is the subject of the identity verification request in the captured image, and receive the IC card information of the registered person as the result of face matching in response to the face matching request.

[0032] The face authentication server 40 receives a face matching request and performs face matching. In a case where a face matching request is sent together with a face image of a subject and information about an IC card, face matching is performed to determine whether the identified subject is the user associated with the IC card based on the face image of the subject included in the face matching request and the face image registered by the user (registered user) associated with the information on the held IC card. For example, face matching calculates a matching score indicating the degree of similarity between the face image of the identified subject included in the face matching request and the face image registered by the user (registered user) associated with the information on the held IC card. Then, in face matching, if the matching score is equal to or greater than a threshold, it is determined that the identified subject is the user associated with the IC card, and if the matching score is less than the threshold, it is determined that the identified subject is not the user associated with the IC card.

[0033] The face authentication server 40 transmits a face matching result indicating the result of face matching to the face authentication control device 10. The face matching result is an example of a response to a face matching request, and may be referred to as a face matching response. The face matching result includes a determination result as to whether the identified subject is the user associated with the IC card.

[0034] In the case where a face matching request including only the face image of the subject of the identity verification request in the captured image is transmitted and the IC card information of the registered user is received as a result of face matching in response to the face matching request, the identified subject is identified among the users based on the face image of the subject included in the face matching request and the face images registered by each user. For example, in face matching, a matching score is calculated between the face image of the subject included in the face matching request and the face images registered by each user. Then, in face matching, if the highest matching score calculated for each user is equal to or greater than a threshold, the subject is determined to be the user corresponding to the highest matching score. Note that if the highest matching score is less than the threshold, it is determined that the subject cannot be identified. The face authentication server 40 transmits a face matching result indicating the result of face matching to the face authentication control device 10. The face matching result includes information of the user corresponding to the highest matching score (e.g., IC card information of the user corresponding to the highest matching score).

[0035] The face authentication control device 10 transmits a confirmation result in response to the identity verification request to the ticket gate control device 20. The confirmation result may include a face matching ID that identifies the person who is the subject of the identity verification request. The confirmation result is an example of a response to the identity verification request. The confirmation result may also be referred to as a result of face matching based on a facial image of the person who is the subject of the identity verification request. Here, the person who is the subject of the identity verification request is a person who is determined to have performed a specific action to pass through the ticket gate. For example, in this embodiment, the specific action to pass through the ticket gate is the action of holding an IC card over a reading unit.

[0036] Although FIG. 1 shows one face authentication server 40, the present disclosure is not limited to this.

[0037] <Device configuration> 2 is a block diagram showing an example of the configuration according to this embodiment. The face authentication control device 10 includes a camera control unit 101, a face detection unit 102, a skeleton detection unit 103, a holding determination unit 104, a tracking management unit 105, and a communication unit 106. The face authentication control device 10 may be configured by an information processing device (for example, a personal computer). At least one of the camera control unit 101, the face detection unit 102, the skeleton detection unit 103, the holding determination unit 104, and the tracking management unit 105 may be included in a processing unit or a processor.

[0038] The camera control unit 101 controls the camera 30 and causes the camera 30 to take a photograph. The camera control unit 101 receives photographed images from the camera 30. The camera control unit 101 outputs the received photographed images to the face detection unit 102 and the skeleton detection unit 103. Each photographed image is assigned an identification number, which is a consecutive number that is automatically incremented when photographed and identifies each photographed image. In addition, each photographed image is assigned time information that indicates the time when the photograph was taken. Note that the time information may be information with an accuracy of the order of milliseconds.

[0039] The face detection unit 102 executes a face frame detection process. In the face frame detection process, the face detection unit 102 detects a human face from a captured image and determines coordinates corresponding to the position of the detected face. The face detection unit 102 outputs coordinate information indicating the determined coordinates to the tracking management unit 104.

[0040] In a captured image, an image of a portion corresponding to a human face may be referred to as a face image. In a captured image, an outer frame surrounding a face image may be referred to as a face frame, and information indicating the face frame may be referred to as face frame information. Multiple face frames may exist in a captured image. Face frames smaller than a specific size may be discarded. For example, the size of the face frame may be compared with a preset size, and face frames smaller than the preset size may be discarded without being used in subsequent processing. Here, the preset size may be referred to as a minimum face frame size.

[0041] Note that instead of comparing the sizes of the face frames, a specific distance within the face frames may be compared. For example, the distance between the eyes of the face image within the detected face frame may be compared to a preset distance, and face frames having a distance shorter than the preset distance may be discarded and not used in subsequent processing. Note that the distance between the eyes may be the distance between the center points of the left and right eyes. The distance may be expressed in units such as meters or millimeters, or may be expressed in the number of pixels.

[0042] Skeleton detection unit 103 executes skeleton detection processing. In the skeleton detection processing, skeleton detection unit 103 performs image processing on the captured image, detects people present in the captured image, and generates skeleton information indicating the skeleton of the people. The generated skeleton information is output to tracking management unit 105 and holding determination unit 104. The skeleton information includes coordinate information indicating nodes corresponding to the head, shoulders, arms, joints, etc. of the human body. In the skeleton detection processing, likelihood information may also be added. The skeleton information will be described later.

[0043] The holding determination unit 104 determines whether a person corresponding to the skeletal information has held an IC card over the reading unit based on the skeletal information and the position of the reading unit of the ticket gate. The determination of whether a person corresponding to the skeletal information has held an IC card over the reading unit is sometimes referred to as holding determination. Note that information managed by the tracking management unit 105 (e.g., a person tracking list) may be used in the holding determination. For example, since the reading unit of a ticket gate is often located on the right side as viewed from the direction in which a passerby enters the ticket gate, the holding posture may differ depending on the passerby's dominant hand. For example, a left-handed person may hold an IC card over the gate using their left hand, so the holding determination unit 104 performs holding determination based on the positions of both hands of the person indicated by the skeletal information. Note that holding determination will be described later.

[0044] The holding determination unit 104 identifies the head position of a person who has been determined to have held an IC card by holding determination from the skeletal information of the person, and associates the area of ​​the head position with face frame information indicating the face frame detected by the face detection unit 102. In this way, the holding determination unit 104 identifies the face of the person who held the IC card.

[0045] Note that the holding determination unit 104 may determine whether or not the railroad user has performed an action other than holding the IC card over the ticket gate to pass through the ticket gate.

[0046] The tracking management unit 105 uses the face frame information and skeletal information to manage the position and direction of people passing through the ticket gate for each photographic frame and for each person. Information used to track people's movements is called a person tracking list. The person tracking list will be described later.

[0047] The tracking management unit 105 transmits a face matching request including the face image to the face authentication server 40 to request face matching of the face image included in the face frame information. The tracking management unit 105 records the face matching result received from the face authentication server 40 in the person tracking list.

[0048] When the tracking management unit 105 receives an identity verification request from the ticket gate control device 20, it identifies the person who is determined to have swept the IC card at the timing closest to the timing of receiving the identity verification request. The tracking management unit 105 determines a face matching ID indicating the person who swept the IC card based on the face matching result included in the person tracking list of the identified person, and transmits the verification result including the face matching ID to the ticket gate control device 20.

[0049] The communication unit 106 is an interface that performs communication between the face authentication control device 10 and the ticket gate control device 20, and communication between the face authentication control device 10 and the face authentication server 40. The tracking management unit 105 transmits information, etc. to the ticket gate control device 20 or the face authentication server 40 via the communication unit 106, and receives information, etc. from the ticket gate control device 20 or the face authentication server 40.

[0050] The ticket gate control device 20 includes a communication unit 201, a reading unit 202, a ticket gate determination control unit 203, and an equipment control unit 204. The ticket gate control device 20 may be configured as an information processing device (for example, a personal computer). At least one of the reading unit 202, the ticket gate determination control unit 203, and the equipment control unit 204 may be included in a processing unit or a processor.

[0051] The communication unit 201 is an interface that performs communication between the face authentication control device 10 and the ticket gate control device 20. The ticket gate determination control unit 203 transmits information, etc. to the face authentication control device 10 and receives information, etc. from the face authentication control device 10 via the communication unit 201.

[0052] The reading unit 202 is provided in the ticket gate, reads information from an IC card or the like, and outputs the information to the ticket gate determination control unit 203 .

[0053] Based on the read IC card information, the ticket gate determination control unit 203 determines whether the ticket indicated by the IC card information is a ticket that may pass through the ticket gate. For example, the ticket gate determination control unit 203 sends a ticket gate determination request including ticket information to the management server 50, and receives a ticket gate determination response from the management server 50. The ticket gate determination response includes information indicating whether the ticket indicated by the ticket gate determination request information is a ticket that may pass through the ticket gate, and information indicating whether the ticket indicated by the ticket gate determination request information is a ticket that requires identity verification.

[0054] If the ticket indicated by the information in the ticket examination determination request is not a ticket that may pass through the ticket gate, the ticket examination determination control unit 203 instructs the equipment control unit 204 to perform control to prevent the ticket from passing through the ticket gate.If the ticket indicated by the information in the ticket examination determination request is a ticket that may pass through the ticket gate and is not a ticket that requires personal identification, the ticket examination determination control unit 203 instructs the equipment control unit 204 to perform control to allow the ticket to pass through the ticket gate.

[0055] If the ticket indicated by the ticket inspection determination request information is a ticket that may pass through the ticket gate and requires identity verification, the ticket inspection determination control unit 203 sends an identity verification request to the face authentication control device 10.

[0056] The device control unit 204 controls devices provided in the ticket gate. The devices provided in the ticket gate include, for example, doors that block passage through the ticket gate passage, a display unit that presents information to people passing through the ticket gate, a speaker, etc. The device control unit 204 performs control to block passage through the ticket gate or control to allow passage through the ticket gate in accordance with instructions from the ticket gate determination control unit 203. For example, the control to block passage through the ticket gate includes at least one of the following: a control to close the doors, a control to display information on the display unit indicating that passage is not permitted, and a control to output a sound from the speaker indicating that passage is not permitted. For example, the control to allow passage through the ticket gate includes at least one of the following: a control to open the doors, a control to display information on the display unit indicating that passage is permitted, and a control to output a sound from the speaker indicating that passage is permitted.

[0057] The face authentication server 40 includes a communication unit 401 and a face matching unit 402. The face authentication server 40 may be configured by an information processing device (for example, a personal computer). The face matching unit 402 may be included in a processing unit or a processor.

[0058] The communication unit 401 is an interface that performs communication between the face authentication control device 10 and the face authentication server 40. The face matching unit 402 transmits information, etc. to the face authentication control device 10 via the communication unit 401, and receives information, etc. from the face authentication control device 10.

[0059] When the face matching unit 402 receives a face matching request from the face authentication control device 10, it performs face matching on the face image to be matched based on the information on the face matching target indicated by the face matching request. In the face matching process, a face matching score indicating the degree of similarity between the face image of the face matching target included in the face matching request and each of the registered face images is calculated. The higher the degree of similarity between the face images, the higher the face matching score. In the face matching process, if the highest face matching score calculated for each of the registered face images is equal to or greater than a threshold, it is determined that the registered face image corresponding to the highest face matching score matches the face image included in the face matching request. If the highest face matching score is less than the threshold, it is determined that the face image included in the face matching request does not match the face image of any of the registered people.

[0060] The face matching unit 402 transmits a face matching response obtained as a result of the face matching process to the face authentication control device 10. The face matching response is an example of a response to a face matching request. In the face matching process, if it is determined that the face image to be matched matches the face image of a specific person registered, the face matching response includes a face matching ID that identifies the specific person and a face matching score. In addition, in the face matching process, if it is determined that the face image to be matched does not match the face image of any of the registered people, the face matching response may include information indicating that identification was not possible (for example, information indicating that matching was NG) and a face matching score.

[0061] In a case where a face matching request is transmitted together with a face image of a person to be face matched and information on an IC card, face matching unit 402 may match whether or not the person is the user associated with the IC card, based on the face image of the person included in the face matching request and a face image registered by the user (registered user) associated with information on the held-over IC card. In this case, the face matching response includes information indicating whether or not the person is the user associated with the IC card.

[0062] <Example of skeletal information> Next, we will explain the skeletal information obtained by the skeletal detection process. The skeletal detection process is a process of detecting the joint positions of a human body from an image and generating a skeletal model based on the detected joint positions.

[0063] Fig. 3 is a diagram showing an example of skeleton information. Fig. 3 shows an example of a person's skeleton. S1 to S14 in Fig. 3 are examples of coordinates included in the skeleton information.

[0064] The skeletal information includes coordinates S1 to S14 shown in Fig. 3. For example, S1 corresponds to the position of the center of the face, S2, S3, and S4 correspond to the positions of the right shoulder, neck, and left shoulder, respectively. S5 and S6 correspond to the positions of the right elbow and right hand, respectively, and S7 and S8 correspond to the positions of the left elbow and left hand, respectively. S9 and S10 correspond to the base of the right foot and the base of the left foot, respectively. S11 and S12 correspond to the positions of the right knee and right ankle, and S13 and S14 correspond to the positions of the left knee and left ankle.

[0065] 3 indicates a face frame. The face frame may be referred to as face frame information. The face frame may be indicated by the coordinates of the four corners of the rectangle and / or the lengths of the four sides. In the skeleton information and face frame information of the same person, the face frame of the face frame information surrounds S1 of the skeleton information.

[0066] <Example of a person tracking list> Next, a person tracking list that includes skeleton information and face frame information of a specific person and is generated to track changes such as the movement of the specific person will be described.

[0067] FIG. 4 is a diagram showing a first example of a person tracking list. FIG. 4 shows list items for a certain passerby. The passerby is identified by a "passer number." A person tracking list is generated for each passerby. In the person tracking list corresponding to a passerby, information is generated for each captured image. Note that at least one of the items shown in FIG. 4 does not have to be included in the person tracking list, and items not shown in FIG. 4 may be included in the person tracking list.

[0068] A "passer number" is an identification number that identifies a passerby. Different identification numbers are assigned to different passersby. For example, if two passersby are included in one photographed image, the two passersby are assigned different identification numbers.

[0069] Hereinafter, the passer with passer number #k (k is an integer equal to or greater than 1) in Fig. 4 will be simply referred to as passer #k. Each item in the person tracking list for passer #k will be described below.

[0070] The "photographed frame number" is an identification number that identifies a frame of one photographed image. The photographed frame numbers may be in chronological order.

[0071] Below, the information in the person tracking list will be explained using as an example an item associated with a photographing frame number #i (i is an integer equal to or greater than 1). Also, the photographed image with photographing frame number #i will simply be described as photographing frame number #i.

[0072] "Camera number" is an identification number that identifies the camera that captured the frame with frame number #i. For example, if a passerby is captured by multiple cameras, the camera number is used to identify which of the multiple cameras captured the image.

[0073] "Image capture date and time" is the date and time when the captured frame number #i was captured.

[0074] The "skeleton detection score" is a score that indicates the accuracy of the skeleton detection process for passerby #k in photographed frame number #i. For example, the greater the range of the skeleton detected from the photographed image (e.g., the number of contact points), the higher the score. For example, the more body parts included in the photographed image, the higher the score.

[0075] The "face frame detection flag" indicates whether or not the face frame of passerby #k has been detected in photographic frame number #i.

[0076] "Face frame coordinates" indicate the coordinates of the face frame of passerby #k detected in shooting frame number #i. The face frame coordinates may represent four points corresponding to the four corners of a rectangle corresponding to the face frame. Alternatively, the face frame coordinates may represent a single point of the rectangle corresponding to the face frame (for example, the upper left point of the face frame).

[0077] "Face frame size" indicates the size of the face frame of passerby #k. The face frame size may be expressed by the vertical and horizontal lengths from the point indicated by the face frame coordinates. If the rectangle is a square, the face frame size may be expressed by the length of one side of the square.

[0078] The "face matching score" is the face matching score when face matching processing is performed on the face image of passerby #k included in shooting frame number #i. Note that if face matching processing has not been performed, the "face matching score" item may be blank (for example, None).

[0079] The "face matching ID" is an identification number for identifying passerby #k, obtained as a result of the face matching process. For example, if the face authentication server 40 has registration information for passerby #k, the "face matching ID" is the identification number included in the registration information for passerby #k. The fact that an identification number has been obtained means that the identity of passerby #k has been identified. Note that if face matching process has not been performed, the "face matching ID" item may be blank (for example, "None").

[0080] "Photographed image" is the photographed image with photographed frame number #i.

[0081] The "face frame image" is a face image of passerby #k obtained by extracting the area surrounded by the face frame of passerby #k from the captured image with shooting frame number #i. The "face frame image" may also be called a face image or a thumbnail.

[0082] "Holding judgment" indicates the result of the holding judgment process when the holding judgment process is performed on the skeleton of passerby #k based on the skeleton information of passerby #k detected in the captured image of shooting frame number #i. "Holding judgment" indicates, for example, "holding" or "not holding." Note that if holding judgment is not performed, "holding judgment" may be blank or may indicate that holding judgment was not performed.

[0083] "S1 coordinate" to "S14 coordinate" respectively indicate the coordinates S1 to S14 of the skeleton of pedestrian #k detected in the captured image with frame number #i. Note that it is not necessary to obtain all of the coordinates S1 to S14 in the skeleton detection process. For example, if the right ankle is not visible in the captured image, S12 will not be detected, so "S12 coordinate" may be blank or may indicate that it was not detected.

[0084] Note that the person tracking list illustrated in Fig. 4 is an example of a form in which information about passersby is associated and recorded, and the present disclosure is not limited to this. The information for each item illustrated in Fig. 4 may be associated and recorded in a form different from that of Fig. 4.

[0085] <First example of wave detection> Fig. 5 is a diagram showing a first example of the hold-up determination. Fig. 5 shows the state before a passerby holds up their IC card and the state after holding up their IC card. In each state, the side wall of the ticket gate and the "hold-up determination area" are shown.

[0086] In the example of Figure 5, a "determination marker" is provided near the IC card reader on the side wall of the ticket gate on the entrance side. In the example of Figure 5, the determination marker is a linear mark extending horizontally in the figure.

[0087] The holding-over detection area is a part of the captured image that corresponds to the area surrounding the IC card reader and the detection marker.

[0088] The holding determination unit 104 performs holding determination based on whether or not the determination marker is detected as a single straight line in the holding determination area of ​​the captured image.

[0089] For example, in the state before the IC card is held over the device in Fig. 5, the determination marker is detected as a single straight line in the holding determination area of ​​the captured image. When the determination marker is detected as a single straight line in this way, holding determination unit 104 determines that the IC card is not being held over the device.

[0090] For example, in the state in Fig. 5 where the IC card is held over the device, the determination marker is not detected as a single straight line in the holding determination area of ​​the captured image, but is detected as being divided into two line segments. When the determination marker is detected as being divided into two line segments in this way, holding determination unit 104 determines that holding is occurring.

[0091] Note that while FIG. 5 shows an example in which a "determination marker" is provided near an IC card reader provided on the side wall of the ticket gate on the entrance side, the present disclosure is not limited to this. The position, shape, and size of the "determination marker" are not limited to the example shown in FIG. 5. Also, while FIG. 5 shows an example in which a "determination marker" is provided on the ticket gate itself, the "determination marker" may be provided within the holding determination area of ​​the acquired captured image rather than on the ticket gate itself. For example, a virtual straight-line determination marker may be provided within the holding determination area, and it may be determined that holding is present when the virtual straight line intersects with the skeleton information.

[0092] <Second example of wave detection> Fig. 6A is a diagram showing a second example of wave detection. Fig. 6B is a diagram showing four cases in Fig. 6A. Fig. 6A shows the skeleton of a passerby holding an IC card. Fig. 6A also shows the side wall of the ticket gate and the "wave detection area."

[0093] The holding detection area is a part of the captured image and is used for holding detection. The holding detection area corresponds to the area in the captured image that surrounds the IC card reader. In Figure 6A, the four corners of the holding detection area are represented as points a to d, respectively.

[0094] In the example of Fig. 6A, S6 is included in the holding judgment area. Also, in the example of Fig. 6A, the vector of the line connecting S5 to S6 intersects with side ab. In this way, when S6 is included in the holding judgment area and it is detected that the vector of the line connecting S5 to S6 intersects with side ab, holding judgment unit 104 determines that holding is occurring.

[0095] 6B shows four cases in which it is determined that there is holding. The rectangle connecting points a to d in each case corresponds to the holding determination area in FIG. 6A.

[0096] 6B, similar to FIG. 6A, is a case where S6 is included in the holding detection area and the vector of the line connecting S5 to S6 intersects with side ab. When such a case is detected, holding detection unit 104 determines that holding is occurring.

[0097] Case 2 in Figure 6B corresponds to a case where the right wrist corresponding to S6 is not captured in the captured image. In such a case 2, in the skeleton detection process, S6 is interpolated based on a straight line vector corresponding to the skeleton extending in the direction from S5 to S6, the average length between S5 and S6, etc. The interpolated S6 will be referred to as the interpolated S6 below.

[0098] In Case 2 of Fig. 6B, interpolation S6 is included in the holding judgment area. Also, in Case 2 of Fig. 6B, the line vector connecting S5 to interpolation S6 intersects with side ab. In this way, when interpolation S6 is included in the holding judgment area and it is detected that the line vector connecting S5 to interpolation S6 intersects with side ab, holding judgment unit 104 judges that holding is occurring.

[0099] 6B is a case where S6 is included in the holding detection area, and the vector of the line connecting S5 to S6 intersects with side bc. When such a case is detected, holding detection unit 104 determines that holding is occurring.

[0100] 6B corresponds to a case where the right wrist corresponding to S6 is not visible in the captured image, similar to case 2. In such a case 4, in the skeleton detection process, S6 is interpolated based on the vector of a straight line corresponding to the skeleton extending in the direction from S5 to S6, the average length between S5 and S6, etc.

[0101] In Case 4 of Fig. 6B, interpolation S6 is included in the holding judgment area. Also, in Case 4 of Fig. 6B, the line vector connecting S5 to interpolation S6 intersects with side bc. In this way, when interpolation S6 is included in the holding judgment area and it is detected that the line vector connecting S5 to interpolation S6 intersects with side bc, holding judgment unit 104 judges that holding is occurring.

[0102] <Third example of wave detection> The second example of the above wave detection corresponds to a case where a passerby holds an IC card held in their right hand over the reader. The third example below corresponds to a case where a passerby holds an IC card held in their left hand over the reader.

[0103] FIG. 7A is a diagram showing a third example of the wave detection. FIG. 7B is a diagram showing four cases in FIG. 7A. FIG. 7A shows a state in which a passerby holds an IC card over the ticket gate. FIG. 7A also shows the side wall of the ticket gate and the "wave detection area." In FIG. 7A and FIG. 7B, as in FIG. 6A and FIG. 6B, the four corners of the wave detection area are represented as points a to d, respectively.

[0104] In the example of Fig. 7A, S8 is included in the holding judgment area. Also, in the example of Fig. 7A, the vector of the line connecting S7 to S8 intersects with side ab. In this way, when S8 is included in the holding judgment area and it is detected that the vector of the line connecting S7 to S8 intersects with side ab, holding judgment unit 104 judges that holding is occurring.

[0105] 7B shows four cases in which it is determined that there is holding of the phone. The rectangle connecting points a to d in each case corresponds to the holding of the phone determination area in FIG. 7A.

[0106] 7B, similar to FIG. 7A, is a case where S8 is included in the holding determination area and the vector of the line connecting S7 to S8 intersects with side ab. When such a case is detected, holding determination unit 104 determines that holding is occurring.

[0107] Case 2 in Figure 7B corresponds to a case where the left wrist corresponding to S8 is not visible in the captured image. In such a case 2, in the skeleton detection process, S8 is interpolated based on a straight line vector corresponding to the skeleton extending in the direction from S7 to S8, the average length between S7 and S8, etc. The interpolated S8 will be referred to as the interpolated S8 below.

[0108] In Case 2 of Fig. 7B, interpolation S8 is included in the holding judgment area. Also, in Case 2 of Fig. 7B, the line vector connecting S7 to interpolation S8 intersects with side ab. In this way, when interpolation S8 is included in the holding judgment area and it is detected that the line vector connecting S7 to interpolation S8 intersects with side ab, holding judgment unit 104 judges that holding is occurring.

[0109] 7B is a case where S8 is included in the holding determination area, and the vector of the line connecting S7 to S8 intersects with side bc. When such a case is detected, holding determination unit 104 determines that holding is occurring.

[0110] 7B corresponds to a case where the left wrist corresponding to S8 is not visible in the captured image, similar to case 2. In such a case 4, in the skeleton detection process, S8 is interpolated based on the vector of a straight line corresponding to the skeleton extending in the direction from S7 to S8, the average length between S7 and S8, etc.

[0111] In Case 4 of Fig. 7B, interpolation S8 is included in the holding judgment area. Also, in Case 4 of Fig. 7B, the line vector connecting S7 to interpolation S8 intersects with side bc. In this way, when interpolation S8 is included in the holding judgment area and it is detected that the line vector connecting S7 to interpolation S8 intersects with side bc, holding judgment unit 104 judges that holding is occurring.

[0112] <Fourth example of holding something up> The second and third examples of the above-mentioned wave judgment correspond to cases where a passerby approaches the ticket gate from the front, which corresponds to an extension of the ticket gate passage. The fourth example corresponds to a case where a passerby approaches the ticket gate from the right side of the front, which corresponds to an extension of the ticket gate passage, and waves the IC card held in their right hand over the reader.

[0113] FIG. 8A is a diagram showing a fourth example of the wave detection. FIG. 8B is a diagram showing four cases in FIG. 8A. FIG. 8A shows a state in which a passerby holds an IC card over the ticket gate. FIG. 8A also shows the side wall of the ticket gate and the "wave detection area." In FIGS. 8A and 8B, as in FIGS. 6A and 6B, the four corners of the wave detection area are represented as points a to d, respectively.

[0114] In the example of Fig. 8A, S6 is included in the holding judgment area. Also, in the example of Fig. 8A, the right hand is extended from the right side rather than the front and the IC card is held over the card, so the vector of the line connecting S5 to S6 intersects with side ad. In this way, when S6 is included in the holding judgment area and it is detected that the vector of the line connecting S5 to S6 intersects with side ad, holding judgment unit 104 judges that holding is occurring.

[0115] 8B shows two cases in which it is determined that there is holding of the phone. The rectangle connecting points a to d in each case corresponds to the holding of the phone determination area in FIG. 8A.

[0116] 8B, similar to FIG. 8A, is a case where S6 is included in the holding detection area and the vector of the line connecting S5 to S6 intersects with side ad. When such a case is detected, holding detection unit 104 determines that holding is occurring.

[0117] 8B corresponds to a case where the right wrist corresponding to S6 is not captured in the captured image. In such a case 2, in the skeleton detection process, S6 is interpolated based on a straight line vector corresponding to the skeleton extending in the direction from S5 to S6, the average length between S5 and S6, etc.

[0118] In Case 2 of Fig. 8B, interpolation S6 is included in the holding judgment area. Also, in Case 2 of Fig. 8B, the vector of the line connecting S5 to interpolation S6 intersects with side ad. In this way, when interpolation S6 is included in the holding judgment area and it is detected that the vector of the line connecting S5 to interpolation S6 intersects with side ad, holding judgment unit 104 judges that holding is occurring.

[0119] <Fifth example of holding something up> The fifth example of the wave detection corresponds to a case where a passerby approaches the ticket gate from the right side of the front, which corresponds to the extension of the ticket gate passage, and waves the IC card held in their left hand over the reading part.

[0120] FIG. 9A is a diagram showing a fifth example of the wave detection. FIG. 9B is a diagram showing four cases in FIG. 9A. FIG. 9A shows a state in which a passerby holds an IC card over the ticket gate. FIG. 9A also shows the side wall of the ticket gate and the "wave detection area." In FIGS. 9A and 9B, as in FIGS. 6A and 6B, the four corners of the wave detection area are represented as points a to d, respectively.

[0121] In the example of Fig. 9A, S8 is included in the holding judgment area. Also, in the example of Fig. 9A, the vector of the line connecting S7 to S8 intersects with side ad. In this way, when S8 is included in the holding judgment area and it is detected that the vector of the line connecting S7 to S8 intersects with side ad, holding judgment unit 104 judges that holding is occurring.

[0122] 9B shows two cases in which it is determined that there is holding. The rectangle connecting points a to d in each case corresponds to the holding determination area in FIG. 9A.

[0123] 9B, similar to FIG. 9A, is a case where S8 is included in the holding determination area and the vector of the line connecting S7 to S8 intersects with side ad. When such a case is detected, holding determination unit 104 determines that holding is occurring.

[0124] 9B corresponds to a case where the left wrist corresponding to S8 is not captured in the captured image. In such a case 2, in the skeleton detection process, S8 is interpolated based on a straight line vector corresponding to the skeleton extending in the direction from S7 to S8, the average length between S7 and S8, etc.

[0125] In Case 2 of Fig. 9B, interpolation S8 is included in the holding judgment area. Also, in Case 2 of Fig. 9B, the vector of the line connecting S7 to interpolation S8 intersects with side ad. In this way, when interpolation S8 is included in the holding judgment area and it is detected that the vector of the line connecting S7 to interpolation S8 intersects with side ad, holding judgment unit 104 judges that holding is occurring.

[0126] The second to fifth examples described above may be used in combination. For example, if S6 or interpolated S6 is included in the holding judgment area and the vector of the line connecting S5 to S6 or interpolated S6 intersects with any one of sides ab, bc, and ad, holding judgment unit 104 will determine that there is holding. Alternatively, even if S6 or interpolated S6 is not included in the holding judgment area, if S8 or interpolated S8 is included in the holding judgment area and the vector of the line connecting S7 to S8 or interpolated S8 intersects with any one of sides ab, bc, and ad, holding judgment unit 104 will determine that there is holding.

[0127] <Example of person tracking processing: A case where multiple people are trying to enter a ticket gate> Next, the person tracking process and updating of the person tracking list will be described. As an example, the person tracking process for each of two people in a case where multiple people are trying to enter a ticket gate will be described.

[0128] Fig. 10A is a diagram showing an example of a case where two people are about to enter a ticket gate, and Fig. 10B is a diagram showing an example of a person tracking list corresponding to Fig. 10A.

[0129] 10A, in a case where two people, pedestrian A and pedestrian B, are about to enter a ticket gate, the face frames of the two people may be detected in the face frame detection process, and the skeleton information of the two people may be detected in the skeleton detection process. In such a case, a person tracking list is created for each of the two people.

[0130] A face frame detection process and a skeleton detection process are performed for each captured image (each captured frame number), and the face frame information and skeleton information are linked to the created person tracking list.

[0131] In the person tracking process, when the face frames and / or skeletons of multiple people are detected, the face frame information and / or skeleton information in the person tracking list that has already been created are compared, and linked to the person tracking list that is closest in distance.

[0132] For example, in a case where a created person tracking list for person A and a created person tracking list for person B exist, if two face frames, face frame a and face frame b, are detected in a captured image, the coordinates of face frame A indicated by the latest face frame information in the person tracking list for person A are compared with the coordinates of face frame a and the coordinates of face frame b, respectively, and the coordinates of face frame B indicated by the latest face frame information in the person tracking list for person A are compared with the coordinates of face frame a and the coordinates of face frame b, respectively. Then, if the coordinates of face frame A are closer to the coordinates of face frame a than the coordinates of face frame b, face frame a is linked to the person tracking list for person A, and if the coordinates of face frame B are closer to the coordinates of face frame b than the coordinates of face frame a, face frame b is linked to the person tracking list for person B.

[0133] In addition, if the distance between the face frame detected in the captured image and the face frame indicated by the latest face frame information in the created person tracking list is greater than a predetermined distance, the face frame detected in the captured image does not need to be linked to the created person tracking list.

[0134] For skeletons, similarly to face frames, for example, when two skeletons, skeleton a and skeleton b, are detected in a captured image, any coordinate (for example, at least one of S1 to S14) of skeleton A indicated by the latest skeleton information in the person tracking list of person A is compared with the corresponding coordinates of skeleton a and skeleton b, and skeleton B indicated by the latest skeleton information in the person tracking list of person A is compared with skeleton a and skeleton b. Then, if skeleton A is closer to skeleton a than skeleton b, skeleton a is linked to person A's person tracking list, and if skeleton B is closer to skeleton b than skeleton a, skeleton b is linked to person B's person tracking list.

[0135] For each person in the person tracking list, a holding judgment is performed based on the skeleton information, and the holding judgment result is recorded in the person tracking list.

[0136] FIG. 10B shows the results of the holding determination for each photographic frame number included in the person tracking list for each of passersby A and B.

[0137] When the face authentication control device 10 receives a request for identity verification, it searches the person tracking list and determines the person with "Holding a headset" as the person to be verified. In the example of Fig. 10B, passerby B is the person to be verified.

[0138] <Example of processing flow of face authentication control device 20> Next, an example of the flow of processing by the face authentication control device 10 including the holding determination of the second to fifth examples described above will be described.

[0139] FIG. 11A is a flowchart showing an example of processing performed by the face authentication control device 10 in this embodiment.

[0140] The face authentication control device 10 takes a photograph (ST11).

[0141] The face authentication control device 10 performs a skeleton detection process (ST12), which provides skeleton information.

[0142] If a skeleton is not detected, the skeleton information may be blank.

[0143] The face authentication control device 10 performs a face frame detection process (ST13). Face frame information is obtained by the face frame detection process. If a face frame is not detected in the face frame detection process, the face frame information may be blank.

[0144] In FIG. 11A, the skeleton detection process is executed before the face frame detection process, but the face frame detection process may be executed before the skeleton detection process, and the order of the processes does not matter.

[0145] The face authentication control device 10 transmits a face matching request to the face authentication server 40, requesting face matching of the face image surrounded by the detected face frame, and receives a face matching result from the face authentication server 40 (ST14). Note that if a face frame is not detected, the face authentication control device 10 does not transmit a face matching request.

[0146] Next, the face authentication control device 10 performs holding determination (ST15). The flow of the holding determination process will be described later. Note that in Fig. 11A, the transmission of the face matching request is executed before the holding determination process, but the holding determination process may be executed before the transmission of the face matching request, and the order of the processes does not matter.

[0147] The face authentication control device 10 determines whether or not the face is held up (ST16).

[0148] If it is determined that there is holding (YES in ST16), the face authentication control device 10 performs a person tracking process, identifies a person tracking list corresponding to the skeleton of the object of the holding judgment, and records the judgment result of "holding" in the identified person tracking list (ST17).

[0149] If it is not determined that there is holding (NO in ST16), that is, if it is determined that there is no holding, the face authentication control device 10 identifies a person tracking list corresponding to the skeleton of the object of the holding judgment by person tracking processing, and records the determination result of ``no holding'' in the identified person tracking list (ST18).

[0150] The face authentication control device 10 determines whether or not an identity verification request has been received from the ticket gate control device 20 (ST19).

[0151] If an identity verification request has not been received (NO in ST19), the flow returns to ST11. Note that cases in which an identity verification request is not received even though the IC card is held over the card include, for example, cases in which the ticket shown on the IC card is one that is not permitted to pass through the ticket gate, or cases in which the ticket shown on the IC card is one that does not require identity verification. A ticket that does not require identity verification is, for example, a ticket that is not limited to certain purchasers, or a ticket that does not require any specific qualifications or conditions for purchase.

[0152] When the request for identity verification is received (YES in ST19), the face authentication control device 10 searches the person tracking list and identifies the person to be identified (ST20). For example, the face authentication control device 10 determines the person in the person tracking list associated with the determination result of "present with headset" as the person to be identified.

[0153] The face authentication control device 10 extracts the face matching result of the identified person (ST21). For example, in ST21, the face matching result (for example, face matching ID) associated with the determination result of "with headset" is extracted from the person tracking list of the identified person.

[0154] The face authentication control device 10 transmits a confirmation result based on the face matching result to the ticket gate control device 20 (ST22). The confirmation result includes a face matching ID that identifies the person who is the subject of identity verification.

[0155] 11A indicates that the person who was the subject of face matching is a person who is permitted to pass through the ticket gate without holding up an IC card, the face authentication control device 10 may notify the ticket gate control device 20 that the person is permitted to pass through the ticket gate before receiving an identity verification request from the ticket gate control device 20. For example, the face authentication server 40 may register in advance whether a registered user is a person who is permitted to pass through the ticket gate without holding up an IC card. In this case, the determination of whether the person holds up an IC card may be omitted. For example, in a case where both IC card authentication and face authentication are used depending on the passerby, a process of permitting passage based on face matching without holding up an IC card may be performed.

[0156] Next, the flow of the holding determination process executed in ST15 of FIG. 11A will be described.

[0157] Fig. 11B is a flowchart showing an example of the holding determination process in ST15 of Fig. 11A. Fig. 11B is a flowchart corresponding to an example of the holding determination in the second to fifth examples described above.

[0158] The face authentication control device 10 extracts skeleton information of the skeleton of the object of the holding judgment from the person tracking list (ST31).

[0159] The face authentication control device 10 calculates a right arm vector (ST32). The right arm vector is a vector corresponding to the position of the person's right arm in the skeletal information. In the second and fourth examples of the holding judgment described above, the right arm vector corresponds to the vector from S5 corresponding to the right elbow to S6 corresponding to the right wrist.

[0160] The face authentication control device 10 determines whether the right arm vector is included in the holding judgment area (ST33). For example, in the second and fourth examples of the holding judgment described above, if S6 is within the holding judgment area and the right arm vector intersects with any of the sides ab, bc, and da of the holding judgment area, the face authentication control device 10 determines that the right arm vector is included in the holding judgment area. For example, in the second and fourth examples of the holding judgment described above, if S6 is not within the holding judgment area or the right arm vector does not intersect with any of the sides ab, bc, and da of the holding judgment area, the face authentication control device 10 determines that the right arm vector is not included in the holding judgment area.

[0161] If the right arm vector is included in the holding determination area (YES in ST33), the face authentication control device 10 determines that there is holding (ST34), and the flow of holding determination ends.

[0162] If the right arm vector is not included in the holding determination area (NO in ST33), the face authentication control device 10 calculates the left arm vector (ST35). The left arm vector is a vector corresponding to the position of the person's left arm in the skeleton information. In the third and fifth examples of holding determination described above, the left arm vector corresponds to the vector from S7 corresponding to the left elbow to S8 corresponding to the left wrist.

[0163] The face authentication control device 10 determines whether the left arm vector is included in the holding judgment area (ST36). For example, in the third and fifth examples of the holding judgment described above, if S8 is within the holding judgment area and the left arm vector intersects with any of the sides ab, bc, and da of the holding judgment area, the face authentication control device 10 determines that the left arm vector is included in the holding judgment area. For example, in the third and fifth examples of the holding judgment described above, if S8 is not within the holding judgment area or the left arm vector does not intersect with any of the sides ab, bc, and da of the holding judgment area, the face authentication control device 10 determines that the left arm vector is not included in the holding judgment area.

[0164] If the left arm vector is included in the holding determination area (YES in ST36), the face authentication control device 10 determines that there is holding (ST34), and the flow of holding determination ends.

[0165] If the left arm vector is not included in the holding determination area (NO in ST36), the face authentication control device 10 determines that there is no holding (ST37), and the flow of holding determination ends.

[0166] <Variations of holding judgment> Fig. 12 is a diagram showing variations of the wave determination. Similar to Fig. 6A etc., Fig. 12 shows a state in which a passerby holds up an IC card. Also, similar to Fig. 6A etc., Fig. 12 shows the side wall of the ticket gate and the "wave determination area." In addition to Fig. 6A etc., Fig. 12 also shows the "wave point foot area."

[0167] The holding point foot area is a part of the captured image and is an area used for holding determination. The holding point foot area corresponds to the area surrounding the feet and the area above the knee height when a passerby assumes a position to hold their IC card over the IC card reader in the captured image. For example, the holding point foot area corresponds to the area surrounding the position of the feet where a passerby is expected to stand when they assume a position to hold their IC card over the IC card reader in the captured image. Holding determination using the holding point foot area may be used in combination with holding determination using the holding determination area.

[0168] 12, the holding point foot area includes S11 corresponding to the right knee, S12 corresponding to the right ankle, S13 corresponding to the left knee, and S14 corresponding to the left ankle. In this way, when it is detected that S11 to S14 are included in the holding point foot area, it can be determined that the face frame at the position corresponding to the coordinates included in the holding point foot area is the face frame of the passerby who held the IC card.

[0169] Note that, in the example of FIG. 12, holding determination unit 104 determines that there is holding when S11 to S14 are included in the holding point foot area, but the present disclosure is not limited to this. For example, holding determination unit 104 may determine that there is holding when at least N (N is an integer between 1 and 4) of S11 to S14 are included in the holding point foot area. Alternatively, the holding point foot area may be set to an area that includes the right ankle and the left ankle, but does not include both knees. In this case, holding determination unit 104 may determine that there is holding when at least one of S12 and S14 is included.

[0170] For example, the holding determination unit 104 may determine that there is holding only when it determines that there is holding in both the holding determination using the holding determination area and the holding auxiliary determination using the holding point foot area.

[0171] Whether to use the holding detection using the holding detection area or how to combine these two holding detection methods may be determined based on the camera's shooting range, the structure of the ticket gate, the accuracy required for holding detection, etc. For example, if the camera's shooting range does not include the feet of a passerby, holding detection using the holding detection area is adopted.

[0172] In the above-described holding judgment, if it is judged that there is holding, S1 of the skeletal information of the judgment target is compared with the face frame information, and it is judged whether or not the person corresponding to the face frame information corresponds to the person corresponding to the skeletal information of the judgment target. For example, as in the example of FIG. 12, if S1 of the skeletal information of the judgment target is surrounded by a face frame, it is judged that the person corresponding to the face frame information corresponds to the person corresponding to the skeletal information of the judgment target. For example, if S1 of the skeletal information of the judgment target is located outside the face frame, it is judged that the person corresponding to the face frame information does not match the person corresponding to the skeletal information of the judgment target.

[0173] <Case 1 of wave detection> FIG. 13 is a diagram showing an example of Case 1 of wave detection. FIG. 13 shows the skeletons of two passersby, Passer A and Passer B, who enter the ticket gate consecutively. In the case of FIG. 13, Passer A and Passer B each enter the ticket gate from the front, and Passer A, who is in front of Passer B, holds his / her IC card over the gate. FIG. 13 also shows the "wave detection area" similar to FIG. 6A etc. and the "wave point foot area" similar to FIG. 12.

[0174] In the example of FIG. 13, the faces of pedestrians A and B are included in the photographed range, so the face frames of pedestrians A and B are detected in the photographed image.

[0175] In the example of Fig. 13, S6 of passerby A is included in the holding judgment area. Also, in the example of Fig. 13, the vector of the line connecting S5 to S6 of passerby A intersects with side ab. In this way, when S6 of passerby A is included in the holding judgment area and when it is detected that the vector of the line connecting S5 to S6 of passerby A intersects with side ab, holding judgment unit 104 judges that passerby A is holding up his / her hand.

[0176] Furthermore, the holding point foot area includes S11 corresponding to passerby A's right knee, S12 corresponding to his right ankle, S13 corresponding to his left knee, and S14 corresponding to his left ankle. When it is detected that S11 to S14 are included in the holding point foot area, the face frame information of the two face frames, S1 of the skeletal information of passerby A and S1 of the skeletal information of passerby B, who are the subject of judgment, is compared. In the example of FIG. 13, of the two face frames, the face frame corresponding to the holding point foot area is the face frame information of the face frame surrounding S1 of passerby A's skeletal information, so it is determined that the person corresponding to the face frame information of the face frame surrounding S1 of passerby A's skeletal information is the person corresponding to the skeletal information of the judgment subject.

[0177] In the example of Figure 13, the face frame detection process detects the face frames of two people, passerby A and passerby B. However, even in cases where multiple face frames are detected as described above, it is possible to determine which of the two people, passerby A or passerby B, has held up the IC card by comparing S1, the skeletal information of passerby A, who is the subject of judgment and has been determined to have held up the IC card, with the face frame information of the two face frames.

[0178] <Case 2 of wave detection> FIG. 14 is a diagram showing an example of Case 2 of the wave detection. FIG. 14 shows the skeletons of two passersby, Passer A and Passer B, as they enter a ticket gate. In the case of FIG. 14, Passer B enters the ticket gate from the front, while Passer A enters from the right side of the ticket gate, cutting in. In the case of FIG. 14, Passer A, whose face frame is not included in the camera's shooting range, is waving his / her IC card, rather than Passer B, whose face frame is included in the camera's shooting range. FIG. 14 also shows a "wave detection area" similar to that of FIG. 6A etc.

[0179] At the stage shown in Fig. 14, the reading unit reads the information from the IC card and sends an identity verification request to the face authentication control device 10. In Fig. 14, the face of passerby A is not included in the camera's shooting range, but the face of passerby B is included, so the image captured by the camera does not include the face of passerby A, but includes the face of passerby B.

[0180] According to the above-mentioned holding judgment, S8 of passerby A is included in the holding judgment area, and the vector of the straight line connecting S7 to S8 intersects with side ab of the holding judgment area, so holding judgment unit 104 judges that passerby A is holding his / her hand.

[0181] 14, however, the face of pedestrian A is not included in the camera's shooting range, and therefore S1 of pedestrian A is not detected in the skeleton detection process based on the captured image. In other words, S1 is not included in the skeleton information of pedestrian A.

[0182] As described above, even in cases where multiple face frames are detected, S1 is not included in the skeletal information of passerby A, who is the subject of the judgment that an IC card has been held up, so it is possible to avoid making the mistaken judgment that passerby B has held up an IC card.

[0183] In this way, by comparing the skeletal information and face frame information of the object to be determined as having held the IC card, it is possible to improve the accuracy of determining who is holding the IC card. For example, as in the example of Fig. 14, even if the captured image does not include the face of passerby A but does include the face of passerby B, and passerby A is holding the IC card, it is possible to avoid making an erroneous determination that the person holding the IC card is passerby B.

[0184] 14, it is determined that the person holding the IC card is not Passerby B who appears in the captured image, but it is not determined that the person holding the IC card is Passerby A. In this case, the identity verification result is not sent in response to the identity verification request, or the sending of the identity verification result is put on hold.

[0185] Also, as in the example of Figure 14, if skeletal information of a person other than passerby B is detected, a new person tracking list different from passerby B may be created at that point, and tracking of that person may be started using skeletal detection processing and face frame detection processing.

[0186] In the example of FIG. 14, a person tracking list for pedestrian B is created before that for pedestrian A, and tracking of pedestrian B begins. While tracking of pedestrian B has begun, pedestrian A interrupts. In this example, the skeleton detection process detects S4, S7, and S8, which are the left arm of pedestrian A. A new person tracking list is created for S4, S7, and S8, which are the left arm of pedestrian A.

[0187] Then, a holding-up determination is made based on the skeleton information of passerby A. In the example of Fig. 14, the holding-up determination determines that passerby A is holding up. As a result of this determination, the information "holding up" is recorded in the person tracking list for passerby A.

[0188] Upon receiving an identity verification request from the ticket gate, the face authentication control device 10 searches for a person tracking list in which the holding judgment has confirmed "holding" from among the person tracking lists. In the example of FIG. 14, of the person tracking list of passerby A and the person tracking list of passerby B, the person tracking list of passerby A corresponds to the person tracking list in which "holding" has been confirmed. The face authentication control device 10 searches for the person tracking list of passerby A with the highest face matching score. At the stage of FIG. 14, the face frame of passerby A has not been detected. In this case, the face authentication control device 10 notifies the ticket gate that face authentication is incomplete or suspends its response for a certain period of time. If the response is suspended, the face authentication control device 10 continues to perform the skeleton detection process and face frame detection process on the captured image captured after the stage of FIG. 14. Then, if the face frame of passerby A is detected in the face frame detection process, the face matching process of the face frame is performed.

[0189] <Case 3 of wave detection> FIG. 15A is a diagram showing an example of case 3 of wave detection. FIG. 15B is a diagram showing an example of changes in the person tracking list in case 3 shown in FIG. 15A. FIG. 15A shows the skeletons of two passersby, passerby A and passerby B, who enter the ticket gate consecutively. In the case of FIG. 15A, passerby A and passerby B each enter the ticket gate from the front, and passerby B, who is behind passerby A, holds up his IC card. FIG. 15A also shows a "wave detection area" similar to that shown in FIG. 6A and the like.

[0190] In the example of FIG. 15A, the faces of pedestrians A and B are included in the photographed range, so the face frames of pedestrians A and B are detected in the photographed image.

[0191] In the example of Fig. 15A, S6 of passerby B is included in the holding judgment area. Also, in the example of Fig. 15A, the vector of the line connecting S5 to S6 of passerby B intersects with side ab. In this way, when S6 of passerby B is included in the holding judgment area and the vector of the line connecting S5 to S6 of passerby B is detected to intersect with side ab, holding judgment unit 104 judges that passerby B is holding up his / her hand.

[0192] Case 3 of wave judgment can also be applied to cases where IC card authentication and facial authentication are used together. For example, this corresponds to a case where a judgment of whether or not a pedestrian can pass through a ticket gate based on facial authentication of the pedestrian's face and a judgment of whether or not a pedestrian can pass through a ticket gate based on the pedestrian's IC card when the pedestrian holds the card over the gate are used in combination.

[0193] At the stage in FIG. 15A, the reading unit reads the information from the IC card and notifies the face authentication control device 10 that the IC card has been held over the face authentication control device 10.

[0194] For example, when the face authentication control device 10 is notified that an IC card has been swiped, it stops determining whether or not to allow passage through the ticket gate based on face authentication and instead determines whether or not to allow passage through the ticket gate based on the IC card. In other words, when the face authentication control device 10 is notified that an IC card has been swiped, it switches the determination of whether or not to allow passage through the ticket gate from a determination based on face authentication to a determination based on the IC card.

[0195] For example, if the system switches to IC card-based judgment without performing the waving judgment, the judgment based on face authentication of passerby A will be canceled and judgment will be made based on passerby A's IC card. In other words, if the waving judgment is not performed, in the case of Fig. 15A, it will be judged that passerby A is waving his IC card even though passerby B is waving his IC card.

[0196] When performing a hold-up determination in FIG. 15A, S1 of the skeletal information of the person holding up the IC card is included in the face frame of passerby B, not passerby A, and therefore it is determined that the person holding up the IC card is passerby B, not passerby A. For example, as shown in FIG. 15B, when a person tracking list for passerby A and a person tracking list for passerby B are created by the process of tracking two people, it can be confirmed that the person determined to have held up the card is passerby B. This makes it possible to determine whether passerby A is allowed to pass based on face authentication, and whether passerby B is allowed to pass based on the IC card.

[0197] <Example of a sequence showing the flow of operations> Fig. 16A and Fig. 16B are sequence diagrams showing the processing flow of each device in this embodiment. Fig. 16A and Fig. 16B show the processing of the ticket gate control device 20, the face authentication control device 10, and the face authentication server 40. The processing of the face authentication control device 10 is divided into processing related to photography and detection and processing related to person tracking. Also, A1 to A5 in Fig. 16A are connected to A1 to A5 in Fig. 16B, respectively. Fig. 16A and Fig. 16B correspond to an example in which it is determined that there is holding of the handset in the second holding of the handset judgment.

[0198] A user (passenger) approaches the ticket gate (S1).

[0199] The face authentication control device 10 takes an image of the user (S102).

[0200] The face authentication control device 10 performs a bone structure detection process on the captured image (S103), and bone structure information is obtained by S103.

[0201] The face authentication control device 10 performs a face frame detection process on the captured image (S104). Face frame information is obtained by S104. Note that in Fig. 16A, the skeleton detection process is executed before the face frame detection process, but the face frame detection process may be executed before the skeleton detection process, and the order of the processes does not matter.

[0202] The face authentication control device 10 performs a holding determination (S105). At the stage of S105, for example, the user has not performed the action of holding the IC card. Therefore, at the stage of S105, the holding determination determines that there is no holding.

[0203] The skeleton information obtained in S103, the face frame information obtained in S104, and the determination result of "no holding" in the holding determination in S105 are output to the tracking management unit 105 (S106).

[0204] The face recognition control device 10 (for example, the tracking management unit 105) performs a person tracking process (S201). In the person tracking process, the skeleton information and face frame information acquired in S106 are compared with the skeleton information and face frame information of a person tracking list generated in the past, and based on the change over time in the position and shape of the skeleton indicated by the skeleton information and / or the change over time in the position and shape of the face frame indicated by the face frame information, it is determined which person's person tracking list information the skeleton information and face frame information acquired in S106 correspond to. The face recognition control device 10 records the skeleton information and face frame information acquired in S106 in the person tracking list determined by the person tracking process.

[0205] The face authentication control device 10 transmits a face matching request to the face authentication server 40 to request face matching processing of the face image included in the face frame information (S202).

[0206] The face authentication server 40 receives the face matching request and performs face matching processing (S401).

[0207] The face authentication server 40 transmits a face matching response obtained as a result of the face matching process to the face authentication control device 10 (S402).

[0208] The tracking management unit 105 of the face authentication control device 10 records the received face matching result in the person tracking list (S203).

[0209] A user (passenger) holds an IC card over the reader of the ticket gate (S2). The ticket gate control device 20 reads the information on the IC card (S301).

[0210] The ticket gate control device 20 performs an IC card ticket gate determination process based on the information on the IC card (S302). For example, in the IC card ticket gate determination process, the ticket gate control device 20 transmits a ticket gate determination request including the IC card information to the ticket gate determination server 60, and receives a ticket gate determination response from the ticket gate determination server 60. The ticket gate determination response includes information indicating whether the ticket included in the IC card is a ticket that is permitted to pass through the ticket gate.

[0211] The face authentication control device 10 takes a picture of the user (S107).

[0212] The face authentication control device 10 performs a bone structure detection process on the captured image (S108).

[0213] The face authentication control device 10 performs a face frame detection process on the captured image (S109).

[0214] The face authentication control device 10 performs a holding judgment (S110). At the stage of S110, as shown in S2, the user holds the IC card over the face. Therefore, at the stage of S110, it is judged that the IC card has been held over the face.

[0215] The skeleton information obtained in S107, the face frame information obtained in S108, and the determination result of "holding" in the holding determination in S110 are output to the tracking management unit 105 (S111).

[0216] The face authentication control device 10 (for example, the tracking management unit 105) performs the person tracking process (S204) in the same manner as in S201. The face authentication control device 10 records the skeleton information and face frame information acquired in S111 in the person tracking list identified by the person tracking process.

[0217] The face authentication control device 10 transmits a face matching request to the face authentication server 40 to request face matching processing of the face image included in the face frame information (S205).

[0218] The face authentication server 40 receives the face matching request and performs face matching processing in the same manner as in S401 (S403).

[0219] As in S402, the face authentication server 40 transmits a face matching response obtained as a result of the face matching process to the face authentication control device 10 (S404).

[0220] The tracking management unit 105 of the face authentication control device 10 records the received face matching result in the person tracking list (S206).

[0221] As a result of the IC card ticket examination determination process, the ticket gate control device 20 determines that the ticket included in the IC card is a ticket that is permitted to pass through the ticket gate. The ticket gate control device 20 then transmits an identity verification request to the face authentication control device 10 to request confirmation as to whether the user who has swiped the IC card is the registered user associated with the IC card (S303).

[0222] The face authentication control device 10 receives the request for personal identification and searches the person tracking list (S207). In the example shown in Fig. 16A and Fig. 16B, face matching is performed each time an image is captured or a face frame is detected. Therefore, when the face authentication control device 10 searches the person tracking list for a person for whom the information "held" is recorded and identifies a face image corresponding to the information "held", the face matching result of the identified face image is included in the person tracking list. In this case, the face matching ID corresponding to the information "held" is included in the person tracking list.

[0223] The face authentication control device 10 makes a holding confirmation response (S208). In the holding confirmation response, it is determined that the person corresponding to the face matching ID identified from the person tracking list is the person who held the IC card over the device.

[0224] The face authentication control device 10 transmits the confirmation result (S209). The confirmation result is an example of a response to the identity verification request. The confirmation result includes the face matching ID determined to correspond to the person who held up the IC card.

[0225] The ticket gate control device 20 receives the confirmation result and confirms passage through the ticket gate (S304). The ticket gate control device 20 determines whether the face matching ID included in the confirmation result matches the ID of the person associated with the IC card read from the IC card. If the IDs match, it is determined that the person who swiped the IC card matches the person associated with the IC card, and the person who swiped the IC card is permitted to pass through the ticket gate.

[0226] The ticket gate control device 20 writes information to the IC card indicating that the pedestrian will pass through the ticket gate (S305). Note that if a pedestrian intending to pass through the ticket gate holds up a terminal such as a smartphone or a smartwatch instead of holding up an IC card, the information indicating that the pedestrian will pass through the ticket gate may be recorded in an external server (for example, the management server 50 and / or the ticket gate determination server 60) instead of being written to the IC card.

[0227] The ticket gate control device 20 displays information indicating that the identity verification has been successful, and performs passage control to permit passage through the ticket gate (S306).

[0228] The user (passenger) checks the display and passes through the ticket gate (S3).

[0229] According to the processing illustrated in Figures 16A and 16B, face matching is performed in advance each time an image is captured or a face frame is detected, so the face authentication control device 10 does not need to request face matching after receiving a personal identification request, and can quickly send the confirmation results, thereby improving the overall processing speed.

[0230] 16A and 16B show cases where the person holding the IC card matches the person associated with the IC card, but there is a possibility that the person holding the IC card does not match the person associated with the IC card. In cases where the person holding the IC card does not match the person associated with the IC card, the face matching ID included in the confirmation result does not match the ID read from the IC card in S208 of FIG. 16B. If the IDs do not match, it is determined that the person holding the IC card does not match the person associated with the IC card, and the person holding the IC card is not permitted to pass through the ticket gate. In this case, the ticket gate control device 20 displays information indicating that identity verification has failed and performs control to prevent the person from passing through the ticket gate.

[0231] As described above, in this embodiment, when the face authentication control device 10 (an example of a control device) receives an identity verification request from the ticket gate control device 20 (an example of an external device), it transmits to the ticket gate control device 20 the result of face matching based on the facial image of the person determined to have swiped an IC card. The face authentication server 40 performs face matching and transmits the face matching result to the face authentication control device 10. Here, the identity verification request is an example of a request signal requesting information about a person who is about to pass through the ticket gate, and the action of swiping an IC card is an example of a specific action for passing through the ticket gate.

[0232] 11A and other figures, the face authentication control device 10 transmits a face matching request to the face authentication server 40, and records the face matching result received from the face authentication server 40 in the person tracking list in association with the determination result of whether or not the IC card has been held over the person. Then, when the face authentication control device 10 receives an identity verification request, it identifies the person who has been determined to have held over the IC card, and transmits the face matching result of the identified person to the ticket gate control device 20.

[0233] <Embodiment Variation 1> In the above-described embodiment, an example has been described in which the face authentication control device 10 transmits a face matching request each time an image is captured or a face frame is detected. In variation 1, an example will be described in which the face authentication control device 10 transmits a face matching request after receiving an identity verification request.

[0234] <Second example of a sequence showing the flow of operations> 17A and 17B are sequence diagrams showing an example of the processing flow of each device in Variation 1 of this embodiment. FIGS. 17A and 17B show the processing of the ticket gate control device 20, the face authentication control device 10, and the face authentication server 40, respectively. The processing of the face authentication control device 10 is divided into processing related to photography and detection and processing related to person tracking. B1 to B5 in FIG. 17A are connected to B1 to B5 in FIG. 17B, respectively. In FIGS. 17A and 17B, processing similar to that in FIGS. 16A and 16B is denoted by the same reference numerals, and a description thereof may be omitted. Similar to FIGS. 16A and 16B, FIGS. 17A and 17B correspond to an example in which the second holding-over determination determines that holding-over is present.

[0235] The examples shown in Figures 17A and 17B differ from the examples shown in Figures 16A and 16B in the timing of the face matching process. Specifically, in the examples shown in Figures 16A and 16B, a face matching request is sent every time an image is captured or a face frame is detected, but in the examples shown in Figures 17A and 17B, the face authentication control device 10 sends a face matching request after receiving an identity verification request. This difference will be explained below.

[0236] In the second example, the face authentication control device 10 does not transmit a face matching request after performing the person tracking process (after S201 and S204). Therefore, S202, S401, S402, and S203 shown in Fig. 16A are omitted in Fig. 17A, and S205, S403, S404, and S206 shown in Fig. 16B are omitted in Fig. 17B. Instead of transmitting a face matching request, the face authentication control device 10 records a face image in the thumbnail buffer.

[0237] The face authentication control device 10 receives the request for personal identification and searches the person tracking list (S501). For example, the face authentication control device 10 searches the person tracking list for a person in which the information "held up" is recorded, and identifies a face image corresponding to the information "held up".

[0238] The face authentication control device 10 transmits a face matching request including information for matching a person whose face image corresponds to the information "with head covering" to the face authentication server 40 (S502).

[0239] The face authentication server 40 receives the face matching request and performs face matching processing (S503).

[0240] The face authentication server 40 transmits a face matching response obtained as a result of the face matching process to the face authentication control device 10 (S504). The face matching response is an example of a response to a face matching request.

[0241] The face authentication control device 10 sends a hold-over confirmation response (S505). In the hold-over confirmation response, it is determined that the person corresponding to the face matching ID included in the face matching response is the person who has held over the IC card.

[0242] In the above-described variation 1, a face matching request is not made each time a photograph is taken or a face frame is detected, so the face matching result may not be recorded in the person tracking list. An example of the person tracking list in variation 1 will be described below.

[0243] FIG. 18 is a diagram showing an example of a person tracking list in variation 1. FIG. 18 shows the same person tracking list as FIG. 4, but in variation 1, a face matching request is not made each time a photograph is taken or a face frame is detected. In other words, face matching results corresponding to each photographed frame number are not recorded. Therefore, as shown in FIG. 18, the face matching results, that is, the face matching score and face matching ID, are not recorded in the person tracking list.

[0244] 17A, 17B, 18, etc., the face authentication control device 10 associates the determination result of whether or not the IC card has been held over the captured image with the face image acquired from the captured image, and records the association in the person tracking list. Then, when the face authentication control device 10 receives an identity verification request, it identifies the person who has been determined to have held over the IC card, transmits a face matching request for the identified person to the face authentication server 40, and transmits the face matching result of the identified person received from the face authentication server 40 to the ticket gate control device 20.

[0245] According to the variation 1 described above, the face authentication control device 20 requests face matching when it receives an identity verification request, so that the number of face matching processes can be reduced and the processing speed can be improved.

[0246] <Variation 2 of the embodiment> In the above-described variation 1, an example was shown in which the face authentication control device 10 records a face image in a person tracking list, and when an identity verification request is received, selects a face image to be used for face matching from the recorded face images. In variation 2, an example will be described in which the face authentication control device transmits a face image to a face authentication server, and the face authentication server records the face image.

[0247] FIG. 19 is a diagram showing an example of the configuration of variation 2 of this embodiment. In FIG. 19, the same components as those in FIG. 2 are denoted by the same numbers, and descriptions thereof may be omitted. Comparing FIG. 19 with FIG. 2, the face authentication server 40 in FIG. 2 has been replaced with a face authentication server 40a in FIG. 19. The face authentication server 40a in FIG. 19 differs from the face authentication server 40 in FIG. 2 in that it includes a thumbnail management unit 403.

[0248] The person tracking unit 105 of the face authentication control device 10 transmits a face image to the face authentication server 40a every time an image is captured or a face frame is detected. A thumbnail serial number for identifying the face image is assigned to the transmitted face image. The thumbnail management unit 403 records (buffers) the face image and its thumbnail serial number transmitted from the face authentication control device 10 in a thumbnail buffer.

[0249] When requesting face matching from the face authentication server 40a, the person tracking unit 105 of the face authentication control device 10 transmits a face matching request including a thumbnail serial number to be matched. The thumbnail management unit 403 outputs a thumbnail of a face image corresponding to the thumbnail serial number included in the face matching request to the face matching unit 402.

[0250] <Example of a sequence showing the operation flow of Variation 2> 20A and 20B are sequence diagrams showing an example of the processing flow of each device in Variation 2 of this embodiment. FIGS. 20A and 20B show the processing of the ticket gate control device 20, the face authentication control device 10, and the face authentication server 40, respectively. The processing of the face authentication control device 10 is divided into processing related to photography and detection and processing related to person tracking. C1 to C5 in FIG. 20A are connected to C1 to C5 in FIG. 20B, respectively. In FIGS. 20A and 20B, processing similar to that in FIGS. 16A and 16B and 17A and 17B is given the same reference numerals, and a description thereof may be omitted. Similar to FIGS. 16A and 16B, FIGS. 20A and 20B correspond to an example in which the second holding-over determination determines that holding-over is present.

[0251] The example shown in Figures 20A and 20B differs from the example shown in Figures 17A and 17B in that face images are managed in the face authentication server 40. This difference will be described below.

[0252] After the face authentication control device 10 performs face frame detection processing on the image captured in S102 and acquires face frame information (after S104), the face authentication control device 10 transmits the face image (e.g., thumbnail) and the thumbnail serial number to the face authentication server 40a (S601). The thumbnail serial number is identification information that identifies the face image.

[0253] The face authentication server 40a records the received face image in the thumbnail buffer (S602).

[0254] After the face authentication control device 10 performs a face frame detection process on the image captured in S107 and acquires face frame information (after S109), the face authentication control device 10 transmits the face image and thumbnail serial number to the face authentication server 40a (S601).

[0255] The face authentication server 40a records the received face image in the thumbnail buffer (S604).

[0256] After receiving the personal identification request (after S303), the face authentication control device 10 searches the person tracking list (S501). For example, the face authentication control device 10 searches the person tracking list for a person in which the information "held up" is recorded, and identifies a face image corresponding to the information "held up".

[0257] The face authentication control device 10 transmits a face matching request to the face authentication server 40a to match the person corresponding to the information "held up" (S605). The face matching request in the example shown in Fig. 17A and Fig. 17B includes the face image corresponding to the information "held up" identified in S204, while the face matching request in the example shown in Fig. 20A and Fig. 20B includes the thumbnail serial number assigned to the face image corresponding to the information "held up" identified in S204.

[0258] The face authentication server 40a receives the face matching request, reads out the face image corresponding to the thumbnail serial number included in the face matching request, and performs face matching processing on the read out face image (S606).

[0259] The face authentication server 40 transmits a face matching response obtained as a result of the face matching process to the face authentication control device 10 (S504).

[0260] In the above-described variation 2, as illustrated in FIGS. 20A and 20B, the face authentication control device 10 associates the determination result, obtained based on the captured image, of whether or not the IC card was held over the person with the thumbnail number of the face image acquired from the captured image, and records the associated data in the person tracking list. The face authentication control device 10 transmits the face image acquired from the captured image and the thumbnail number of the face image to the face authentication server 40a. The face authentication server 40a records the associated face image and thumbnail number of the face image. Then, upon receiving an identity verification request, the face authentication control device 10 identifies the person determined to have held over the IC card, transmits a face matching request to the face authentication server 40a including a thumbnail number indicating the face image of the identified person, and transmits the face matching result of the identified person received from the face authentication server 40a to the ticket gate control device 20. In this case, the face authentication server 40a performs face matching based on the face image corresponding to the thumbnail number included in the face matching request, and transmits the result of the face matching to the face authentication control device 10.

[0261] According to the variation 2 described above, the face authentication control device 10 requests face matching when it receives an identity verification request, so the number of face matching processes can be reduced and the processing speed can be improved. Also, because the face image is transmitted to the face authentication server 40a in advance before the face matching request is made, the traffic related to the transmission of the face matching request can be reduced and the processing speed can be improved.

[0262] <Embodiment Variation 3> In the above-described variation 1, an example was shown in which the face authentication control device 10 records face images in a person tracking list, and when an identity verification request is received, selects a face image to be used for face matching from the recorded face images. In variation 3, the face authentication control device 10 checks the quality of the face image and records the result of the quality check together with the face image. Then, an example will be described in which, when an identity verification request is received, the face authentication control device 10 selects a face image to be used for face matching from the recorded face images based on the result of the quality check.

[0263] The configuration of Variation 3 will be described with reference to FIG. 2. In Variation 3, the face detection unit 102 checks the quality of the face image when it detects a face frame in a captured image and extracts a face image. The higher the quality of the face image, the more accurate the face matching. For example, the quality of the face image is determined based on the size of the face image, the resolution of the face image, whether the face is facing forward, whether features such as the eyes, nose, and mouth are included in the face image, and so on. The face detection unit 102 quantifies the quality of the face image. The result of the face image quality check is expressed numerically. For example, the result of the face image quality check is expressed as a number between 0 and 100, with a higher number indicating a higher quality face image.

[0264] The face detection unit 102 outputs face frame information including the result of the face image quality check to the tracking management unit 105. The tracking management unit 105 records the result of the face image quality check in a person tracking list.

[0265] When requesting face matching from the face authentication server 40, the tracking management unit 105 determines the face image with the best quality (i.e., the best shot) based on the face image included in the person tracking list of the person to be matched and the face image quality check result. The tracking management unit 105 transmits a face matching request including the best shot to the face authentication server 40.

[0266] <Example of people tracking list for variation 3> Fig. 21 is a diagram showing an example of a person tracking list of Variation 3. Fig. 21 shows a list for one particular passerby. Fig. 21 is the same person tracking list as Fig. 5, but as can be seen from a comparison between Fig. 5 and Fig. 21, the person tracking list in Fig. 21 is the person tracking list in Fig. 5 with an additional item, "Facial Image Quality Check Result."

[0267] "Facial image quality check result" indicates the result of a quality check on the facial image of the photographed image of the corresponding photographed frame number. The facial image quality check result is an example of quality information indicating the quality of the facial image.

[0268] <Example of a sequence showing the operation flow of Variation 3> 22A and 22B are sequence diagrams showing an example of the processing flow of each device in Variation 3 of this embodiment. FIGS. 22A and 22B show the processing of the ticket gate control device 20, the face authentication control device 10, and the face authentication server 40. The processing of the face authentication control device 10 is divided into processing related to photography and detection and processing related to person tracking. D1 to D5 in FIG. 22A are connected to D1 to D5 in FIG. 22B, respectively. In FIGS. 22A and 22B, processing similar to that in FIGS. 16A and 16B and 17A and 17B is given the same reference numerals, and a description thereof may be omitted. Similar to FIGS. 16A and 16B, etc., FIGS. 22A and 22B correspond to an example in which the second holding-over determination determines that holding-over is present.

[0269] The example shown in Figures 22A and 22B differs from the example shown in Figures 17A and 17B in that a quality check of the face image is performed in the face authentication control device 10. This difference will be described below.

[0270] After performing a holding determination on the image captured in S102 (after S105), the face authentication control device 10 checks the quality of the face image (S701). The face authentication control device 10 outputs the result of the quality check to the tracking management unit 105 (S702).

[0271] After performing the holding determination on the image captured in S107 (after S110), the face authentication control device 10 checks the quality of the face image (S703). The face authentication control device 10 outputs the result of the quality check to the tracking management unit 105 (S704).

[0272] After receiving the personal identification request (after S303), the face authentication control device 10 searches the person tracking list (S501). For example, the face authentication control device 10 searches the person tracking list for a person in which the information "held up" is recorded, and identifies the person corresponding to the information "held up".

[0273] The face authentication control device 10 determines the face image with the best quality (that is, the best shot) based on the face image included in the person tracking list of the person identified in S501 and the face image quality check result (S705).

[0274] The face recognition control device 10 transmits a face matching request including the determined best shot to the face recognition server 40 in order to match who the person is that corresponds to the information "held up" (S706). The face matching request in the example shown in Fig. 17A and Fig. 17B includes the face image that corresponds to the information "held up" identified in S501, while the face matching request in the example shown in Fig. 22A and Fig. 22B includes the face image of the best shot selected in S705.

[0275] 22A, 22B, etc., the face authentication control device 10 associates the determination result of whether or not an IC card waving action has been performed based on the captured image with the face image acquired from the captured image and quality information indicating the quality of the face image, and records these in the person tracking list. Then, when the face authentication control device 10 receives an identity verification request, it identifies the person determined to have performed the IC card waving action, transmits a face matching request to the face authentication server 40 requesting face matching based on the face image associated with the quality information indicating the best quality among the face images of the identified person, and transmits the face matching result of the identified person received from the face authentication server 40 to the ticket gate control device 20.

[0276] According to the variation 3 described above, the face authentication control device 10 checks the quality of the face image and uses the face image with the best quality for face matching, thereby suppressing the occurrence of face matching errors, thereby reducing the number of face matching processes and improving processing speed.

[0277] <Embodiment Variation 4> In the above-mentioned Variation 2, an example was described in which the face authentication control device 10 transmits a face image to the face authentication server 40a, and the face authentication server 40a records the face image, and in Variation 3, an example was described in which the face authentication control device 10 checks the quality of the face image and records the result of the quality check together with the face image. In Variation 4, an example will be described in which, like Variation 2, when the face authentication server 40a records a face image, the quality of the face image is checked and the result of the quality check is recorded together with the face image, like Variation 3.

[0278] The configuration according to Variation 4 will be described with reference to Fig. 19. In Variation 4, the thumbnail management unit 403 of the face authentication server 40a checks the quality of the face image transmitted from the face authentication control device 10. The thumbnail management unit 403 records (buffers) the result of the face image quality check in the thumbnail buffer together with the face image transmitted from the face authentication control device 10 and its thumbnail serial number.

[0279] When the person tracking unit 105 of the face authentication control device 10 requests face matching from the face authentication server 40a, it sends a face matching request including the passerby number to be matched. The thumbnail management unit 403 determines the face image with the best quality (i.e., the best shot) from the person tracking list corresponding to the passerby number included in the face matching request based on the face image quality check result. The thumbnail management unit 403 outputs the best shot to the face matching unit 402.

[0280] <Example of a sequence showing the operation flow of Variation 4> 23A and 23B are sequence diagrams showing an example of the processing flow of each device in Variation 4 of this embodiment. FIGS. 23A and 23B show the processing of the ticket gate control device 20, the face authentication control device 10, and the face authentication server 40. The processing of the face authentication control device 10 is divided into processing related to photography and detection and processing related to person tracking. E1 to E5 in FIG. 23A are connected to E1 to E5 in FIG. 23B, respectively. In FIGS. 23A and 23B, processing similar to that in FIGS. 16A and 16B, 17A and 17B, 20A and 20B, and 22A and 22B is denoted by the same reference numerals, and a description thereof may be omitted. Similar to FIGS. 16A and 16B, FIGS. 23A and 23B correspond to an example in which the second holding-over determination determines that the device is holding over the card.

[0281] The example shown in Figures 23A and 23B differs from the example shown in Figures 20A and 20B in that the quality of the face image is checked by the face authentication server 40. This difference will be explained below.

[0282] After the face authentication control device 10 performs face frame detection processing on the image captured in S102 and acquires face frame information (after S104), the face authentication control device 10 transmits the face image, the passerby number, and the thumbnail serial number to the face authentication server 40a (S801). The thumbnail serial number is identification information that identifies the face image.

[0283] The face authentication server 40a checks the quality of the received face image (S802).

[0284] The face authentication server 40a records the face image (for example, a thumbnail) in a thumbnail buffer together with the face image quality check result (S803).

[0285] After the face authentication control device 10 performs a face frame detection process on the image captured in S107 and acquires face frame information (after S109), the face authentication control device 10 transmits the face image, the passerby number, and the thumbnail serial number to the face authentication server 40a (S804).

[0286] The face authentication server 40a checks the quality of the received face image (S805).

[0287] The face authentication server 40a records the face image (for example, a thumbnail) in a thumbnail buffer together with the face image quality check result (S806).

[0288] After receiving the personal identification request (after S303), the face authentication control device 10 searches the person tracking list (S501). For example, the face authentication control device 10 searches the person tracking list for a person in which the information "held up" is recorded, and identifies the passerby number of the person corresponding to the information "held up".

[0289] The face authentication control device 10 transmits a face matching request to the face authentication server 40 in order to match the person corresponding to the information "held up" (S807). The face matching request in the example shown in Fig. 17A and Fig. 17B includes the face image corresponding to the information "held up" identified in S204, while the face matching request in the third example shown in Fig. 23A and Fig. 23B includes the passerby number in the person tracking list corresponding to the information "held up" identified in S204.

[0290] The face authentication server 40a receives the face matching request and selects a face image to be used for face matching based on the face images in the person tracking list corresponding to the passerby number included in the face matching request and the face image quality check result (S808). For example, the face image corresponding to the best quality may be used for face matching, or the top N (N is an integer equal to or greater than 1) face images with the best quality may be used for face matching.

[0291] The face authentication server 40a performs face matching processing on the face image selected in S808 (S809).

[0292] In the above-described variation 4, as illustrated in FIGS. 23A and 23B, the face authentication control device 10 associates the determination result of whether or not an IC card has been swept based on the captured image with the passer number identifying the person in the facial image acquired from the captured image, and records the association in the person tracking list. The face authentication control device 10 transmits the facial image and the passer number identifying the person in the facial image to the face authentication server 40a. The face authentication server 40a associates and records the facial image, the passer number of the person in the facial image, and quality information indicating the quality of the facial image. Then, when the face authentication control device 10 receives an identity verification request, it identifies the person determined to have swept an IC card, transmits a face matching request including the passer number identifying the identified person to the face authentication server 40a, and transmits the face matching result of the identified person received from the face authentication server 40a to the ticket gate control device 20. In this case, the face authentication server 40a performs face matching based on the face image associated with the quality information indicating the best quality among the face images of the person corresponding to the passerby number included in the face matching request, and transmits the results of the face matching to the face authentication control device 10.

[0293] According to the above-described variation 4, the face authentication control device 10 requests face matching when it receives an identity verification request, thereby reducing the number of face matching processes. Also, because a face image is transmitted to the face authentication server 40 in advance before a face matching request is issued, traffic related to the transmission of the face matching request can be reduced, improving processing speed. Furthermore, because the face authentication server 40a checks the quality of the face image and only high-quality face images are used for face matching, it is possible to suppress the occurrence of face matching errors, reduce the number of face matching processes, and improve processing speed.

[0294] <Embodiment Variation 5> Variation 5 describes an example in which multiple face authentication servers are provided. As an example, as in Variation 3 described above, when the face authentication control device 10 receives an identity verification request, it selects a face image to be used for face matching from the recorded face images based on the result of the quality check, and transmits a face matching request including the selected face image to multiple face authentication servers.

[0295] Fig. 24 is a diagram showing an example of a ticket gate system according to Variation 5 of the present embodiment. In Fig. 24, the same components as those in Fig. 1 are given the same numbers, and descriptions thereof may be omitted. While the ticket gate system shown in Fig. 1 includes one face authentication server 40, the ticket gate system shown in Fig. 24 includes multiple face authentication servers 40.

[0296] When the face authentication control device 10 receives an identity verification request from the ticket gate control device 20, it identifies the person whose identity verification request is made in the captured image and transmits a face matching request requesting face matching of the identified person to each of the multiple face authentication servers 40. The face matching request includes information indicating the identified person.

[0297] The face authentication server 40 receives a face matching request and performs face matching. For example, multiple face authentication servers 40 each receive multiple face matching requests at various times and perform face matching, and therefore have different processing loads. The face authentication control device 10 can receive face matching results from a face authentication server 40 with a relatively low load by sending a face matching request to each of the multiple face authentication servers 40. This allows the face authentication control device 10 to receive face matching results more quickly, further improving processing speed.

[0298] The present disclosure can be realized in software, hardware, or software in conjunction with hardware.

[0299] Each functional block used in the description of the above embodiments may be partially or entirely realized as an LSI, which is an integrated circuit, and each process described in the above embodiments may be partially or entirely controlled by a single LSI or a combination of LSIs. The LSI may be composed of individual chips, or may be composed of a single chip that includes some or all of the functional blocks. The LSI may have data input and output. Depending on the degree of integration, the LSI may be called an IC, system LSI, super LSI, or ultra LSI.

[0300] The integrated circuit method is not limited to LSI, but may be realized by a dedicated circuit, a general-purpose processor, or a dedicated processor. Also, a field programmable gate array (FPGA) that can be programmed after LSI manufacturing, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells within the LSI, may be used. The present disclosure may be realized as digital processing or analog processing.

[0301] Furthermore, if an integrated circuit technology that can replace LSI emerges due to advances in semiconductor technology or other derivative technologies, it is natural that such technology may be used to integrate functional blocks. The application of biotechnology, etc. is also a possibility.

[0302] The present disclosure may be implemented in any type of apparatus, device, or system (collectively referred to as a communications apparatus) that has a communications function. The communications apparatus may include a wireless transceiver and processing / control circuitry. The wireless transceiver may include a receiver and a transmitter, or both functions. The wireless transceiver (transmitter and receiver) may include a radio frequency (RF) module and one or more antennas. The RF module may include an amplifier, an RF modulator / demodulator, or the like. Non-limiting examples of communication devices include telephones (e.g., cell phones, smartphones), tablets, personal computers (PCs) (e.g., laptops, desktops, notebooks), cameras (e.g., digital still / video cameras), digital players (e.g., digital audio / video players), wearable devices (e.g., wearable cameras, smartwatches, tracking devices), game consoles, digital book readers, telehealth / telemedicine devices, communication-enabled vehicles or mobile transportation (e.g., cars, airplanes, ships), and combinations of the above devices.

[0303] Communications equipment is not limited to portable or mobile equipment, but also includes non-portable or fixed equipment, devices, and systems of any kind, such as smart home devices (such as appliances, lighting equipment, smart meters or metering devices, control panels, etc.), vending machines, and any other "things" that may exist on an IoT (Internet of Things) network.

[0304] Furthermore, in recent years, in the field of IoT (Internet of Things) technology, CPS (Cyber ​​Physical Systems) has been attracting attention as a new concept that creates new added value by linking information between physical space and cyberspace. This CPS concept can also be adopted in the above-mentioned embodiments.

[0305] That is, as a basic configuration of a CPS, for example, an edge server located in physical space and a cloud server located in cyberspace can be connected via a network, and processing can be distributed and performed by processors installed on both servers. Here, it is preferable that each piece of processing data generated on the edge server or cloud server is generated on a standardized platform, and the use of such a standardized platform can improve the efficiency of building a system that includes a variety of sensor groups and IoT application software.

[0306] Communications include data communications via cellular systems, wireless LAN systems, communications satellite systems, etc., as well as data communications via combinations of these.

[0307] A communications apparatus also includes devices such as controllers and sensors connected or coupled to a communications device that performs the communications functions described in this disclosure, such as controllers and sensors that generate control and data signals used by the communications device to perform the communications functions of the communications apparatus.

[0308] The communication apparatus also includes infrastructure facilities, such as base stations, access points, and any other apparatus, device, or system that communicates with or controls the various apparatuses listed above, but are not limited to these.

[0309] Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure. Furthermore, the components of the above-described embodiments may be combined in any manner without departing from the spirit of the disclosure.

[0310] Although specific examples of the present disclosure have been described in detail above, these are merely examples and do not limit the scope of the claims. The technology described in the claims includes various modifications and alterations of the specific examples exemplified above. [Industrial Applicability]

[0311] An embodiment of the present disclosure is suitable for a face authentication system. [Explanation of symbols]

[0312] 10. Face recognition control device 20 Ticket gate control device 30 Camera 40, 40a Face recognition server

Claims

1. a control device that, when receiving a request signal from an external device requesting information about a person attempting to pass through the gate, transmits to the external device a result of face matching based on a facial image of the person determined to have performed a specific action to pass through the gate; a face authentication server that performs the face matching and transmits the result of the face matching to the control device; An authentication system comprising:

2. The control device transmitting a face matching request to the face authentication server, the face matching request being based on the face image of the person acquired from a photographed image of the person; receiving a result of the face matching from the face authentication server; a determination result of whether or not the specific action has been performed based on the captured image and a result of the face matching are stored in association with each other; When the request signal is received from the external device, the person who is determined to have performed the specific action is identified, and a result of the face matching based on the face image of the identified person is transmitted to the external device. The authentication system of claim 1 .

3. The control device a determination result of whether or not the specific action has been performed based on a photographed image of the person is stored in association with the face image of the person acquired from the photographed image; When the request signal is received from the external device, the person who has been determined to have performed the specific action is identified, a face matching request is transmitted to the face authentication server to request face matching based on the face image of the identified person, and a result of the face matching is received from the face authentication server. transmitting a result of the face matching based on the face image of the identified person to the external device; The authentication system of claim 1 .

4. The control device a determination result of whether or not the specific action has been performed based on a photographed image of the person is stored in association with an identification number of the face image of the person acquired from the photographed image; transmitting the facial image and the identification number of the person to the face authentication server; When the request signal is received from the external device, the person who has been determined to have performed the specific action is identified, a face matching request including an identification number indicating a specific face image among the face images of the identified person is transmitted to the face authentication server, and a result of the face matching is received from the face authentication server. transmitting a result of the face matching based on the face image of the specified person to the external device; The face authentication server The received facial image of the person is stored in association with the identification number, and When the face matching request is received, face matching is performed based on the face image corresponding to the identification number included in the face matching request; transmitting a result of face matching based on the face image corresponding to the identification number to the control device; The authentication system of claim 1 .

5. The control device a determination result of whether or not the specific action has been performed based on a photographed image of the person, the face image of the person acquired from the photographed image, and quality information indicating the quality of the face image are stored in association with each other; When the request signal is received from the external device, the person who has been determined to have performed the specific action is identified, and a face matching request is transmitted to the face authentication server, requesting face matching based on a face image associated with quality information indicating the best quality among the face images of the identified person, and the result of the face matching is received from the face authentication server. transmitting a result of the face matching based on the face image of the identified person to the external device; The authentication system of claim 1 .

6. The control device a determination result of whether or not the specific action has been performed based on a photographed image of the person is stored in association with an identification number of the person obtained from the photographed image; transmitting the facial image and the identification number of the person to the face authentication server; When the request signal is received from the external device, the person who is determined to have performed the specific action is identified, a face matching request including an identification number of the identified person is transmitted to the face authentication server, and a result of the face matching is received from the face authentication server. transmitting the face matching result of the identified person to the external device; The face authentication server The received facial image of the person, the identification number, and quality information indicating the quality of the facial image are stored in association with each other; When the face matching request is received, the face matching is performed based on a face image associated with quality information indicating the best quality among a plurality of face images of the person corresponding to the identification number included in the face matching request; transmitting a result of the face matching based on the face image associated with the quality information indicating the best quality to the control device; The authentication system of claim 1 .

7. The control device determining whether the specific movement has been performed based on the position of the arm indicated by skeletal information of the person acquired from a photographed image of the person; The authentication system of claim 1 .

8. The control device determining that the specific movement has been performed when the position of the arm indicated by the skeletal information of the person acquired from the photographed image of the person is included in a specific range of the photographed image; The authentication system of claim 7.

9. The control device When it is detected in the photographed image that the position of an arm indicated by skeletal information of the person acquired from the photographed image of the person intersects with at least one of a marker provided on the gate or a marker set on the photographed image, it is determined that the person has performed the specific action. The authentication system of claim 7.

10. The control device Detecting a face frame surrounding a face included in the photographed image; determining the face frame of the person determined to have performed the specific action from the position of the face indicated by the skeletal information of the person and the position of the face frame; The authentication system of claim 7.

11. The control device determining whether the specific movement has been performed based on arm positions and foot positions indicated by skeletal information of the person acquired from a photographed image of the person; The authentication system of claim 1 .

12. The control device receiving a request signal from an external device requesting information about a person attempting to pass through the gate; When the request signal is received, a result of face matching based on a face image of the person determined to have performed a specific action to pass through the gate is transmitted to the external device. Authentication method.

13. The control device receiving a request signal from an external device requesting information about a person attempting to pass through the gate; When the request signal is received, a result of face matching based on a face image of the person determined to have performed a specific action to pass through the gate is transmitted to the external device. A program that executes a process.

Images