Gate device

Abstract

To provide a gate device that can immediately, accurately, and highly reliably determine whether a target person passes through a target gate.SOLUTION: A gate device 100 includes: a data acquisition unit 41 that acquires image data including a passerby HU who is a target person; a vector extraction unit 62 that extracts a plurality of types of vectors related to traveling from the acquired image data; a vector generation unit 71 that combines the extracted plurality of types of vectors to generate an estimated traveling vector (traveling direction vector) PV of the passerby HU; and a determination unit 80 that determines whether the passerby HU passes through a target gate based on the generated estimated traveling vector PV.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a gate device that authenticates a subject and determines whether or not the subject can pass through. [Background technology]

[0002] For example, a known entrance / exit authentication system determines the direction of travel by ensuring that the gaze in a captured image of a face is directed toward a specific space (see Patent Document 1). Also, a known information processing device that estimates the position of an object passing through a specific area is provided with a function for estimating a person's position based on changes in the position of the face in the horizontal direction, separate from face image detection, and determines whether the person can pass through a gate (see Patent Document 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7075702 [Patent Document 2] JP 2022-164659 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the above Patent Document 1, if only information on the line of sight is relied upon, for example, the line of sight and the direction of travel may differ, and the determination may be unreliable. Also, in the above Patent Document 2, there is a possibility that the imaging area in which face authentication can be performed may be limited.

[0005] The present invention has been made in consideration of the above-mentioned points, and aims to provide a gate device that can quickly, accurately and reliably determine whether a target person will pass through a target gate. [Means for solving the problem]

[0006] To achieve the above-mentioned objective, the gate device comprises a data acquisition unit that acquires image data including a subject, a vector extraction unit that extracts multiple types of vectors related to progress from the acquired image data, a vector generation unit that combines the multiple types of extracted vectors to generate an estimated progress vector of the subject, and a determination unit that determines whether or not the subject will pass through the target gate based on the generated estimated progress vector.

[0007] In the above-mentioned gate device, an estimated progress vector of the subject is generated by combining multiple types of vectors related to progress, and based on this, a determination is made as to whether the subject will pass through the target gate, enabling a quick, accurate and reliable determination to be made even in cases where the gates are installed in parallel.

[0008] In a specific aspect of the present invention, the vector extraction unit extracts at least one of a gaze vector and a face vector in addition to the skeleton vector, and can perform a determination based on a combination of the skeleton vector and the other vector.

[0009] In another aspect of the present invention, the vector generating unit weights the skeleton vector more heavily than other vectors when generating the estimated progress vector. In this case, it is possible to make a decision that places emphasis on the skeleton vector.

[0010] In yet another aspect of the present invention, the determination unit sequentially determines the possibility of passing through the target gate based on the vector generated first among the multiple types of vectors, and makes a final determination after the skeleton vector is acquired. In this case, the final determination can be made accurately while making a prediction in advance.

[0011] In yet another aspect of the present invention, a passage permission / denial decision unit is provided that decides whether or not to permit passage through a target gate based on registered face recognition data. In this case, quick and accurate judgment can be made using face recognition.

[0012] In yet another aspect of the present invention, the determination unit determines whether to pass through the target gate or a gate adjacent to the target gate. In this case, it is possible to quickly and accurately determine which of the parallel gates the vehicle will pass through.

[0013] In still another aspect of the present invention, the determination unit makes the determination based on whether or not the estimated moving vector has exceeded the detection boundary a predetermined number of times or more. In this case, the determination can be made uniformly according to a certain criterion. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a perspective view showing a configuration example of a gate device according to an embodiment; [Diagram 2] 1A and 1B are conceptual plan views showing the detection range of a camera provided in a gate device. [Diagram 3] 1A to 1C are conceptual diagrams for explaining vector extraction and synthesis. [Figure 4] 1A and 1B are conceptual plan views for explaining an example of a virtual detection line in a gate device. [Diagram 5] 13(A) to 13(H) are conceptual diagrams for explaining an example of a method for extracting a skeletal vector. [Figure 6] FIG. 2 is a block diagram for explaining a configuration example of a functional aspect of a gate device. [Figure 7] 13(A) to 13(C) are conceptual plan views for explaining an example of a problem that is assumed to occur due to overlapping of detection ranges between adjacent cameras. [Figure 8] 1A to 1D are conceptual plan views showing an example of a problem that is expected to arise due to the use of a camera, and 1E to 1G are conceptual plan views showing another example. [Figure 9] 1 is a flowchart for explaining an example of a series of operations (overall operations) in a gate device. [Figure 10]10 is a flowchart for explaining another example of a series of operations in the gate device. [Figure 11] 10 is a flowchart for explaining an example of determining whether or not there is an intention to pass based on a vector. [Figure 12] FIG. 1 is a conceptual diagram for explaining an overview of a gate device. [Figure 13] 13(A) to 13(D) are diagrams illustrating a modified example of the operation mode of the gate device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] An example of a gate device according to an embodiment will be described below with reference to Fig. 1 etc. Fig. 1 is a perspective view for explaining a gate device 100 according to this embodiment and an application example thereof, and is a diagram showing the external appearance of the gate device 100. In this embodiment, as shown in the figure, a plurality of gate devices 100 are provided in parallel in the horizontal direction, and adjacent gate devices 100 are configured to share, for example, a partition portion PW.

[0016] Each gate device 100 is equipped with a camera 10, an IC card reader 20, and a flap (door) 30, and determines whether or not to allow a pedestrian to pass based on information about the pedestrian obtained via the camera 10 and the IC card reader 20, and opens or closes the flap (door) 30 in accordance with the determination.

[0017] In the gate device 100 of this embodiment, it is possible to determine whether or not to permit passage by face authentication. For this reason, when a pedestrian passes through, the gate device 100 acquires face image data by the camera 10, and compares the acquired face image data with preregistered face image data (face data or face authentication data).

[0018] Furthermore, in this embodiment, the image acquired by the camera 10 is used to determine whether or not the target pedestrian has the intention to pass through the target gate corresponding to the camera 10. In particular, here, not only the gaze vector extracted from the face image included in the target image based on the movement of the eyes and the face vector extracted from the face direction, but also the skeletal vector extracted from the image of the pedestrian about the direction (posture) of the body of the pedestrian is used to generate an estimated movement vector, and whether or not the target gate is to be passed is determined based on the generated estimated movement vector. As a result, even in the case where gates are installed in parallel by a plurality of gate devices 100 as shown in FIG. 1, it is possible to quickly, accurately and highly reliably determine whether or not the target gate corresponding to one of the cameras 10 is intended to be passed through.

[0019] In order to achieve the above-mentioned configuration, as shown in the figure, the camera 10 of the gate device 100 is provided near the center of the partition wall PW in the direction of travel of pedestrians, and captures images of potential pedestrians approaching the gate device 100. As shown in the conceptual plan views of Figures 2(A) and 2(B), within the detection range of the camera 10 provided in the gate device 100, the imaging range IA has a certain degree of spread, and within this, the range particularly close to the camera 10 is the face image acquisition range FA in which face image data (face data) can be acquired. As shown in FIG. 2(A), in a plurality of gate devices 100 (100A, 100B, 100C), passages (passage 1, passage 2, passage 3) are formed, and in the figure, camera 10A provided corresponding to passage 1 is shown, and the same is true in FIG. 2(B), but images (detection) are also captured by corresponding cameras 10B, 10C in the other passages (passages 2, passage 3 in FIG. 2(A)).

[0020] Meanwhile, the gate device 100 is provided with an IC card reader 20, and the gate device 100 can accept authentication by IC card in addition to face authentication. That is, a passerby can obtain permission to pass by holding his / her own IC card (not shown) over the IC card reader 20.

[0021] The flap (door) 30 opens and closes depending on the judgment result based on various information acquired by the camera 10 and the IC card reader 20, i.e., the decision on whether or not to permit passage. As for opening and closing, as illustrated in Fig. 2(A), IR sensors SS1 and SS2 that emit IR light so as to cross the passage are provided before and after the flap (door) 30 in the middle of the passage of the gate device 100 (only the gate device 100A is shown, and the others are not shown), and when a pedestrian passes through the passage, the IR sensors SS1 and SS2 detect the blocking of the IR light, thereby detecting the passage of the pedestrian.

[0022] In addition, as illustrated in FIG. 2(A), the gate device 100 includes a control unit 50 for centrally performing the above-mentioned various judgment processes and operation commands, and a data matching unit 90 for matching preregistered information when performing face authentication or IC card-based judgment. In FIG. 2(A), the control unit 50 is provided for each gate device 100, while the data matching unit 90 is connected to one data matching unit 90 common to multiple gates (gate devices 100A, 100B, ...). However, the present invention is not limited to this. For example, the control unit 50 may be configured such that one control unit 50 centrally controls the operation of multiple gates (gate devices 100A, 100B, ...), and each gate device 100 may have not only the control unit 50 but also the data matching unit 90. In addition, multiple gate devices 100 may be collectively regarded as one gate device.

[0023] 1 shows an example in which IC card readers 20 are provided at both ends of the travel direction of pedestrians to allow entry from both directions, but it is also possible to allow entry from only one direction. The imaging range (angle range) of camera 10, for example, can also be set appropriately according to each mode.

[0024] Hereinafter, the extraction of the above-mentioned multiple types of vectors and their synthesis will be described with reference to an example shown in a conceptual diagram as Fig. 3. Fig. 3(A) is a conceptual diagram showing how each vector is extracted from a person image in an image, Fig. 3(B) is a conceptual plan view showing the state in which each vector is viewed in a plane, and Fig. 3(C) is a plan view showing how each vector shown in Fig. 3(B) is synthesized to generate a traveling direction vector as an estimated direction vector. Note that each vector is a three-dimensional spatial vector, but it is also possible to view it in a plane in the figure and capture it as a two-dimensional vector.

[0025] First, as shown in Fig. 3(A), in the image of a passerby HU showing a subject to be judged (hereinafter, images related to a person will also be simply referred to as passerby HU, etc.), attention is focused on the part corresponding to the human eye, and a gaze vector V1 is extracted from there. There are various possible methods for extracting the gaze vector V1, and for example, a method is considered in which the gaze direction is estimated from the boundary between the white and black parts of the passerby HU's eye and expressed as a vector. Note that various existing methods can be applied to extract the direction corresponding to the gaze vector V1.

[0026] Next, for the face vector V2, it is possible to use a method in which the direction in which the face is facing is estimated from the outline and the arrangement of the eyes, nose, mouth, ears, etc., and this is expressed as a vector. Note that various existing methods can be applied to extract the direction corresponding to the face vector V2.

[0027] Finally, for the skeletal vector V3, a method may be used in which the posture of the passerby HU is estimated based on the entire body of the passerby HU, particularly the direction in which the spine (back muscles) extends and the direction in which the shoulders are aligned, and this is expressed as a vector. An example of this will be described in more detail later.

[0028] The vectors V1, V2, and V3 thus determined are combined to generate an estimated moving vector PV of the target pedestrian HU, as shown in Fig. 3(C). That is, here, the sum of the vectors V1, V2, and V3 is set as the estimated moving vector PV.

[0029] In the gate device 100, a determination is made as to whether or not the pedestrian HU, who is a subject of monitoring, will pass through the passage 1 corresponding to the detection range of the camera 10A constituting the gate device 100A in the illustrated example.

[0030] Here, when generating the estimated progress vector PV as described above, it is possible to give more weight to the skeleton vector V3 among the vectors V1, V2, and V3 than to the other vectors V1 and V2. In this case, it is possible to perform a determination with more emphasis on the skeleton vector V3. If only the information on the line of sight and the direction of the face is relied upon, for example, when the passerby HU is walking with his / her face turned sideways, the line of sight and the direction of the face are different from the direction of travel (the direction in which the passerby HU intends to proceed). Therefore, if the determination is made based only on this, it may be unreliable. To avoid this, in this embodiment, the determination is made based on the skeleton vector V3 added. Therefore, it is considered that by placing emphasis on the skeleton vector V3 in particular, it is possible to more accurately determine whether the passerby HU intends to pass through the target gate.

[0031] In this embodiment, as shown in Fig. 4(A) and Fig. 4(B), a virtual detection line VL is set as a predetermined detection boundary in the gate device 100 (100A), and a judgment is made based on whether or not an estimated moving vector PV (here, regarded as a two-dimensional plane in a planar view) crosses the virtual detection line VL. For example, in the case shown in Fig. 4(A), the virtual detection line VL is set at the entrance of the gate device 100 (100A) that is the target gate. In contrast, as shown in Fig. 4(B), if the tip of the estimated moving vector PV crosses the virtual detection line VL, the pedestrian HU is treated by the gate device 100 (100A) as having an intention to pass through the gate device 100 (100A). From another perspective, the virtual detection line VL indicates whether or not the estimated moving vector PV intersects with a specific area in the gate device 100 (100A) that is the criterion for judging the intention of the pedestrian HU.

[0032] However, with regard to the above, when a pedestrian HU who has no intention of passing through the gate device 100 (100A) momentarily changes his / her posture, the change in posture may accidentally cause the tip of the estimated moving vector PV to cross the virtual detection line VL just once (for just an instant). In response to this, for example, when a state in which the tip of the estimated moving vector PV crosses the virtual detection line VL is detected a predetermined number of times or more (or when it is detected that the crossing has continued for a predetermined period of time or more), it may be possible to handle the pedestrian as having an intention to pass.

[0033] An example of a method for extracting the skeleton vector V3 will be described below with reference to the conceptual diagrams shown in Fig. 5(A) to Fig. 5(H). Fig. 5(A) is a conceptual plan view for explaining the arrangement of the camera 10 that captures the image, and Fig. 5(B) is a diagram showing an example of an image GG captured in the state of Fig. 5(A). Note that the image GG includes a passerby HU who is the target.

[0034] First, as a premise, as shown in Figures 5(A) and 5(B), with regard to the installation state of camera 10, i.e., the direction in which imaging is performed, the lateral direction (horizontal direction) of the image is defined as the x direction, the vertical direction (vertical direction) is defined as the y direction, and the imaging direction perpendicular to the x direction and the y direction (the direction in which the lens axis of camera 10 extends) is defined as the z direction. In this example, as shown in Figure 5(A), camera 10 performs imaging in the -z direction.

[0035] In the above-mentioned imaging situation, the presence of a passerby HU is detected by extracting an image portion GGd including the passerby HU from the captured image GG, as shown in Fig. 5(C). Note that, for example, various existing image processing techniques can be used to extract and detect a person from an image as described above.

[0036] When the image portion GGd including the passerby HU is extracted, the control unit 50 further performs skeleton and posture detection for the image portion GGd. Specifically, as shown in FIG. 5(D), skeleton information HUc consisting of points and lines is extracted from the passerby HU, and further, as shown in FIG. 5(E), a straight line Lb corresponding to the extension direction of the spine (back muscles) of the passerby HU and a straight line Ls corresponding to the alignment direction of both shoulders are extracted from the skeleton information HUc. Then, as shown in FIG. 5(F), the point where the straight lines Lb and Ls intersect is defined as an intersection point CS. Here, as shown in FIG. 5(G), a straight line that passes through the intersection point CS and is perpendicular to both the straight lines Lb and Ls is defined as a straight line PP. Note that FIG. 5(G) has a different viewpoint from the image GG shown in FIG. 5(B) to FIG. 5(F) or the image diagram extracted therefrom, as shown in the xyz coordinate system.

[0037] The direction of extension of the straight line PP in Fig. 5(G) corresponds to the normal direction of the plane spanned by the straight lines Lb and Ls, and the straight line PP extends from the intersection point CS indicating the center of the skeleton of the passerby HU. Here, the direction of the straight line PP is treated as indicating the moving direction of the passerby HU, and a directional vector parallel to the direction of the straight line PP and pointing toward the side approaching the camera 10 is defined as a basic skeleton vector SV, as shown in Fig. 5(H). Here, the basic skeleton vector SV is a vector that extends in a direction parallel to the direction of the straight line PP and has a length of 1, that is, a unit vector.

[0038] Here, in this embodiment, the basic skeleton vector SV is multiplied by a preset weight (scalar quantity) to obtain a skeleton vector V3 (see FIG. 3). In addition, the gaze vector V1 and face vector V2 are also processed to set the length to 1 (normalize) for the extracted gaze direction and face direction to obtain basic vectors (unit vectors), which are then multiplied by a preset weight (scalar quantity) (to be normalized) to define the gaze vector V1 and face vector V2. In other words, the estimated progress vector PV is the sum of the weights (scalar quantities) multiplied by each basic vector. In this case, the preset weights can be appropriately adjusted according to the installation environment of the gate device 100, thereby optimizing the estimated progress vector PV, which is the basis for making a judgment.

[0039] Hereinafter, a configuration example showing the functional aspects of the gate device 100 will be described with reference to the block diagram shown in Fig. 6. Here, the functions of the gate device 100 will be described with a focus on each unit related to the process from image extraction based on the image processing and arithmetic processing described above to vector generation until it is determined whether or not a target person (passerby HU) will pass through the target gate (whether or not the target person has an intention to pass through the gate device 100).

[0040] As shown in FIG. 6 and as described above, the gate device 100 of this embodiment is composed of a camera 10, an IC card reader 20, and a flap (door) 30, but here, the control unit 50 that controls each of the above parts and the data matching unit 90 for matching data will also be described as being part of the gate device 100.

[0041] The control unit 50 is composed of, for example, various circuit devices, and controls the operation of the camera 10, etc., and generally performs various processes related to images, such as the extraction of vectors from the image data, and calculations and judgments related to the extracted vectors. For this purpose, the control unit 50 includes a camera control unit 11, a gate control unit 31, a data acquisition unit 41, a data input / output unit 42, an image processing unit 60, an arithmetic processing unit 70, and a judgment unit 80. In addition, the control unit 50 includes a data storage unit DS composed of, for example, a storage device, in order to store various data.

[0042] Of the above, the camera control unit 11 controls the imaging operation of the camera 10, and the gate control unit 31 controls the opening and closing operation of the flap (door) 30.

[0043] The data acquisition unit 41 receives image data captured by the camera 10 and ID information read by the IC card reader 20, and stores the data in the data storage unit DS to acquire various information (data). The data acquisition unit 41 also outputs the received various data to the image processing unit 60 and the data input / output unit 42. For example, when the data acquisition unit 41 acquires image data including a target person (passerby HU) from the camera 10, it outputs the data to the image processing unit 60. The image processing unit 60 extracts the above-mentioned vectors from the image data acquired by the data acquisition unit 41, and extracts features for face authentication. On the other hand, when the data acquisition unit 41 acquires ID information read by the IC card reader 20, it outputs the ID information to the data input / output unit 42 for matching in the data matching unit 90.

[0044] The data input / output unit 42 functions as an interface unit with the data matching unit 90 that inputs and outputs data for data matching. Specifically, the data input / output unit 42 outputs to the data matching unit 90 the ID information from the IC card reader 20 received via the data acquisition unit 41 and information related to face authentication based on the image data processed by the image processing unit 60 in order to perform authentication judgment. The data input / output unit 42 also receives information on the result of matching from the data matching unit 90.

[0045] The image processing unit 60 includes a human detection unit 61 , a vector extraction unit 62 , and a face image feature amount extraction unit 63 .

[0046] The person detection unit 61 is composed of a person detection engine software HD that performs person detection by extracting a person image included in image data acquired by the camera 10. Note that the person detection engine software HD can apply various programs as long as it is capable of extracting a person, and may use an existing method.

[0047] The vector extraction unit 62 extracts a plurality of types of vectors related to the progress of a person (passerby HU) who has been detected by the person detection unit 61 and has become a target person from the image data acquired by the data acquisition unit 41. For this reason, the vector extraction unit 62 has a gaze direction determination engine software GD, a face direction determination engine software FD, and a posture estimation engine software PE.

[0048] The gaze direction determination engine software GD in the vector extraction unit 62 is configured with a program for estimating the gaze direction of a target person (passerby HU) by analyzing, for example, the eye part from the corresponding image (image part), and expressing it as a vector. For example, it is assumed that the software is configured with a program for identifying the gaze direction using an existing method and generating a fundamental vector (unit vector) for this.

[0049] The face direction determination engine software FD is configured as a program for extracting the part of the image (image portion) corresponding to the target person (passerby HU) that corresponds to the face in particular, analyzing the outline and the arrangement of the eyes, nose, mouth, ears, etc. from the extracted face portion, estimating the direction in which the face is facing, and expressing this as a vector. For example, it is expected to be configured as a program that specifies the direction in which the face is facing using an existing method, and generates a fundamental vector (unit vector) for this.

[0050] The posture estimation engine software PE is configured with a program for extracting, for example, the direction in which the spine (back muscles) extends and the direction of the alignment of both shoulders from the corresponding image (image portion) of a target person (passerby HU), estimating the posture of the passerby HU based on these, and expressing this as a vector. That is, it is assumed to be configured with a program for calculating a basic skeleton vector (basic vector) SV by performing a series of processes for the method described with reference to Fig. 6 etc.

[0051] In the above cases, the vector extraction unit 62 extracts a plurality of types of vectors including a gaze vector and a face vector in addition to a skeleton vector.

[0052] The facial image feature extraction unit 63 is configured with a program (facial image feature extraction program) for extracting a portion of a corresponding image (image portion) of a target person (passerby HU) that corresponds to a face, and extracting a feature for facial authentication from the extracted facial portion. The feature for facial authentication (information related to facial authentication) extracted by the facial image feature extraction unit 63 is output to the data matching unit 90 via the data input / output unit 42, as described above, and the data matching unit 90 performs authentication judgment.

[0053] The calculation processing unit 70 performs various calculation processes, and in this case, in particular, performs calculation processes for generating an estimated moving vector PV that is a reference for determining whether or not a target person (passerby HU) passes through a target gate by combining each vector (multiple types of vectors) extracted by the vector extraction unit 62 of the image processing unit 60. For this reason, the calculation processing unit 70 includes a vector generation unit 71 and a weight setting unit 72, and the vector generation unit 71 has a vector synthesis unit VC that synthesizes each vector extracted by the vector extraction unit 62, and generates an estimated moving vector PV by these. More specifically, in the calculation processing unit 70, the vector generation unit 71 reads out the value of the weight (scalar amount) that is set in advance for each fundamental vector in the weight setting unit 72 for each fundamental vector calculated by the vector extraction unit 62, and in the vector synthesis unit VC, the corresponding fundamental vector is multiplied by the read-out value of the weight (scalar amount) to calculate a gaze vector V1, a face vector V2, and a skeleton vector V3, and further combines these by adding them together to generate an estimated moving vector PV. Here, when the vector generating unit 71 generates the estimated progress vector PV, the skeleton vector V3 is weighted more heavily than the other vectors. This allows the determination to place more emphasis on the skeleton vector V3. As a specific method for this, for example, when setting the weight (scalar amount) value in the weight setting unit 72, if weights m1, m2, and m3 correspond to the gaze vector V1, face vector V2, and skeleton vector V3, respectively, it is considered to set m3>m1 and m3>m2. In this case, the difference is represented as the difference in length between the vectors V1, V2, and V3.

[0054] The determination unit 80 determines whether or not the subject (passerby HU) will pass through the target gate, based on the estimated movement vector PV generated by the calculation processing unit 70. That is, as in the example described with reference to Fig. 4(A) and Fig. 4(B), a determination is made as to whether or not the subject (passerby HU) will pass through the passage 1 corresponding to the camera 10A, based on a predetermined rule, for the virtual detection line VL set as the detection boundary and the generated estimated movement vector PV.

[0055] In this manner, the control unit 50 performs the processes up to the determination as to whether or not the target person (passerby HU) has an intention to pass through the gate apparatus 100 that is the target gate.

[0056] The data matching unit 90 will be described below. As described above, the data matching unit 90 is a device for performing data matching (matching of ID information) to determine whether or not to permit passage through the gate device 100, and here, as described above, the determination of whether or not to permit passage based on face authentication and the determination of whether or not to permit passage based on the ID information of an IC card coexist. In other words, the data matching unit 90 performs matching for face authentication based on face data (face authentication data as ID information) and matching for IC authentication based on IC data (ID information of an IC card).

[0057] To enable the above aspect, the data matching unit 90 has a face authentication server 91 and an IC authentication server 92, as shown in the figure, where the face authentication server 91 is composed of a face matching unit 91c and a face database 91d, and the IC authentication server 92 is composed of an IC matching unit 92c and an IC database 92d.

[0058] Of the above, in the face recognition server 91, the face matching unit 91c compares face data (face recognition data created based on face features) pre-registered in the face database 91d with feature data for face recognition, which is information related to face recognition received via the data input / output unit 42 of the control unit 50, to confirm whether or not a match exists.

[0059] Next, for the IC authentication server 92, the IC matching unit 92c compares the IC data (ID information) pre-registered in the IC database 92d with the ID information read by the IC card reader 20 received via the data input / output unit 42 of the control unit 50 to confirm whether a match exists.

[0060] The data matching unit 90 outputs the matching results from the face matching unit 91c and the IC matching unit 92c to the control unit 50. The control unit 50 determines the control mode of the opening and closing operation of the flap (door) 30 by the gate control unit 31 based on the received matching result. In the above case, for example, when performing face authentication, the control unit 50 functions as a passage permission / rejection determination unit that determines whether or not to permit passage through the target gate based on pre-registered face authentication data. Also, for example, when performing authentication using an IC card, the control unit 50 functions as a passage permission / rejection determination unit that determines whether or not to permit passage through the target gate based on the reading result of the IC card by the IC card reader 20.

[0061] Hereinafter, with reference to FIG. 7 etc., a description will be given of problems that may arise when performing authentication using image capture by the camera 10 and how these problems are dealt with in this embodiment.

[0062] For example, as shown in the conceptual plan views of FIG. 7(A) to FIG. 7(C), when gates are installed in parallel, if the imaging range IA of each camera 10 has a certain degree of spread, an overlapping area IAd may exist between adjacent cameras 10 (for example, between camera 10A and camera 10B, or between camera 10B and camera 10C). For example, if a pedestrian HU passes through the range DD3 shown by the dashed line in FIG. 7(B), it is assumed that the pedestrian HU is heading toward the gate device 100C in which the camera 10C is installed, among the gate devices 100A to 100C. In other words, when a pedestrian HU passing through the range DD3 reaches the imaging range IA of the camera 10C, the pedestrian HU is detected based on the analysis of the image acquired by the camera 10C. However, as shown in FIG. 7(C), when a pedestrian HU reaches the overlapping area IAd of the range DD3, the pedestrian HU is detected not only by the camera 10C but also by the camera 10B, i.e., the gate device 100B. In contrast, in this embodiment, the travel direction of the passerby HU is estimated not only based on the line of sight or the direction of the face of the passerby HU but also by considering the direction of the skeleton, so that even if the camera 10B of the gate device 100B captures the passerby HU passing through the range DD3 as in the above case, it is possible to accurately capture which gate the passerby HU intends to pass through. That is, in the above embodiment, the gate device 100B can determine that the passerby HU has no intention of passing through, while the gate device 100C can determine that the passerby HU has an intention to pass through.

[0063] From another perspective, in order to avoid the above situation, it is possible to narrow the imaging range IA so that an area such as the overlap area IAd is not formed. However, in this case, the start of the passage determination is delayed by the amount of the narrowing of the detection range, and there is a possibility that the capacity of the gate device 100 for the flow of people is limited. In contrast, in this embodiment, it is possible to maintain accurate processing while acknowledging the occurrence of the overlap area IAd.

[0064] As another situation, in the case where passage is permitted or not permitted based on face recognition, if the travel direction of the passerby HU is not estimated as in the present embodiment, for example, as shown in Figures 8(A) to 8(D), a passerby HUα who has pre-registered for face recognition happens to pass near the gate device 100, and a passerby HUβ who has not pre-registered for face recognition, that is, a person who should not be permitted to pass, is able to pass through the gate device 100. To be more specific, as shown in Figure 8(A), a passerby HUβ is heading toward the gate device 100, while a passerby HUα is moving in a direction that crosses the gate device 100 without heading toward it. Then, as shown in Figure 8(B), the passerby HUβ reaches the imaging range IA of the camera 10, and then, as shown in Figure 8(C), the passerby HUα also enters the imaging range IA before the passerby HUβ reaches the entrance of the gate device 100. In this case, the gate device 100 may permit the pedestrian HUα to pass through based on the facial recognition of the pedestrian HUα detected by the camera 10, and as shown in Fig. 8(D), a situation may occur in which the pedestrian HUβ, who should not have been permitted to pass through, passes through the gate device 100. In contrast, in this embodiment, when determining whether or not the pedestrian HUα is permitted to pass through, the traveling direction of the pedestrian HUα is estimated in addition to the facial recognition of the pedestrian HUα, and in the above-mentioned aspect, it is possible to determine that the pedestrian HUα has no intention of passing through, and therefore such a situation can be avoided. That is, in the gate device 100, the determination unit 80 is capable of determining whether the pedestrian HUα will pass through the target gate or the gate adjacent to the target gate.

[0065] In addition, as another situation, for example, as shown in FIG. 8(E) to FIG. 8(G), one gate device 100 (100B) may give permission to both passersby HUγ and HUδ. To explain specifically, it is assumed that passersby HUγ and HUδ, both of which have been preregistered for face recognition, are attempting to pass through the gate device 100B (passage 2), and passersby HUδ are attempting to pass through the gate device 100C (passage 3). In this case, as shown in FIG. 8(E), if passersby HUδ is slightly ahead of passersby HUγ, passersby HUδ is first detected by the gate device 100C. After that, as shown in FIG. 8(F), when passersby HUδ reaches the overlapping area IAd, passersby HUδ is detected not only by the gate device 100C but also by the gate device 100B. Furthermore, under such circumstances, as shown in FIG. 8(G), if passerby HUγ arrives at the imaging range IA of the gate device 100B slightly later than passerby HUδ, the gate device 100B will detect passerby HUγ in addition to passerby HUδ (duplicate detection will occur in one gate device 100B).

[0066] In contrast, in this embodiment, when determining whether or not to allow passage of each of passersby HUγ and HUδ, the traveling directions of passersby HUγ and HUδ are estimated in addition to face authentication of passersby HUγ and HUδ. This allows passerby HUγ to be treated as having an intention to pass through passage 2, and passerby HUδ to be treated as having an intention to pass through passage 3, thereby making it possible to avoid such a situation.

[0067] An example of a series of operations (overall operations) in the gate device 100 will be described below with reference to the flowchart shown in Fig. 9. Here, a series of operations when passing through the gate device 100 using face authentication will be described in particular.

[0068] First, when an image is captured by the camera 10 (step S101), human detection is performed on the captured image (step S102), and it is confirmed whether or not a human has been detected (step S103).

[0069] If no detection is made in step S103 (step S103: No), the operation from step S101 is repeated again, and if a person is detected in step S103 (step S103: Yes), the detected person is set as a target person (passerby HU) and a determination is made as to whether the target person (passerby HU) has an intention to pass (step S104). That is, a determination process is performed as to whether or not the target person (passerby HU) has an intention to pass by each process based on the estimated progress vector PV described above.

[0070] If it is determined in step S105 that the user has no intention of passing (step S105: No), the process up to that point is discarded (step S106), the series of processes is completed, and the process returns to the operation from step S101.

[0071] On the other hand, if the determination in step S105 indicates that there is an intention to pass (step S105: Yes), information on the facial recognition features processed in the control unit 50 at the time of step S104 is transmitted to the facial recognition server 91 of the data matching unit 90 (step S107), and the data matching unit 90 matches the facial data based on the features (facial recognition data as ID information) with the data stored in the face database 91d, i.e., pre-registered facial recognition data (facial data) (step S108).

[0072] If the result of the comparison in step S108 is that there is no matching face data (step S109: No), the process up to that point is discarded (step S110), the series of processes is completed, and the process returns to the operation from step S101.

[0073] On the other hand, if matching face data exists as a result of the comparison in step S108 (step S109: Yes), it is confirmed whether the face data (ID information) is qualified to pass (step S111), and if not (step S111: No), the process up to that point is discarded (step S112), and a series of processes is completed and the operation returns to step S101. On the other hand, if the face data (ID information) is qualified to pass (step S111: Yes), it is confirmed whether a pass credit (permission to pass granted for a certain period of time) has already been granted to the face data (ID information) within a certain period of time (step S113). That is, a confirmation is made to prevent duplicated granting of pass credit. The certain period of time for which a pass permit is granted as a pass credit is set to, for example, about 3 seconds, and the pass credit is discarded after the certain period of time has elapsed (after 3 seconds has elapsed). From another perspective, if 3 seconds or more have elapsed since the previous pass credit was granted, there is no duplication with the previous pass credit, and a pass credit based on a new data comparison is granted.

[0074] In step S113, if it is determined that the passage credit has already been granted (step S113: Yes), the process up to that point is discarded (step S114), the series of processes is completed, and the process returns to the operation from step S101.

[0075] On the other hand, if it is determined in step S113 that the passage credit has not yet been granted (step S113: No), a process of granting the passage credit is performed (step S115). That is, the state of passage permission is maintained for a certain period of time, and maintenance of this state is started. After that, the control unit 50 performs a process of opening and closing the flap 30 in response to detection by the IR sensors SS1 and SS2, which are physical sensors, within the certain period of time (steps S116 to S119). Specifically, when the passing credit starts to be held for a certain period of time in step S115, first, it is confirmed whether or not the IR sensor SS1 on the entrance side detects the blocking of IR light within a certain period of time (for example, within 3 seconds) (step S116), and if the confirmation is made (step S116: Yes), the flap (door) 30 is opened accordingly (step S117), and further, it is confirmed whether or not the IR sensor SS2 on the exit side detects the blocking of IR light within a certain period of time (step S118), and if the confirmation is made (step S118: Yes), the flap (door) 30 is closed accordingly, and the passing is treated as having been completed (step S119). That is, in step S119, a process is performed to treat the granted passing credit as having been consumed.

[0076] When the series of processes is completed in this manner, the process returns to step S101 again to detect the next pedestrian HU.

[0077] If no confirmation is made in step S116 or step S118 (step S116: No or step S118: No), the passage credits granted in step S115 are discarded (steps S120, S121), the series of processes is completed, and the process returns to the operation from step S101. Typical examples of such cases include a case where the target person (passerby HU) does not pass through the gate device 100 for some reason, or a failure of the IR sensors SS1 and SS2.

[0078] Another example of a series of operations in the gate device 100 will be described below with reference to the flowchart shown in Fig. 10. In the flowchart of Fig. 9, a series of operations when passing through the gate device 100 using face authentication has been described in particular, but here, with reference to the flowchart of Fig. 10, an example of operations regarding passing through the gate device 100 using an IC card, that is, passing through the gate based on reading of ID information by the IC card reader 20, will be described.

[0079] First, gate device 100 continuously detects whether an admission pass (IC card) has been presented, i.e., whether the admission pass (IC card) has been held over IC card reader 20 (step S201), and if detection is made (step S201: Yes), it reads the ID information recorded on the IC card (step S202) and confirms the presence or absence of eligibility to pass (step S203). That is, control unit 50 queries IC authentication server 92 of data collating unit 90 about the ID information acquired by IC card reader 20, and collates the ID information.

[0080] If it is determined in step S203 (determining whether or not the person is eligible to pass) that the person is not eligible to pass (step S204: No), the control unit 50 performs processing to prevent the person from passing (determines that the person is not eligible to pass), terminates the series of processing (step S205), and returns to the operation from step S201.

[0081] On the other hand, if it is determined that the user is qualified to pass as a result of the check in step S203 (step S204: Yes), a process of granting a passing credit is performed (step S206). That is, the state of being permitted to pass is maintained for a certain period of time, and this state is started to be maintained. Thereafter, the control unit 50 performs the same process as steps S116 to S121. That is, when the IR sensors SS1 and SS2, which are physical sensors, detect the user within the certain period of time, the control unit 50 performs a process of opening and closing the flap 30 accordingly (steps S207 to S210). On the other hand, when the IR sensor SS1 or the IR sensor SS2 does not detect the blocking of the IR light within the certain period of time, the granted passing credit is discarded (steps S211 and S212), the series of processes is completed, and the operation returns to the operation from step S201.

[0082] The above-mentioned series of processes for passing through the gate device 100 using an IC card may be performed separately from the face authentication process, but may also be incorporated into the face authentication process. Specifically, for example, when performing the processes from confirmation of intention to pass through to face authentication in steps S101 to S108 in Fig. 9, even if the face authentication is NG, the process may not be immediately abandoned, and confirmation of the presentation of the IC card may also be performed.

[0083] Hereinafter, an example of a series of processes for determining the presence or absence of an intention to pass based on the above-mentioned vector will be described with reference to a flowchart shown in FIG. 11. This corresponds to the process of step S104 in FIG. 9, that is, a specific example of a process for determining the presence or absence of an intention of a target person (passerby HU) to pass by each process based on the estimated moving vector PV. Here, a single target person (passerby HU) is checked multiple times (hereinafter, this number is referred to as the number of attempts, and the number of attempts is counted by the attempt number counter), and the presence or absence of an intention to pass is determined according to the result. Specifically, the presence or absence of an intention to pass is determined according to whether or not the number of times (expected value) that the virtual detection line VL illustrated in FIG. 4 is crossed exceeds a preset threshold within a predetermined upper limit of the number of attempts. The number of times that the virtual detection line VL is crossed is counted by a vector crossing counter.

[0084] First, when a person is detected from an image captured by the camera 10 (step S301), the control unit 50 increments the trial number counter for the target person (passerby HU) who is a person in the image (step S302). That is, the control unit 50 counts how many times the target person (passerby HU) has confirmed the intention to pass.

[0085] Next, the control unit 50 extracts information on the skeleton, face, and gaze of the passerby HU in the image (step S303), and further calculates a skeleton vector, face vector, and gaze vector from these (step S304), and combines these to calculate a traveling direction vector, i.e., an estimated traveling vector PV (step S305).

[0086] Next, the control unit 50 checks whether the estimated moving vector (moving direction vector) PV calculated in step S305 intersects with a specific area in the gate device 100, i.e., whether it has crossed the virtual detection line VL illustrated in FIG. 4 (step S306).

[0087] In step S306, if it is determined that the estimated progress vector PV crosses the virtual detection line VL (step S306: Yes), the control unit 50 increments the vector crossing counter (step S307). On the other hand, in step S306, if it is determined that the estimated progress vector PV does not cross the specific area (step S306: No), the control unit 50 proceeds to the next step without performing any particular process.

[0088] Next, the control unit 50 checks whether the trial counter has reached a predetermined number (upper limit) (step S308), and if not (step S308: No), repeats the process from step S301. On the other hand, if the trial counter has reached the upper limit (step S308: Yes), the control unit 50 calculates an expected value indicating the number of times the estimated progress vector PV has crossed the virtual detection line VL (step S309), and checks whether this is equal to or greater than a preset threshold (step S310).

[0089] In step S310, if the expected value is equal to or greater than the threshold value (step S310: Yes), the control unit 50 determines that the target pedestrian HU has the intention to pass through the gate device 100 (step S311), and if the expected value is not equal to or greater than the threshold value (step S310: No) in step S310, the control unit 50 determines that the target pedestrian HU has no intention to pass through the gate device 100 (step S312).

[0090] When the determination result of step S311 or step S312 is reached, the control unit 50 clears the values ​​of the trial number counter and the vector crossing counter (step S313), and returns to the operation from step S301.

[0091] Hereinafter, an overview of the gate device 100 of this embodiment will be described with reference to a conceptual diagram shown in FIG.

[0092] As shown in the figure and as described above, when the gate device 100 of this embodiment acquires image data including a target pedestrian HU (see FIG. 2, etc.) in the data acquisition unit 41, the vector extraction unit 62 and the vector generation unit 71 extract a plurality of types of vectors from the image data, i.e., the image of the pedestrian HU, and further generate a vector (movement direction vector) that serves as a reference when determining whether the pedestrian HU intends to pass through the gate from these vectors. More specifically, the line of sight, face, and bone structure are extracted from the image, and vectors related to these are calculated, and the calculated vectors are combined (added together) to generate an estimated movement vector (movement direction vector) PV (see FIG. 4, etc.). Then, the determination unit 80 determines whether the pedestrian HU will pass through the target gate from the generated estimated movement vector PV.

[0093] As described above, the gate device 100 of this embodiment includes a data acquisition unit 41 that acquires image data including a pedestrian HU who is a target person, a vector extraction unit 62 that extracts multiple types of vectors related to progress from the acquired image data, a vector generation unit 71 that combines the multiple types of extracted vectors to generate an estimated progress vector (movement direction vector) PV of the pedestrian HU, and a determination unit 80 that determines whether or not the pedestrian HU will pass through the target gate based on the generated estimated progress vector PV. The gate device 100 combines multiple types of vectors related to progress to generate an estimated progress vector PV of the pedestrian HU who is a target person, and determines whether or not the pedestrian HU will pass through the target gate based on this, thereby enabling rapid, accurate and highly reliable determination even in a case where gates are installed in parallel.

[0094] 〔others〕 The present invention is not limited to the above-described embodiment, and can be embodied in various forms without departing from the spirit and scope of the present invention.

[0095] First, in the above, the judgment of whether or not to pass through the gate device 100 has been described in principle in relation to the judgment in the case where one pedestrian HU approaches one gate device 100. However, as shown in Figures 13(A) and 13(B), it is also assumed that multiple people line up vertically (in the passing direction) and enter one gate device 100 in succession, and it is possible to deal with such a case as well.

[0096] In the illustrated example, a case where a passerby HUb immediately follows a preceding passerby HUf is illustrated. In such a case, the camera 10 may be able to process the passerby HUf, but may not be able to start processing the passerby HUb. Specifically, the whole body of the passerby HUf is captured as shown by the dashed line in FIG. 13(B), and an estimated moving vector PV is generated based on each vector V1 to V3 as shown in FIG. 13(A), and the determination process is quickly performed in the determination unit 80 (see FIG. 4, etc.). On the other hand, as shown by the dashed line in FIG. 13(B), the face of the passerby HUb is captured, but an image of the whole body cannot be acquired. As shown in FIG. 13(A), the gaze vector V1 and face vector V2 can be provisionally determined, but the skeleton vector V3 cannot be calculated. In such a case, as shown in FIG. 13(A), the determination unit 80 may calculate a provisional vector PVp by adding up the gaze vector V1 and the face vector V2, perform a preliminary determination based on the provisional vector PVp, and perform a final determination based on the estimated progress vector PV generated when the skeleton vector V3 of the passerby HUb can be calculated, as shown in FIG. 13(C) and FIG. 13(D). In this case, for example, the upper limit number of the trial number counter and the threshold value of the vector intersection counter, which are determined in advance, may be changed (reduced) according to the calculated provisional vector PVp. As described above, the determination unit 80 sequentially determines the possibility of passing through the target gate based on the vectors (for example, the gaze vector V1 and the face vector V2) generated first among the multiple types of vectors V1 to V3, and performs the final determination after the skeleton vector V3 is obtained, thereby making it possible to make a prediction in advance and accurately perform the final determination.

[0097] Although the above describes a case where face recognition and IC card authentication coexist, the present invention can also be applied to a gate device that is dedicated to face recognition. In addition, when IC card authentication is used exclusively, the present invention can also be used to predict the intention of pedestrians to pass through.

[0098] In addition, for the gaze detection, facial direction extraction, and skeletal direction extraction, various methods other than those described above can be used as long as they can maintain the required accuracy and speed. In addition, various modifications are possible for the installation mode of the camera 10, such as installing the camera at a position away from the gate body, or installing multiple cameras.

[0099] In addition, in the above, the estimated progress vector PV is generated by converting each vector for the gaze, face, and skeleton into a valley vector, weighting this, and then adding these together to calculate the estimated progress vector PV. However, the method for generating the estimated progress vector PV is not limited to this, and various methods may be used to specify the method for generating the estimated progress vector PV as long as they serve as an appropriate standard. [Explanation of symbols]

[0100] 10, 10A, 10B, 10C... camera, 11... camera control unit, 20... IC card reader, 30... flap (door), 31... gate control unit, 41... data acquisition unit, 42... data input / output unit, 50... control unit, 60... image processing unit, 61... person detection unit, 62... vector extraction unit, 63... face image feature extraction unit, 70... calculation processing unit, 71... vector generation unit, 72... setting unit, 80... judgment unit, 90... data matching unit, 91... face authentication server, 91c... face matching unit, 91d... face database, 92... IC authentication server, 92c... IC matching unit, 92d... IC database, 100, 100A, 100B, 100C... gate device, CS... intersection, DD3... range, DS... data storage part, FA...face image acquisition range, FD...face direction determination engine software, GD...gaze direction determination engine software, GG...image, GGd...image part, HD...human detection engine software, HU,HUb,HUf,HUα,HUβ,HUγ,HUδ...passersby, HUc...skeleton information, IA...imaging range, IAd...overlapping area, Lb,Ls...straight line, PE...posture estimation engine software, PP...straight line, PV...estimated progress vector (travel direction vector), PVp...provisional vector, PW...partition, SS1,SS2...IR sensor, SV...basic skeleton vector (basic vector), V1...gaze vector, V2...face vector, V3...skeleton vector, VC...vector synthesis part, VL...virtual detection line

Claims

1. A data acquisition unit that acquires image data including a subject; A vector extraction unit that extracts a plurality of types of vectors related to progression from the acquired image data; a vector generation unit that combines the extracted vectors to generate an estimated progress vector of the subject; a determination unit that determines whether or not a target person passes through a target gate based on the generated estimated progress vector; A gate device comprising:

2. 2 . The gate device according to claim 1 , wherein the vector extraction unit extracts at least one of a gaze vector and a face vector in addition to a skeleton vector as the plurality of types of vectors.

3. 3. The gate device according to claim 2, wherein the vector generating section weights the skeleton vector more heavily than other vectors when generating the estimated progress vector.

4. 3. The gate device according to claim 2, wherein the determination unit sequentially determines the possibility of passing through the target gate based on a previously generated vector among the plurality of types of vectors, and makes a final determination after the skeleton vector is obtained.

5. The gate device according to claim 1 , further comprising a passage permission / denial determination unit that determines whether or not to permit passage through the target gate based on face authentication data that has been registered in advance.

6. The gate device according to claim 1 , wherein the determination unit determines whether the vehicle is to pass through the target gate or a gate adjacent to the target gate.

7. The gate device according to claim 1 , wherein the determination unit makes the determination based on whether or not the estimated moving vector has exceeded a detection boundary a predetermined number of times or more.

Images