Information processing device, information processing system, and estimation method

Abstract

To provide an information processing device capable of improving the estimation accuracy of estimating the position of a target which is about to pass through a specific area, an information processing system, and an estimation method.SOLUTION: An information processing device comprises a detection unit and an estimation unit. The detection unit detects a first facial image region included in a first image capturing a person entering a gate from a first direction, and a second facial image region included in a second image capturing a person from a second direction different from the first direction. When the difference between a capture time at which the first image has been captured and a capture time at which the second image has been captured is within a predetermined allowable range, the estimation unit estimates a position of the person at the gate on the basis of a change in the distance between a representative point of the first facial image region and a representative point of the second facial image region.SELECTED DRAWING: Figure 2A

Description

[Technical Field]

[0001] This disclosure relates to an information processing device, an information processing system, and an estimation method. [Background technology]

[0002] Technologies for managing the entry and exit of people passing through gates installed at stations, airports, and other locations are known. Patent Document 1 describes a device that tracks whether a person has passed through a gate (i.e., whether they have returned to the gate entrance) based on changes in the position of a wireless card, after the person has obtained permission to pass through the gate using a wireless card and entered the gate from the entrance. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 09-330440 [Overview of the project] [Problems that the invention aims to solve]

[0004] At gates that manage entry and exit, it is desirable to track which person has passed through the gate. Hereafter, this management may also be abbreviated as "passage management" or "tracking management." There is room for improvement in the accuracy of estimating a person's location in tracking management.

[0005] Non-limiting embodiments of this disclosure contribute to the provision of an information processing device, an information processing system, and an estimation method that can improve the accuracy of estimating the position of an object attempting to pass through a specific region. [Means for solving the problem]

[0006] An information processing device according to one embodiment of the present disclosure includes: a detection unit that detects a first face image region included in a first image taken of a person entering a gate from a first direction, and a second face image region included in a second image taken of the person from a second direction different from the first direction; and an estimation unit that estimates the position of the person at the gate based on the change in distance between a representative point of the first face image region and a representative point of the second face image region, when the difference between the time the first image was taken and the time the second image was taken is within a predetermined allowable range.

[0007] An information processing system according to one embodiment of the present disclosure includes: an authentication device that performs authentication processing of a person using at least one of a first image taken of a person entering a gate from a first direction and a second image taken of the person from a second direction different from the first direction; and an information processing device that detects a first face image region included in the first image and a second face image region included in the second image, and estimates the position of the person at the gate based on the change in distance between a representative point of the first face image region and a representative point of the second face image region when the difference between the time the first image was taken and the time the second image was taken is within a predetermined allowable range.

[0008] An estimation method according to one embodiment of the present disclosure involves an information processing device detecting a first face image region included in a first image taken of a person entering a gate from a first direction, and a second face image region included in a second image taken of the person from a second direction different from the first direction. If the difference between the time the first image was taken and the time the second image was taken is within a predetermined tolerance range, the device estimates the position of the person at the gate based on the change in distance between a representative point of the first face image region and a representative point of the second face image region.

[0009] These comprehensive or specific embodiments may be implemented as systems, devices, methods, integrated circuits, computer programs, or recording media, or as any combination of systems, devices, methods, integrated circuits, computer programs, and recording media. [Effects of the Invention]

[0010] Non-limiting embodiments of this disclosure can improve the accuracy of estimating the position of an object attempting to pass through a specific region.

[0011] Further advantages and effects of one embodiment of this disclosure will be evident from the specification and drawings. Such advantages and / or effects are provided by several embodiments and features described in the specification and drawings, but not all of them are necessarily provided in order to obtain one or more identical features. [Brief explanation of the drawing]

[0012] [Figure 1] A diagram showing an example of a gate according to one embodiment. [Figure 2A] This figure shows an example of the conceptual configuration of a passage management system according to one embodiment. [Figure 2B] Block diagram showing an example configuration of a passage management system according to one embodiment. [Figure 3A] A diagram showing an example of a zone defined by a gate. [Figure 3B] A diagram showing an example of a zone defined by a gate. [Figure 4] A diagram showing an example of face frame detection in one embodiment. [Figure 5A] A diagram showing an example of detecting the position of a person in one embodiment. [Figure 5B] A diagram illustrating an example of the relationship between the size and center of the face frame relative to the position of a person. [Figure 6A] Figure showing the first example of the change in the magnitude of the positional difference. [Figure 6B] Figure showing a second example of the change in the magnitude of the positional difference. [Figure 7]A diagram showing an example of the flow of person tracking processing based on a face frame in an embodiment [Figure 8] A diagram showing an example of an area defined in an image area [Figure 9] A flowchart showing an example of the flow of passage management [Figure 10] A diagram showing an example of face frame timeout processing [Figure 11] A flowchart showing an example of multiple face frame exclusion processing of FFFA [Figure 12] A flowchart showing an example of binocular method face frame position estimation processing [Figure 13] A diagram showing an example of a defined area for differences in camera placement [[ID=1,9]] [Figure 14] A flowchart showing an example of new face frame detection using one camera [[ID=2,3]] [Figure 15] A flowchart showing an example of determination of crossing a charging line using one camera

Embodiments for Carrying Out the Invention

[0013] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functions are denoted by the same reference numerals, and redundant descriptions are omitted.

[0014] (An embodiment) [Findings Leading to the Present Disclosure] In gates installed in facilities such as stations and airports for managing entry and exit to and from the facilities, the use of passage management for accurately managing which person has passed through the gate is being considered. If the passage management is not sufficient, for example, a person who entered through the entrance of the gate but did not head towards the exit of the gate and returned to the entrance of the gate may be erroneously determined to have passed through, or a person who actually passed through may be erroneously recognized as not having passed through. Such errors can lead to incorrect billing of fees in services such as billing people who have passed through a gate such as a ticket gate at a station.

[0015] To implement passage control, for example, an authentication process to authenticate a person attempting to pass through (which may include a process to determine if authentication is not possible) and a tracking process to record the person's movement history are performed. These processes should preferably be carried out early to allow time for processes that restrict people's movement, such as recording people's passage or opening and closing doors.

[0016] For example, a camera could be installed above people and gates (e.g., on the ceiling), with the ceiling camera capturing images of people and gates. The captured images could then be analyzed to track the people being photographed.

[0017] However, when installing cameras on the ceiling, the installation location may be limited depending on the structure or environment of the installation site. Furthermore, even if installation is possible, if it requires extensive construction work, the installation cost will increase. Therefore, it may be difficult to implement a passage management system using ceiling-mounted cameras.

[0018] For example, one could consider providing an arch-shaped or pole-shaped support structure extending above the gate, and mounting a camera on the support structure. However, a gate with such a support structure would be taller than a gate without it, which could limit the possible locations for its installation. Furthermore, from a design perspective, providing a support structure to a gate may not always be desirable.

[0019] In this embodiment, a passage management process including authentication and tracking is performed by using images captured by a camera used for facial recognition of a person attempting to pass through the gate to perform tracking processing. By using the same camera for both authentication and tracking, there is no need to separately install equipment for tracking (e.g., a camera mounted on the ceiling). Therefore, the increase in the cost of introducing passage management can be suppressed. In addition, compared to cases where dedicated equipment such as a camera is installed for tracking, the constraints on installation location are relaxed, increasing the flexibility of installation locations and making it easier to introduce a passage management system.

[0020] <Example of gate configuration> Figure 1 shows an example of a gate 10 according to this embodiment. Figure 1 is a view of the gate 10 from above, illustrating how a person h enters through the entrance / exit E1 of the gate 10 and exits through the entrance / exit E2 of the gate 10. However, a person may also enter the gate 10 shown in Figure 1 through entrance / exit E2 and exit through entrance / exit E1. In other words, the gate 10 allows people to pass through in both directions.

[0021] Gate 10 has, for example, two side walls V facing each other, and a passage L is formed between the side walls V to guide a person passing through gate 10. Cameras 11 are installed on the upper part of one side wall V, which is about 1 m high, at two positions closer to the entrances E1 and E2, respectively, than to the center of the side wall V, and a total of four cameras 11 (11-R1, 11-R2, 11-L1, 11-L2) are installed for the two side walls V.

[0022] Cameras 11-R1 and 11-L1 are installed, for example, on the side wall V located closer to entrance E2 than the center of gate 10, and are used to photograph people entering gate 10 from the opposite entrance E1 and passing through to entrance E2.

[0023] On the other hand, cameras 11-R2 and 11-L2 are installed, for example, on the side wall V located closer to the entrance E1 than the central part of gate 10, and are used to photograph people entering gate 10 from the opposite entrance E2 and passing through to entrance E1.

[0024] For example, camera 11-R1 is positioned to photograph a person entering through entrance E1 from the right front. Camera 11-L1 is positioned to photograph a person entering through entrance E1 from the left front.

[0025] Camera 11-R2 is positioned to capture a person entering from entrance E2, which is opposite to entrance E1, from the right front. Camera 11-L2 is positioned to capture a person entering from entrance E2, which is opposite to entrance E1, from the left front.

[0026] Therefore, a person entering gate 10 from entrance E1 and passing through entrance E2 is photographed from two directions (for example, left and right) by two cameras 11-R1 and 11-L1, which are installed at a distance from each other on the upper part of two side walls V, with a passage L in between.

[0027] On the other hand, a person passing through gate 10 in the opposite direction, that is, a person entering gate 10 from entrance E2 and passing through to entrance E1, is photographed from two directions (for example, left and right) by two cameras 11-R2 and 11-L2, which are installed at a distance from each other on the upper part of the two side walls V, with the passage L in between.

[0028] Figure 1 shows an example configuration in which a person can enter from both entrance E1 and entrance E2 of gate 10, but this disclosure is not limited to this. For example, gate 10 may be configured so that a person can enter from one entrance (e.g., entrance E1) but not from the other entrance (e.g., entrance E2). If gate 10 is configured not to allow a person to enter from entrance E2, cameras 11-R2 and 11-L2 do not need to be provided. If gate 10 is configured not to allow a person to enter from entrance E1, cameras 11-R1 and 11-L1 do not need to be provided.

[0029] In the following section, we will describe an example of how to manage the passage of people entering through entrance E1 and passing through entrance E2 at gate 10 shown in Figure 1, using cameras 11-R1 and 11-L1. For convenience, cameras 11-R1 and 11-L1 may be collectively referred to as camera 11.

[0030] Additionally, camera 11-R1 is sometimes referred to as "right camera 11," and images captured by right camera 11 may be labeled as "right image." Similarly, camera 11-L1 is sometimes referred to as "left camera 11," and images captured by left camera 11 may be labeled as "left image."

[0031] Furthermore, individuals entering Gate 10 are considered to be subject to processing, including facial recognition. Hereafter, individuals subject to processing will be referred to as "subjects."

[0032] Note that the gate 10 in Figure 1 is illustrative, and this disclosure is not limited thereto. For example, the gate 10 may be equipped with five or more cameras 11, or with three or fewer cameras 11. By varying the shooting direction and / or angle of the cameras 11, a wider range of a person's face can be captured.

[0033] If two cameras 11 are provided, one may be positioned to photograph a person entering the gate 10 from a first direction, and the other may be positioned to photograph a person entering the gate 10 from a second direction. Camera 11 may be positioned to photograph the face of a person entering the gate 10 from the front, or it may be positioned to photograph at least a part of the face (for example, the right half or left half of the face). For example, camera 11 may be positioned to capture an image in which a face frame can be detected by face frame detection, which will be described later.

[0034] Note that the multiple cameras 11 do not have to be identical to each other. For example, the multiple cameras 11 may be configured to capture images with different resolutions, angles of view, and image quality. Also, the installation position and / or shooting direction of the cameras 11 may be fixed or adjustable.

[0035] <System Configuration> Figure 2A is a diagram showing an example of the conceptual configuration of the passage management system according to this embodiment. Figure 2B is a block diagram showing an example of the configuration of the passage management system according to this embodiment. The passage management system 1 according to this embodiment is a system that manages the passage of people at gates 10 (e.g., entrance gates, ticket gates, etc.) installed at the entrances and exits of facilities such as airports, train stations, and event venues.

[0036] In the passage management system 1 according to this embodiment, the entry and exit management of users of the facility is performed by facial recognition, as an example. For example, when a user enters the facility by passing through gate 10, facial recognition determines whether or not the user is a person authorized to enter the facility. Also, when a user exits the facility by passing through the gate, facial recognition determines which user is exiting the facility. It should be noted that "facial recognition" can be understood as a concept included in "matching using facial images."

[0037] The passage management system 1 includes, for example, the gate 10 illustrated in Figure 1, cameras 11 (right camera 11 and left camera 11), a facial recognition function unit 13, a person position estimation unit 14, a passage management function unit 15, a facial recognition server 16, and a passage history management server 17. In the passage management system 1, there may be one gate 10 or multiple gates 10.

[0038] Gate 10 is installed in facilities such as airports, train stations, and event venues. Users authorized to use the facility pass through Gate 10 when entering and / or leaving the facility. Gate 10 also controls access to prevent unauthorized persons from passing through.

[0039] Camera 11 is installed, for example, on the side wall V of gate 10, as shown in Figure 1. Camera 11 captures the shooting range including the faces of people passing through gate 10 and people attempting to pass through gate 10. For example, the shooting range of camera 11 is the range that can capture the frontal face of a person.

[0040] The images captured by camera 11 may be used in the person detection process (or person tracking process) described later, or in the face recognition process described later.

[0041] The facial recognition function unit 13 performs facial recognition processing on an image. For example, the facial recognition function unit 13 includes a camera control unit 131 and a face matching processing unit 132.

[0042] The camera control unit 131 controls, for example, the shooting timing of camera 11. For example, camera 11 shoots at a speed of about 5 fps under the control of the camera control unit 131. Also, the right camera 11 and the left camera 11 may shoot simultaneously or with a difference in shooting timing within an acceptable range, under the control of the camera control unit 131. In other words, the shooting timing of the right camera 11 and the shooting timing of the left camera 11 are synchronized, for example, under the control of the camera control unit 131.

[0043] The camera control unit 131 detects a face frame from images (right image and / or left image) captured by the camera 11, for example. The method for detecting a face frame is not particularly limited, but for example, it may be a method that detects parts of the face (eyes, nose and mouth) from the image, and then detects the boundary between the face region and the region outside the face based on the position and color information of the detected parts, thereby detecting the frame surrounding the face region (face frame). If a face frame is detected, for example, the camera control unit 131 outputs information about the detected face frame (face frame information) and the captured image to the face matching processing unit 132.

[0044] The face matching processing unit 132 extracts the face region included in the image based on the face frame information, and notifies the face authentication server 16 of a face matching request including the information of the extracted face region. The information of the face region may be, for example, an image of the face region, or information indicating feature points extracted from the image of the face region.

[0045] The facial recognition server 16 has, as an example, registered facial images of persons permitted to pass through gate 10. Facial images registered with the facial recognition server 16 may be referred to as registered facial images. Registered facial images may be associated with information that uniquely identifies or specifies a person, such as the ID of the registered person. Furthermore, registered facial images may also represent information indicating feature points extracted from the image.

[0046] When the face recognition server 16 receives a face matching request from the face matching processing unit 132, for example, it determines whether the face of the same person as the face in the face region included in the face matching request is included in the registered face image. The face recognition server 16 then notifies the face matching result, including the determination result, to the face matching processing unit 132. The face matching result may include, for example, information indicating whether the face of the same person as the face in the face region is included in the registered face image (for example, a flag indicating "OK" or "NG"), and, if the face of the same person as the face in the face region is included in the registered face image, information about the person associated with the registered face image (for example, an ID).

[0047] Matching, for example, involves comparing a registered facial image with the facial image of a person passing through gate 10 to determine whether the pre-registered facial image matches the facial image of the person passing through gate 10, or whether the pre-registered facial image and the facial image of the person passing through gate 10 belong to the same person.

[0048] On the other hand, authentication is the process of proving to an external party (for example, gate 10) that the person whose facial image matches a pre-registered facial image is indeed that person (in other words, that they are a person who should be allowed to pass through gate 10).

[0049] However, in this disclosure, the terms "verification" and "authentication" may be used interchangeably.

[0050] For example, the matching process involves comparing the feature points of a pre-registered face image with the feature points extracted from the detected face region to identify whose face is in the image data. This matching process may employ, for example, a machine learning method. The matching process may be performed, for example, in the face recognition server 16, but it may also be performed in another device such as the gate 10, or it may be distributed and performed by multiple devices.

[0051] The face matching processing unit 132 outputs information including the matching processing result to the passage management function unit 15. The matching processing result may include, for example, information about the registered face image and a matching score. The information output from the face matching processing unit 132 may also include, for example, face frame detection information and the timestamp of the face camera image in which the face frame was detected.

[0052] The person position estimation unit 14 performs person tracking processing, for example, based on face frame information. The person position estimation unit 14 includes, for example, a person tracking processing unit 141.

[0053] The person tracking processing unit 141 estimates the position of a person relative to the gate 10, for example, based on face frame information. Then, the person tracking processing unit 141 determines events that occur to the person by tracking the estimated position of the person. For example, events that occur to a person include the appearance of a new person, tracking of a person, and the disappearance of a person. The person tracking processing unit 141 determines events based on the person's position, for example, and tracks the person by associating the determined event with information such as the person's position and the time of detection.

[0054] The person tracking processing unit 141 outputs, for example, information related to person tracking to the passage management function unit 15. For example, the information related to person tracking includes information such as the person's location and the time of detection.

[0055] The passage management function unit 15 manages the status of persons located around the gate 10 by, for example, associating information output from the face recognition function unit 13 with information output from the person position estimation unit 14. Persons located around the gate 10 include, for example, persons passing through the gate 10, persons attempting to pass through, and persons passing by the gate 10. Here, persons attempting to pass through the gate 10 are not limited to persons authorized to pass through the gate 10 (for example, persons whose face image has been registered with the face recognition server 16), but may also include, for example, persons whose face image has not been registered with the face recognition server 16 but who are attempting to pass through. Persons passing by the gate 10 include, for example, persons who are not attempting to pass through the gate 10 but are passing through the shooting range of the camera 11, or persons who are not attempting to pass through the gate 10 but are entering the shooting range. Furthermore, the status of a person may be, for example, a status related to the person's movement, such as whether the person is moving or stationary, and, if the person is moving, the direction of movement.

[0056] The passage management function unit 15 includes, for example, a passage management state transition processing unit 151, a history management unit 152, and a history database (DB) 153.

[0057] The passage management state transition processing unit 151 transmits control information to gate 10 regarding the control of gate 10 when a person authorized to pass through gate 10 attempts to pass through, for example, in the process of managing the passage of a person. The passage management state transition processing unit 151 also transmits control information to gate 10 regarding the control of gate 10 when a person not authorized to pass through gate 10 attempts to pass through.

[0058] The history management unit 152 stores and manages information (passage history information) that shows the history of people who have passed through gate 10, for example. The history management unit 152 also stores the passage history information in the history DB 153 and transmits the passage history information to the passage history management server 17. For example, in a railway network, the history management unit 152 may manage local passage history information on a station (or ticket gate) basis.

[0059] The passage history management server 17 stores and manages information (passage history information) that indicates the history of people who have passed through gate 10. For example, the passage history management server 17 may manage passage history information for multiple gates 10. For example, in a large facility with multiple entrances and exits, the passage history information for gates 10 installed at each of the multiple entrances and exits may be aggregated and managed by the passage history management server 17. Also, for example, in a railway network, the passage history information for each of the gates 10 installed at the ticket gates of multiple stations may be aggregated and managed by the passage history management server 17.

[0060] The passage management function unit 15 may, for example, output information related to passage management (passage management information) to a display device (not shown in the figure). The passage management information may include, for example, information output from the face recognition function unit 13 and information output from the person position estimation unit 14. The display device may, for example, display the person's status (for example, the result of face recognition of the person and the direction of movement). For example, the display device may display the right image and / or the left image and superimpose the detected face frame onto the right image and / or the left image. The display device may also, for example, superimpose information about the person obtained by face recognition (person's ID) onto the right image and / or the left image.

[0061] The facial recognition function unit 13 described above may operate synchronously with, for example, the passage management function unit 15, or it may operate asynchronously. In the case of asynchronous operation, for example, the facial recognition function unit 13 may operate when a face frame is detected in the camera control unit 131.

[0062] The three components described above—the facial recognition function unit 13, the person position estimation unit 14, and the passage management function unit 15—may each take the form of a single information processing device (e.g., a server device), or two or more of the three may be included in a single information processing device. For example, the facial recognition function unit 13 may take the form of a single information processing device, and the person position estimation unit 14 and the passage management function unit 15 may be included in a single information processing device. Furthermore, at least one of the facial recognition function unit 13, the person position estimation unit 14, and the passage management function unit 15, which take the form of an information processing device, may be included in the gate 10.

[0063] The information processing device described above may include a processor, memory, and an input / output interface used for transmitting various types of information. The processor is a computing device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The memory is a storage device implemented using RAM (Random Access Memory), ROM (Read Only Memory), etc. The processor, memory, and input / output interface are connected to a bus, and various types of information are exchanged via the bus. The processor, for example, reads programs and data stored in ROM onto RAM and executes processing, thereby realizing the functions of the components included in the information processing device.

[0064] In the person position estimation unit 14 and passage management function unit 15 described above, for example, an area (or zone) may be defined at gate 10, and person detection and passage management may be performed based on the defined zone. An example of a zone defined at gate 10 is described below.

[0065] <Gate Area Management> Figures 3A and 3B show examples of zones defined for gate 10. Figures 3A and 3B show examples of multiple zones when gate 10 is viewed from above. In addition, Figures 3A and 3B show examples in which the side wall V of gate 10 that forms the passage L extends along the vertical direction of the paper.

[0066] As shown in Figure 1, of the entrances and exits E1 and E2 of gate 10, for example, the upstream side along a specific direction of entry (for example, the direction of entry) corresponds to the entrance, and the downstream side corresponds to the exit.

[0067] Figure 3A shows an example of the zone defined when a person enters gate 10 from entrance E2, where a person can enter from both entrance E1 and E2. Figure 3B shows an example of the zone defined when a person enters from entrance E1.

[0068] If gate 10 allows entry in both directions, the direction of movement may differ depending on the entrance / exit used. For example, the direction of movement of a person from entrance / exit E1 to entrance / exit E2 may be the normal direction of movement for a person entering from entrance / exit E1, but it may be the abnormal direction of movement for a person entering from entrance / exit E2. To address these differences in regulations, the passage management function may, for example, define entrance / exit E1 as the "North side" and entrance / exit E2 as the "South side".

[0069] The terms "north side" and "south side" are examples only, and this disclosure is not limited to these terms. For example, the terms "north side" and "south side" do not limit the arrangement of gate 10 to an arrangement along the geographical north-south direction. For example, even if the passage L of gate 10 is located along a direction other than the north-south direction, or if the passage includes a curve, one side may be designated as the "north side" and the other as the "south side."

[0070] For example, Figure 3A shows an example of zones defined when a person enters through entrance E2. In Figure 3A, the following zones are defined for gate 10: "Zone outside-S" (south side zone outside area), "Zone A" (Zone A), "Zone B" (Zone B), and "Zone C" (Zone C).

[0071] In contrast, Figure 3B shows an example of zones defined when a person enters through entrance E1. In Figure 3B, "Zone outside-N" (north side zone outside area), "Zone A" (Zone A), "Zone B" (Zone B), and "Zone C" (Zone C) are defined for gate 10.

[0072] The following sections will explain each zone using the example in Figure 3A. Note that the example in Figure 3B is the same as in Figure 3A, except that the person enters through entrance E1, and that "Zone outside-S" (south zone outside area) is replaced with "Zone outside-N" (north zone outside area).

[0073] The boundary between the southern zone's outer area and Zone A may be referred to, for example, as the "facial recognition start line."

[0074] The "face recognition start line" is used, for example, to determine whether or not to start the face recognition process. For example, if a person crosses the "face recognition start line" and enters gate 10, the face recognition process is started. For example, a face matching request is issued from the face frame information, the matching result (face recognition ID) is linked with the person detection information, and the person tracking is started. The "face recognition start line" is sometimes referred to as the "A line (A LINE)".

[0075] The "face recognition start line" may be located outside the gate 10 (for example, upstream along the path of the gate 10). Furthermore, the "face recognition start line" is not limited to a single line segment, but may have multiple line segments, for example, in the shape of a U. Note that the shape with multiple line segments is not limited to a shape corresponding to a part of a rectangular shape such as a U, but may also be a shape corresponding to a part of a polygonal shape. Alternatively, the "face recognition start line" may have an arc, or a shape in which straight lines and curves are mixed. For example, by having multiple line segments and / or arcs in the "face recognition start line," face recognition processing can be started not only when a person enters the gate 10 from the front, but also when a person enters from a direction offset from the front, such as the side.

[0076] The boundary between Zone A and Zone B may be referred to, for example, as the "door closing limit line."

[0077] The "closed gate limit line" indicates, for example, the position at which the exit-side gate door, in response to a closing instruction, will close in time for a person to pass through. The "closed gate limit line" is determined by considering, for example, the maximum speed at which a person is expected to pass through gate 10 (e.g., 6 km / h; hereinafter referred to as the "maximum passable speed") and the time required to physically close the gate door (e.g., 0.5 seconds). For example, the "closed gate limit line" is set in front of the physical position of the gate door ("gate door position") by a distance equivalent to the maximum passable speed multiplied by the time required to physically close the gate door. In this way, if a person who is not permitted to pass through gate 10 passes the "closed gate limit line" and moves at the maximum passable speed, the exit-side gate door will close before that person passes through.

[0078] The "door closing limit line" may also be referred to as the "unauthorized intrusion detection line" or "B line (B LINE)".

[0079] The boundary between Zone B and Zone C can be referred to as the "ejection line."

[0080] The "exit line" indicates, for example, the position where it is determined that the person has exited gate 10. The "exit line" may be located outside gate 10, for example, similar to the "face recognition start line" described above. Furthermore, the "exit line" is not limited to a single line segment, but may have multiple line segments, for example, in a U-shape. Alternatively, the "exit line" may have an arc. The "exit line" may also be referred to as, for example, the "Z line (Z LINE)".

[0081] In passage management, the gate door position may simply be a passing point, and in this case, the gate door position may be different from or the same as the logically defined "exit line." For example, in actual operation, the "gate door position" and the "exit line" may be set to be the same.

[0082] For example, in the case of gate 10, which charges a fee to people passing through, the "exit line" may correspond to the "charge line."

[0083] For example, if a person who has entered Gate 10 crosses the charge line (for example, entering Zone C from Zone B), that person will be charged. In other words, if a person has not crossed the charge line (for example, has not entered Zone C), that person will not be charged. By setting up this charge line, it is possible to avoid the error of charging a person who has entered Gate 10 but turned back before crossing the charge line.

[0084] In the above example, the "charge line" was shown as being equivalent to the "Z line" ("exit line"), but for example, the "charge line" could also be equivalent to the "B line".

[0085] In the example described above, three zones are defined, excluding the areas outside the northern and southern zones, but this disclosure is not limited to this example. The number, size, location, and shape of the zones may be modified depending on the situation to which this disclosure applies.

[0086] By detecting the position of a person in the aforementioned zones, movement between zones can be estimated, for example. In this embodiment, the position of a person is estimated using, for example, images captured by the right camera 11 and the left camera 11.

[0087] Figure 4 shows an example of face frame detection in this embodiment. Figure 4 shows right images R1 and R2 captured by the right camera 11, and left images L1 and L2 captured by the left camera 11. Right image R1 and left image L1 are images captured at the same time t1, for example. Right image R2 and left image L2 are images captured at the same time t2, for example. Time t2 is later than time t1. For example, the position of the person at time t2 is closer to the charging line of gate 10 than the position of the person at time t1.

[0088] The right images R1 and R2, the left image L1, and the left image L2 include, for example, a person passing through gate 10 and a frame (face frame) surrounding the person's face.

[0089] For example, when comparing left image L1 and left image L2, the face frame in left image L2 is closer to the left edge of the image area than the face frame in left image L1. Also, when comparing right image R1 and right image R2, the face frame in right image R2 is closer to the right edge of the image area than the face frame in right image R1.

[0090] In other words, by checking the positional relationship between the face frame in the left image and the face frame in the right image (for example, the positional relationship in the left-right direction within the image area), the position of a person can be detected. Hereafter, the left-right direction within the image area will be defined as the horizontal direction or the X-axis direction.

[0091] Figure 5A shows an example of detecting a person's position in this embodiment. Figure 5A shows two images based on the image shown in Figure 4, and an extracted image obtained by extracting the face frame from the two images.

[0092] Image T1 in Figure 5A shows an example of comparing the positional relationship of the face frames of the left image L1 and the right image R1 at time t1 shown in Figure 4. For example, the right side of image T1 shows a partial region including the face frame of the left image L1 shown in Figure 4, and the left side of image T1 shows a partial region including the face frame of the right image R1 shown in Figure 4. In image T1, the face frame of the left image L1 is located to the right of the face frame of the right image R1.

[0093] The image T2 in FIG. 5A shows an example of comparing the positional relationship of the face frames between the left image L2 and the right image R2 at the time t2 shown in FIG. 4. For example, the right side of the image T2 shows a partial region including the face frame of the right image R2 shown in FIG. 4, and the left side of the image T2 shows a partial region including the face frame of the left image L2 shown in FIG. 4. In the image T2, the face frame of the left image L2 is located to the left of the face frame of the right image R2.

[0094] As shown in FIG. 5A, in the right image and the left image taken at the same time, the positional relationship between the face frame of the right image and the face frame of the left image changes according to the position of the person. Therefore, in the present embodiment, the position of the person is estimated based on the difference between the position of the face frame of the right image and the position of the face frame of the left image.

[0095] For example, the position of the face frame is represented by a representative point of the face frame. Hereinafter, an example in which the representative point is the center point of the face frame will be described. Since the center point of the face frame does not change significantly even if the size of the face is different, by using the center point of the face frame as the representative point, the position of the person can be stably estimated even when there is variation in the size of the person's face. However, the present disclosure is not limited to this. When the face frame is rectangular, the representative point may be a point indicating the corner of the rectangle. When the face frame is elliptical, the representative point may be the focus of the ellipse.

[0096] As shown in the extracted images U1 and U2, a point P indicating the center of the face frame of the right image R and a point P indicating the center of the face frame of the left image L The distance between, for example, starting from point P R Point P R From point P L It may be represented by the horizontal component (the component along the X-axis) of the vector to R Hereinafter, the point P of the face frame of the right image L The distance between and the point P of the face frame of the left image is sometimes referred to as the horizontal coordinate position difference or the position difference.

[0097] For example, in the extracted image U1, point P R Is point P LBecause it is to the left of point P, the horizontal component of the vector has a positive value. Also, in the extracted image U2, point P R Point P L Because it is to the right of the point, the horizontal component of the vector has a negative value.

[0098] Figure 5B shows an example of the relationship between the size and center of the face frame relative to the position of the person. Figure 5B shows the positional relationship of the center of the face frame for three different distances (far, medium, and close) from the charging line of gate 10, as well as the relationship between the size of the face frame and the direction of the vector defined by the center of the face frame.

[0099] As shown in Figure 5B, in the case where the distance from the charging line is far, when comparing the center of the face frame in the right image with the center of the face frame in the left image within the image region, the center of the face frame in the right image is located to the left, and the center of the face frame in the left image is located to the right. Furthermore, as the distance approaches the charging line, the center of the face frame in the right image moves to the right, and the center of the face frame in the left image moves to the left. Therefore, the direction of the vector from the center of the face frame in the right image to the center of the face frame in the left image changes from positive to negative; in other words, the polarity of the horizontal distance from the center of the face frame in the right image to the center of the face frame in the left image reverses.

[0100] The person's position is estimated by comparing the position difference with a threshold. For example, if the position difference is less than or equal to the threshold, it is determined that the person is located beyond the charging line.

[0101] Alternatively, when a person passes through a gate, the position difference gradually decreases from a positive value to a negative value. When the position difference changes from a positive value to a negative value (determined by polarity), the positional relationship between the face frame in the left image and the face frame in the right image is reversed. In other words, the face frame in the left image and the face frame in the right image cross at a position difference of zero. Therefore, when the position difference becomes zero, it can be determined that the person is located beyond the charging line.

[0102] For example, the camera's field of view, the gate size, and the position of the charge line may be determined such that the point where the face frame of the left image and the face frame of the right image intersect (cross point) coincides with the charge line.

[0103] The position of the point where the face frame in the left image and the face frame in the right image intersect (the cross point) is less affected by the size of the face frame (i.e., the size of the person's face). Therefore, if the position of the cross point corresponds to the billing line, the accuracy of billing line detection can be improved.

[0104] Furthermore, by using the center of the face frame and calculating the difference in the horizontal coordinates of the center to perform estimation, the influence of differences in face frame size due to face size and / or height can be avoided, thereby suppressing or avoiding a decrease in estimation accuracy.

[0105] Figure 6A shows a first example of the progression of the magnitude of the positional difference. Figure 6B shows a second example of the progression of the magnitude of the positional difference. In Figures 6A and 6B, the arrangement of the right camera 11 and the left camera 11 is different from each other. In Figures 6A and 6B, the horizontal axis shows the distance along the passage from a position where face recognition is possible outside the entrance of gate 10, and the vertical axis shows the magnitude of the positional difference. Note that when the positional difference is a positive value, the center of the face frame in the left image is to the right of the face frame in the right image. Note that when the positional difference is a negative value, the face frame in the right image is to the right of the face frame in the left image.

[0106] As shown in Figures 6A and 6B, the positional difference is zero at a certain position. Furthermore, when comparing Figure 6A and Figure 6B, the position where the positional difference is zero may differ depending on the arrangement of camera 11.

[0107] As described above, if the charge line for gate 10 is defined, the position where the positional difference is zero can be associated with the charge line by, for example, adjusting the placement and / or angle of camera 11. Alternatively, since the position where the positional difference is zero is defined by the camera placement, the charge line can be adjusted.

[0108] Next, we will explain the process of tracking a person based on the detected face frame. Figure 7 is a diagram showing an example of the process of tracking a person based on a face frame in this embodiment.

[0109] The face frame position detection library 201 is a library that, for example, acquires face frame detection information and has the function of detecting the position of a person corresponding to the face frame. The face frame position detection library 201 detects the position of a person from the position of the face frame in the image, for example, by the method described above.

[0110] Furthermore, the face frame position determination library 201 may, for example, assign a new person ID to the face frame detection information if the acquired face frame detection information is not continuous with previously acquired face frame detection information in terms of time and / or coordinate space.

[0111] Furthermore, the face frame position determination library 201, for example, if the acquired face frame detection information has continuity with previously acquired face frame detection information in terms of time and / or coordinate space, will continue tracking the person based on the continuity of the face frame detection information.

[0112] The face frame position determination library 201 outputs, for example, person tracking information to the passage management library 202.

[0113] Person tracking information may include, for example, a person ID to identify a person. Person tracking information may also include, for example, information about a person's location (e.g., appearance of a person, tracking of a person, disappearance of a person, etc.). Person tracking information may be compatible between, for example, the face frame position determination library 201 and the passage management library 202.

[0114] The passage management library 202 has a function to identify passage management events at gate 10 based on, for example, person tracking information. The passage management library 202 also outputs passage management events to the passage management processing unit 203. A passage management event includes, for example, at least one of several events, such as an event indicating that a person has moved between zones, an event indicating that a person has crossed a line defining the zones, an event indicating that a person has appeared in a certain zone, and an event indicating that a person has disappeared in a certain zone, in the multiple zones shown in Figures 3A and 3B.

[0115] The passage management processing unit 203 outputs passage information indicating, for example, whether a person has passed through gate 10 or whether a person has passed through the billing line, based on a passage management event.

[0116] The result output unit 204 outputs, for example, the tracking results of the person indicated by the passage information. For example, the result output unit 204 displays the results on a display.

[0117] Next, we will explain the area defined for the face frame detected in the image.

[0118] Figure 8 shows an example of an area defined within an image region. Figure 8 shows an image region with the top left corner as the origin (0,0), where the horizontal length (X-axis direction) of the frame is represented as "XframeMax" and the vertical length as "YframeMax". Figure 8 also shows, exemplarily, a face frame with side length wf and top-left coordinates (Xf,Yf), and the center of the face frame (Xc,Yc). The information representing the face frame may be expressed, for example, by the X,Y coordinates representing the top-left point of the face frame and W representing one side of the face frame, as shown in Figure 8.

[0119] As shown in Figure 8, three areas may be defined within the image region of the left image. For example, of the three areas, the right area represents the First Face Frame Area (FFFA), the center area represents the Middle Face Frame Area (MFFA), and the left area represents the Passed Face Frame Area (PFFA).

[0120] Note that Figure 8 shows three areas in the image region of the left image, but in the image region of the right image, the left and right sides of the three areas may be reversed. In other words, in the image region of the right image, the left area may represent FFFA and the right area may represent PFFA. Also, the size and position of the three areas in Figure 8 are illustrative and the disclosure is not limited thereto. The size and position of the three areas may differ for each camera 11.

[0121] For example, the face frame position determination library 201 may determine, based on the face frame detection information, which of the three areas in the image region the center of the detected face frame is located in.

[0122] Figure 9 is a flowchart showing an example of the passage management flow. The flow shown in Figure 9 may be started, for example, each time information from an image captured by camera 11 is acquired.

[0123] The facial recognition function unit 13 acquires information from the image captured by the camera 11 and performs a process to detect a face frame from the image (S101). The image information acquired from the camera 11 may include information identifying the camera that took the picture (hereinafter referred to as the camera ID), information identifying the frame that was captured (hereinafter referred to as the frame ID), the date and time of capture, etc. The facial recognition function unit 13 generates information regarding the detected face frame and outputs the face frame information and the image information to the person position estimation unit 14. The face frame information may include information indicating the position and size of the face frame. If no face frame is detected from the image, the processes from S102 onward do not need to be executed.

[0124] Next, the person position estimation unit 14 performs the following processing based on the face frame position determination library 201.

[0125] The person position estimation unit 14 performs a face frame timeout process (S102). In the face frame timeout process, the unit performs a timeout process for the face frame detection information held in the face frame information list based on the elapsed time since the first acquisition of the face frame detection information. The face frame timeout process will be described later.

[0126] Next, the person position estimation unit 14 performs face frame detection information generation processing (S103). For example, in the face frame detection information generation processing, face frame detection information is generated, which includes information such as the camera ID, frame ID, shooting date and time, and the coordinates of the detected face frame. The face frame detection information may also include information about the area where the face frame was detected. The area where the face frame was detected is determined based on area definitions, for example, as illustrated in Figure 8. Area definitions may be included in face frame appearance area definition information. Note that if multiple face frames are detected in a single image, the face frame detection information generation processing is performed for each of the multiple face frames. The face frame detection information generation processing will be described later.

[0127] Next, the person position estimation unit 14 performs FFFA multiple face frame elimination processing (S104). In FFFA multiple face frame elimination processing, it is determined whether or not multiple face frames exist in FFFA, and if multiple face frames exist in FFFA, the subsequent processing is not executed. The FFFA multiple face frame elimination processing will be described later.

[0128] The person position estimation unit 14 performs a time-series face frame detection evaluation process (S105). In the time-series face frame detection evaluation process, the face frame detection information is evaluated by comparing the latest information in the face frame detection table with the information immediately preceding it. For example, if the face frame shown in the face frame detection information is the face frame of a newly appeared person, a new ID is assigned to that person.

[0129] Next, the person position estimation unit 14 performs a two-eye face frame position estimation process (S106). The two-eye face frame position estimation process generates person tracking information. The two-eye face frame position estimation process will be described later.

[0130] Next, the person position estimation unit 14 outputs the person tracking data to the passage management function unit 15 (S106). Then, the flow shown in Figure 9 is completed, and the passage management function unit 15 performs passage management processing based on the passage management library.

[0131] <Face frame detection information generation process> In the face frame detection information generation process (S103), the detected position of the face frame is determined. For example, in both the right and left images, it is determined which area defined within the image region the detected face frame falls into.

[0132] For example, extract one face frame from the face frame information. Calculate the center of the extracted face frame and determine which area (FFFA, MFFA, or PFFA) the center lies in.

[0133] For example, if the center of a face frame does not exist in any area, the information for that face frame is discarded. This is because if the center of a face frame does not exist in any area, it is highly likely that it belongs to a fraudulent person.

[0134] If the center of a face frame lies within one of the three areas, the size of that face frame is compared to the minimum face frame size set for that area. If the size of the face frame is smaller than the minimum face frame size, the information for that face frame is discarded. This is because a face frame smaller than the minimum face frame size is likely to be the face of an unauthorized person, such as a person unlikely to enter Gate 10, or a face depicted on clothing or a poster being detected, or a pattern or other non-face image being mistakenly identified as a face.

[0135] If multiple face frames are detected in a single image, the process described above—determining the area where the face frames exist and comparing it to the minimum face frame size set for that area—is performed for each of the multiple face frames.

[0136] A list may be generated from the face frame detection information for performing time-series determination processing. The generated list may be called the face frame information list (or the queue structure list).

[0137] The face frame information list may include the center coordinates of the face frame and information about the area where it appears in the image region.

[0138] <Face frame timeout processing> Next, we will explain the face frame timeout process (S102). Figure 10 shows an example of the face frame timeout process.

[0139] In the face frame timeout process (S102), the retention time from the time the face frame to be judged was first detected, for example, the time when the face frame to be judged first appeared and a new ID was assigned, to the time the image containing the face frame was captured, is compared with the face frame retention limit (face frame retention limit). The retention time may be determined by referring to a face frame information list, for example. The face frame retention limit is a predetermined threshold and is stored in a memory unit or similar.

[0140] For example, in the face frame timeout process, if the retention time of information for a face frame with an assigned ID exceeds the time limit, the information for that face frame is removed from the face frame information list.

[0141] This process prevents information about individuals whose faces were detected but who did not pass through gate 10 and were therefore outside the camera's field of view from remaining in the list.

[0142] Figure 9 shows an example in which the face frame timeout process is executed before the face frame detection information generation process, but this disclosure is not limited to this example. The face frame timeout process may also continuously check whether a timeout has occurred in the face frame information.

[0143] <FFFA Multiple Face Frame Exclusion Process> Next, the FFFA multiple face frame exclusion process (S104) will be described. FIG. 11 is a flowchart showing an example of the FFFA multiple face frame exclusion process. For example, the flow shown in FIG. 11 starts after S103 shown in FIG. 9.

[0144] In the initial setting, the person position estimation unit 14 sets the variable i to the number of detected face frames (S201). Note that i in FIG. 11 is an integer of 1 or more. Also, hereinafter, the case where the number of detected face frames is N (N is an integer of 1 or more) will be described. In this case, the N detected face frames are represented as face frame [1] to face frame [N].

[0145] The person position estimation unit 14 sets the variable "count" to 0 (S202).

[0146] The person position estimation unit 14 determines whether i is greater than 0 (S203).

[0147] [[ID=2o]] If i is greater than 0 (YES in S203), the person position estimation unit 14 determines whether face frame [i] is detected within the FFFA (S204).

[0148] If face frame [i] is detected within the FFFA (YES in S204), the person position estimation unit 14 determines whether the size of face frame [i] is equal to or greater than a predetermined size (S205). Note that the predetermined size may be, for example, the size of a face frame at which face authentication can be started, or a size defined based on the size of a face frame at which face authentication can be started. ]>

[0149] If the size of face frame [i] is equal to or greater than the predetermined size (YES in S205), the person position estimation unit 14 adds 1 to count (S206). ]>

[0150] Then, the person position estimation unit 14 subtracts 1 from i (S207). Then, the process of S203 is executed.

[0151] If face frame [i] is not detected within FFFA (NO in S204), or if the size of face frame [i] is not greater than or equal to a predetermined size (NO in S205), the person position estimation unit 14 subtracts 1 from i (S207). Then, the process in S203 is executed.

[0152] If i is not greater than 0 (NO in S203), for example, after processing S204 to S206 is completed for each detected face frame, the person position estimation unit 14 determines whether count is 2 or greater (S208).

[0153] If the count is 2 or greater (YES in S208), that is, if there are two or more face frames of a predetermined size or larger within the FFFA, the person position estimation unit 14 deletes the information of the face frames corresponding to the face frames within the FFFA (S209). Then the flow shown in Figure 11 ends.

[0154] If the count is not 2 or greater (NO in S208), that is, if there are not two or more face frames of a predetermined size or larger within the FFFA, the person position estimation unit 14 may terminate processing without deleting the face frame information.

[0155] As described above, if there are multiple face frames in the FFFA that are larger than the threshold size for initiating face recognition, face recognition processing will not be performed for each of the detected face frames. If face recognition processing is not performed for each of the detected face frames, the information of the face frames corresponding to the detected face frames may be deleted from the FFFA. In this case, since face recognition processing cannot be performed, error processing is executed. For example, the output control process of the gate device may instruct a person attempting to pass through the gate device to display a warning message via an audio output device and / or a display device.

[0156] This process eliminates fraudulent passage even when multiple people's face frames are arranged in a vertical column and multiple people attempt to pass through the gate device illegally at the same time.

[0157] <Two-lens facial frame position estimation processing> Next, the binocular face frame position estimation process (S106) will be explained. Figure 12 is a flowchart of an example of the binocular face frame position estimation process. For example, the flow shown in Figure 12 starts after S105 shown in Figure 9.

[0158] The person position estimation unit 14 determines whether or not a face frame exists in the face frame information list (S301).

[0159] If a face frame is detected (YES in S301), the person position estimation unit 14 determines whether the difference between the time the left image was captured by the left camera 11 and the time the right image was captured by the right camera 11 is within an acceptable range (S302). In other words, S302 determines whether the time the left image was captured and the time the right image was captured are simultaneous, nearly simultaneous, or have an unacceptable difference. In other words, it determines whether the time the left image was captured and the time the right image was captured are synchronized.

[0160] If the difference in shooting time is outside the acceptable range (NO in S302), the person position estimation unit 14 records a synchronization error in the log (S306).

[0161] If the difference in shooting time is within an acceptable range (YES in S302), the person position estimation unit 14 calculates the difference between the X coordinate of the center of the face frame in the left image and the X coordinate of the center of the face frame in the right image (S303). This difference in the X coordinate of the center of the face frame may be referred to as the "center coordinate difference value" below.

[0162] The person position estimation unit 14 then converts the calculated difference into distance information (S304). For example, a conversion table may be used for this conversion. An example of a conversion table will be described later.

[0163] The person position estimation unit 14 stores the converted result (S305). Then the flow ends.

[0164] If no face frame is detected (NO in S301), or after recording a synchronization error (after S306), the person position estimation unit 14 stores a result indicating that position estimation was not performed (S307). For example, the result storage area may contain information indicating "no two-eye detection." The flow then ends.

[0165] <Conversion Table> Next, an example of a conversion table used in S304 in Figure 12 will be described.

[0166] For example, the transformation table records the difference value of the center coordinates and the estimated location where the target person is located in association with each other. For example, the numerical information stored in the transformation table may be recorded in pixel units (e.g., 1 pixel units). The difference value may also be negative. Furthermore, the estimated location where the target person is located may be represented by the distance from a reference point provided at gate 10, or by two-dimensional coordinates from the reference point (e.g., X coordinate and Y coordinate).

[0167] For example, the conversion table may be loaded in memory to speed up processing related to transit management.

[0168] Furthermore, if gate 10 is passable from both directions, a conversion table may be provided for each direction, for example.

[0169] The conversion table may be created when gate 10 is installed, or it may be provided from an external source.

[0170] In this embodiment, the center of the face frame in the right image, which shows the face region of a person entering through the entrance of gate 10, is determined, as is the center of the face frame in the left image, which shows the face from the left front of the person. The position of the person at gate 10 is then estimated based on the change in the positional relationship between the center of the face frame in the right image and the center of the face frame in the left image. This configuration improves the accuracy of estimating the position of a person attempting to pass through the gate's charging line.

[0171] Furthermore, according to this embodiment, a passage management process including authentication and tracking is performed by using the image captured by the camera used for facial recognition processing to perform tracking processing. As a result, the camera used for authentication processing can be used for both authentication and tracking processing without the need to install a separate device for tracking processing (for example, a camera mounted on the ceiling), thereby suppressing the increase in the cost of introducing passage management and enabling the introduction of a passage management system without location restrictions.

[0172] Furthermore, according to this embodiment, since the position of a person is estimated based on the distance between the centers of the face frames obtained from the left image and the right image, estimation can be performed even if there are differences in the installed position, field of view, image quality, and resolution between the right camera 11 and the left camera 11. For example, in a stereo camera, the distance to the subject is measured using the parallax of both cameras. Therefore, in a stereo camera, it is necessary to precisely adjust the angles of both cameras, and dedicated control processing for stereo cameras is required. In this embodiment, since the position is estimated using the positional relationship of the face frames detected in the face recognition process, estimation can be performed more simply than with the stereo camera method without having to precisely adjust the cameras.

[0173] The configuration shown in this embodiment is merely an example, and this disclosure is not limited thereto. The following describes variations in the position of the camera 11 installed in the gate 10.

[0174] Figure 1 shows an example where the distance from the entrance / exit E1 of gate 10 to the right camera 11 (camera 11-R1) and the distance from the entrance / exit E1 to the left camera 11 (camera 11-L1) are equal, but the disclosure is not limited to this. For example, one of the right camera 11 and the left camera 11 may be installed closer to the entrance / exit E1 of gate 10 than the other. In this case, since there is a difference in the installation positions of the right camera 11 and the left camera 11 with respect to the entrance / exit E1 of gate 10, there is a difference in front-to-back positioning between the shooting range of the right image and the shooting range of the left image.

[0175] For example, if the left camera 11 is positioned closer to the entrance / exit E1 than the right camera 11, comparing the face frames in the left and right images, there is a high probability that the face frame in the left image will move outside the image area before the face frame in the right image, resulting in the face frame not being detected. In such cases, the area defined in the image region should be changed. The following explanation will use the case where the left camera 11 is positioned closer to the entrance / exit E1 than the right camera 11 as an example.

[0176] Figure 13 shows an example of a defined area for a difference in camera placement. Figure 13 shows images taken by the right camera 11 and the left camera 11 at two different time points, the face frame detected in each image, and the defined area in each image.

[0177] Each image in Figure 13 includes person A entering gate 10 and person B entering gate 10 from behind person A. As described above, if the left camera 11 is positioned closer to the entrance E1 than the right camera 11, the face frame of person A in the left image will move out of the image area before the face frame of person A in the right image. For example, in the example in Figure 13, since the face frame of person A in the left image moves out of the image area, there is a possibility of misidentifying the face frame of person B in the left image and the face frame of person A in the right image as belonging to the same person. Therefore, as shown in Figure 13, the PFFA in the image from the right camera 11 is set to be narrower horizontally than the PFFA in the image from the left camera 11. This setting allows, for example, the timing at which the center of the face frame moves out of the PFFA to be synchronized between the left and right images.

[0178] <Example of control when an image cannot be acquired from one camera 1> For example, in the two-camera system described above, there may be cases where a face frame cannot be detected in the image captured by one of the two cameras. For example, cases where a face frame cannot be detected may include cases where the face frame detection process fails in the image captured by one of the cameras, or cases where there is a problem with the shooting of one of the cameras (e.g., camera malfunction, temporary malfunction). Below, we will explain an example in which, when a face frame cannot be detected in the image captured by one of the two cameras, passage management is performed based on the face frame detected in the image captured by the other camera. Note that a processing method performed using an image captured by one camera is sometimes referred to as a single-camera system.

[0179] For example, if face frame detection fails in the left image captured by the left camera, but succeeds in the right image captured by the right camera, the person's position may be estimated based on the amount of movement of the face frame in the right image. In this person position estimation process, for example, the face frame of a person who newly appears passing through gate 10 in the time series is detected. An example of detecting the face frame of a new person is described below.

[0180] Figure 14 is a flowchart illustrating a new face frame detection method using a single camera. The flow shown in Figure 14 may be executed, for example, when image information is acquired from either the right camera 11 or the left camera 11, but not from the other, or when either the right camera 11 or the left camera 11 fails to capture an image.

[0181] The facial recognition unit 13 acquires information from the image captured by the camera 11 and performs a process to detect a face frame from the image (S401).

[0182] The person position estimation unit 14 determines whether the size of the face frame is greater than or equal to the specified size (S402).

[0183] If the face frame size is greater than or equal to the specified size (YES in S402), the person position estimation unit 14 performs a person identification process (S403). For example, the person position estimation unit 14 determines whether the person in the detected face frame is the same person as the person in the face frame shown in the past face frame information list. For example, this determination may be performed by comparing the respective feature points, similar to the face recognition process.

[0184] Based on the results of the person identification process, the person location estimation unit 14 determines whether or not the individuals are the same person (S404).

[0185] If they are not the same person (YES in S404), the person position estimation unit 14 determines whether the center position of the detected face frame is included in the new appearance zone (for example, FFFA) (S405).

[0186] If the center position of the detected face frame is included in the newly appearing zone (YES in S405), the person position estimation unit 14 determines that the detected face frame is the face frame of a newly appearing person and registers the information of that face frame (S406). The flow then ends.

[0187] In S406, for example, if the person corresponding to the detected face frame is a different person from the person corresponding to the face frames detected so far, and the face frame is within a zone where it can be determined that a person has newly appeared, then it is recognized as the face frame of a new person. For example, the information may be registered in the face frame tracking management table. In this case, a face matching request may be notified.

[0188] If the face frame size is not equal to or greater than the specified size (NO in S402), or if the center position of the detected face frame is not included in the newly appearing zone (NO in S405), the person position estimation unit 14 determines that the detected face frame is a face frame outside the scope of management (S407). The flow then ends.

[0189] If the individuals are the same person (NO in S404), the person location estimation unit 14 performs the tracking process for the same person (S408). Then the flow ends.

[0190] In the above flow, it was determined that an initial face frame for a new person had appeared when the center point of the face frame was in the new appearance zone. However, for example, if a face frame is detected in the new appearance zone a predetermined number of times consecutively, it may be determined that the person corresponding to the detected face frame is a new person. By making a determination based on consecutive detections, it is possible to avoid registering a face frame that will disappear next after only one appearance.

[0191] The process for determining whether two images belong to the same person is not particularly limited. For example, it may be determined whether two images belong to the same person based on the amount of movement of the face frame. For instance, a predetermined range of movement set based on the shooting interval (the range in which a person can move relative to the shooting interval (e.g., 60 msec)) may be compared with the amount of movement of the face frame between two images. If the amount of movement of the face frame falls within the predetermined range, it may be determined that the two images belong to the same person.

[0192] <Example 2 of control when an image cannot be acquired from one camera> In the example described above, we showed a method for detecting the face frame of a newly appearing person in a dual-camera system (a system that processes images captured by two cameras) when a face frame is not detected in the image captured by one of the two cameras. Below, we will explain an example in a dual-camera system that determines whether a person passing through gate 10 has crossed the charge line when a face frame is not detected in the image captured by one of the two cameras.

[0193] Figure 15 is a flowchart showing an example of determining whether a charge line has been crossed using a single camera. The flow shown in Figure 15 may be executed, for example, when information from an image captured by either the right camera 11 or the left camera 11 is acquired, but information from the other is not acquired, or when either the right camera 11 or the left camera 11 is unable to capture an image.

[0194] The person position estimation unit 14 calculates the center position of the face frame (S501).

[0195] The person position estimation unit 14 determines whether the center position of the face frame is within the billing zone (e.g., PFFA) (S502).

[0196] If the center position of the face frame is not within the billing zone (e.g., PFFA) (NO in S502), the person position estimation unit 14 may terminate processing.

[0197] If the center position of the face frame is within the billing zone (YES in S502), the person position estimation unit 14 determines whether the initial appearance of the face frame is in a zone different from the new appearance zone (e.g., FFFA) (S503). For example, it refers to the information of the first face frame detected among the face frame information that indicates that it is the same person as a face frame whose center position is within the billing zone, and determines whether the zone in which the referenced face frame was detected is a new appearance zone.

[0198] If the initial appearance of the central position is in a zone different from the new appearance zone (YES in S503), for example, if the person in the face frame is not a person attempting to pass through gate 10, but rather a person passing around gate 10 (for example, another gate adjacent to gate 10), then no charge will be made to that person, and the flow will terminate. In this case, a determination result indicating that no charge will be made to that person may be recorded.

[0199] If the initial appearance of the center position is not in a zone other than the new appearance zone (for example, FFFA) (NO in S503), the person position estimation unit 14 determines whether the size of the face frame is greater than or equal to the face frame size threshold defined in the billing zone (S504).

[0200] If the size of the face frame is not greater than or equal to the face frame size defined in the billing zone (NO in S504), the person position estimation unit 14 may terminate processing.

[0201] If the size of the face frame is greater than or equal to the face frame size defined in the billing zone (YES in S504), the person position estimation unit 14 determines whether the size of the face frame is increasing (S505). For example, the person position estimation unit 14 may refer to face frame information indicating that it is the same person as the face frame being determined, and determine whether the size of the face frame is increasing as the time of the image being taken progresses.

[0202] If the size of the face frame is not increasing (NO in S505), for example, if the size of the face frame has not changed (the person has not moved for a certain period of time), or if the size of the face frame has decreased (the person has moved away from the billing zone), the person position estimation unit 14 may terminate because no charge will be made for that person. In this case, a determination result indicating that no charge was made may be recorded. If the same determination result continues for a certain period of time for that person, it is equivalent to the person remaining inside the gate 10 for a certain period of time, and a warning may be issued from the gate 10.

[0203] If the size of the face frame is increasing (YES in S505), the person position estimation unit 14 confirms that the person has entered the billing zone (e.g., zone C) at gate 10 (S506). The person position estimation unit 14 notifies the server (e.g., the face recognition server 16 and / or the passage history management server 17 in Figure 2A) of the confirmed result, and the person is charged. The flow then ends.

[0204] As described above, the system can determine whether a person has entered the charging zone using an image captured by one of the cameras 11. Therefore, even if one of the cameras 11 malfunctions, or if one of the cameras 11 fails to capture an image due to an obstruction, the system can still properly charge the person who has passed through the gate 10.

[0205] In the embodiments described above, it was assumed that the FFFA, MFFA, and PFFA zones do not overlap with each other, but this is not the case. Two or more zones may overlap with each other. In this case, a threshold for the size of the face frame may be set for each zone, and if the center position of the face frame belongs to a zone and the size of the face frame exceeds the threshold of the zone to which it belongs, it may be determined that the face frame is located in the zone to which it belongs. By performing this determination in the order of PFFA, which has the largest face frame size, then MFFA, and then FFFA, it is possible to appropriately determine which zone the face frame belongs to even if the center position of the face frame lies in an overlapping area of ​​the zones. For example, if the center position of the face frame belongs to both PFFA and MFFA (it lies in the zone where PFFA and MFFA overlap), and the size of the face frame exceeds the threshold associated with PFFA, the determination of whether the size of the face frame exceeds the threshold associated with MFFA is omitted, and the face frame is determined to belong to PFFA. Furthermore, if the size of the face frame is less than or equal to the threshold associated with PFFA, and the size of the face frame exceeds the threshold associated with MFFA, then it is determined that the face frame belongs to MFFA. Similarly, if the center position of the face frame belongs to both MFFA and FFFA, it is first determined whether the face frame belongs to MFFA, and only if it is determined that the face frame does not belong to MFFA is it determined whether the face frame belongs to FFFA. Generally, the size of the face frame is largest in PFFA and smallest in FFFA, so the thresholds associated with each zone may be values ​​corresponding to this size relationship. However, other values ​​may be used, such as setting the thresholds associated with all zones to be the same.

[0206] Furthermore, in the above-described embodiment, in the dual-camera system, if a face frame is not detected in the image captured by one of the two cameras, a single-camera system is used to determine if the person has crossed the charge line. However, the system is not limited to this. For example, even if the dual-camera system is functioning correctly, the single-camera system may be used. In this case, if a charge is made in both systems when a person crosses the charge line, and no charge is made in either system when a person does not cross the charge line, the charge determination can be made more precisely. Also, if a charge is made when a person crosses the charge line in either system, the charge can be made reliably even if the determination in one of the systems is incorrect. Alternatively, only the single-camera system may be used.

[0207] The above-described embodiment described a system for managing the passage of persons passing through gate 10, but this disclosure is not limited thereto. For example, this disclosure may apply to cases where there are no side walls of the passageway and no restricting parts (e.g., doors) that restrict the passage of persons. For example, this disclosure may apply to a travel path from one zone to another zone where entry of persons is permitted according to an authentication process. In this case, a camera that photographs the faces of persons passing through the travel path may be installed, for example, on a support part provided on the travel path. Furthermore, the above-described embodiment showed an example of charging persons who pass through gate 10, but this disclosure is not limited thereto. For example, this disclosure may apply to recording and / or managing passage without charge. Also, for example, in cases such as a ticket gate at a train station, where entry is recorded upon entry and charge is made based on the entry record upon exit, this disclosure may apply to both entry and exit.

[0208] Furthermore, while the embodiment used right and left images taken at the same time for the determination, it is not limited to this. For example, right and left images taken at synchronized times, even if not at the same time, may be used. In this case, by using the right and left images taken at the closest possible timings to each other to determine whether the distance between the face frames falls below a threshold, the same determination result as in the embodiment described above can be obtained.

[0209] Furthermore, although this embodiment shows an example where the object of authentication is a person, this disclosure is not limited to this. For example, it may be applied to animals, moving objects such as vehicles, etc. Also, although this embodiment shows an example of facial recognition, this disclosure is not limited to this. For example, this disclosure may be applied to authentication using an ID card that indicates the right to pass through a gate, and to other authentication methods such as biometric authentication.

[0210] Furthermore, facial recognition may be used in combination with other authentication methods. Even if passage is not permitted by facial recognition as disclosed in the above-described embodiments, passage may be exceptionally permitted if ID card information is entered.

[0211] Furthermore, although the above-described embodiment explains that the camera 11 is used for both authentication and tracking, this disclosure is not limited to this. For example, the image captured by the camera 11 may be used for person tracking (location estimation) and not for facial recognition. For example, if authentication using an ID card indicating the right to pass through the gate, and other authentication methods such as biometric authentication are used instead of facial recognition, the image captured by the camera 11 may be used for person tracking (location estimation) and not for facial recognition.

[0212] Furthermore, in the above-described embodiment, the camera 11 is not limited to being mounted on the side wall V. For example, the camera 11 may be attached to a support provided on the gate 10. The support may be, for example, a pole extending vertically from the gate 10, or an arch-shaped member provided to cover the side wall of the gate 10. Also, the gate 10 is not limited to an example where a passage is formed by two side walls V. For example, there may be no side walls V. If there are no side walls V, the camera 11 may be positioned at a desired location by a member such as a pole. Even in a configuration without side walls V, person tracking is performed based on the positional relationship of the face frame detected from the image captured by the camera 11, similar to the above-described embodiment. Furthermore, person tracking using a camera that takes pictures from another angle may be used in combination with the configuration shown in the above-described embodiment. For example, person tracking using a ceiling camera may be used in combination. By using person tracking in combination, the accuracy of estimating the position and direction of movement of a person can be improved. For example, a surveillance camera installed in a station may be used as the ceiling camera.

[0213] Furthermore, in the embodiment described above, the success or failure of the facial recognition process by the facial recognition function unit 13 and the success or failure of the pass-through of the exit line determination by the pass-through management function unit 15 may be output by voice and / or image. In this case, different voices and / or images may be output for the success or failure of the facial recognition process and the success or failure of the pass-through of the exit line determination. By doing so, a person attempting to pass through gate 10 can be made aware that both facial recognition and pass-through of the payment line are being determined. In addition, if a person fails to pass through gate 10, it is possible to distinguish and notify whether the failure occurred at the stage of obtaining permission to pass through gate 10 (success or failure of the facial recognition process) or at the stage of confirming passage through the gate (success or failure of passage).

[0214] Furthermore, while the above-described embodiment used a door as a means to restrict passage through gate 10, passage may be restricted directly or indirectly by other means. For example, an alarm may sound or a warning light may be activated to indicate to a person attempting to pass through gate 10 that passage is restricted. Alternatively, a notification may be sent to a terminal or other device owned by an employee near gate 10, allowing that employee to restrict passage.

[0215] Furthermore, depending on the congestion level, it may be decided whether or not to implement control measures to prevent passage, or the means of restricting passage may be switched. For example, in environments where it would be dangerous to prevent or restrict the passage of people, such as when a large number of people are entering or exiting, passage through gate 10 may not be prevented, and information indicating that unauthorized passage occurred may be recorded. In this case, the facial image or facial recognition result of the person who passed through illegally may be recorded in association with the information indicating that unauthorized passage occurred. This makes it possible to track down the person who passed through illegally and claim compensation for the right of passage.

[0216] Furthermore, in the embodiment described above, the passage management system 1 managed both entry into and exit from facilities such as airports, train stations, and event venues. However, at either the entrance or exit, it is also possible to manage only one of the two, and not the other.

[0217] This disclosure can be implemented using software, hardware, or software integrated with hardware.

[0218] Each functional block used in the description of the above embodiments may be implemented partially or entirely as an integrated circuit (LSI), and each process described in the above embodiments may be controlled partially or entirely by a single LSI or a combination of LSIs. An LSI may consist of individual chips, or it may consist of a single chip that includes some or all of the functional blocks. An LSI may have data inputs and outputs. Depending on the degree of integration, LSIs may be referred to as ICs, system LSIs, super LSIs, or ultra LSIs.

[0219] The method of integration is not limited to LSIs; it may also be implemented using dedicated circuits, general-purpose processors, or dedicated processors. Furthermore, FPGAs (Field Programmable Gate Arrays) that can be programmed after LSI manufacturing, or reconfigurable processors that allow for the reconfiguration of the connections and settings of circuit cells within the LSI, may also be used. This disclosure may be implemented as digital or analog processing.

[0220] Furthermore, if advancements in semiconductor technology or related technologies lead to the emergence of integrated circuit technologies that replace LSIs, then naturally, these technologies can be used to integrate functional blocks. The application of biotechnology, for example, is a possible possibility.

[0221] This disclosure is applicable to all types of devices, systems, and equipment having communication capabilities (collectively referred to as communication equipment). Communication equipment may include a radio transceiver and a processing / control circuit. A radio transceiver may include a receiver and a transmitter, or both as functions. A radio transceiver (transmitter, receiver) may include an RF (Radio Frequency) module and one or more antennas. The RF module may include an amplifier, an RF modulator / demodulator, or similar. Non-exclusive examples of communication devices include telephones (mobile phones, smartphones, etc.), tablets, personal computers (PCs) (laptops, desktops, notebooks, etc.), cameras (digital still / video cameras, etc.), digital players (digital audio / video players, etc.), wearable devices (wearable cameras, smartwatches, tracking devices, etc.), game consoles, digital book readers, telehealth / telemedicine devices, vehicles or mobile transport with communication capabilities (cars, airplanes, ships, etc.), and combinations of the above-mentioned devices.

[0222] Communication devices are not limited to portable or movable devices, but also include all kinds of non-portable or fixed devices, devices, and systems, such as smart home devices (appliances, lighting equipment, smart meters or measuring instruments, control panels, etc.), vending machines, and any other "things" that may exist on an IoT (Internet of Things) network.

[0223] Furthermore, in recent years, Cyber-Physical Systems (CPS), a new concept in IoT (Internet of Things) technology that creates new added value through information linkage between the physical and cyber spaces, has been attracting attention. This CPS concept can also be adopted in the above-described embodiment.

[0224] In other words, as a basic configuration of CPS, for example, edge servers located in physical space and cloud servers located in cyberspace are connected via a network, and processing can be distributed and performed by the processors installed on both servers. Here, it is preferable that each processing data generated on the edge server or cloud server is generated on a standardized platform, and by using such a standardized platform, it is possible to improve efficiency when building systems that include various diverse groups of sensors and IoT application software.

[0225] Communication includes data communication via cellular systems, wireless LAN systems, and communication satellite systems, as well as data communication using combinations of these.

[0226] Furthermore, the communication device also includes devices such as controllers and sensors that are connected to or linked to a communication device that performs the communication functions described in this disclosure. For example, this includes controllers and sensors that generate control signals and data signals used by the communication device that performs the communication functions of the communication device.

[0227] Furthermore, communication equipment includes infrastructure facilities such as base stations, access points, and any other devices, devices, and systems that communicate with or control the aforementioned non-limited types of equipment.

[0228] Although various embodiments have been described above with reference to the drawings, it goes without saying that this disclosure is not limited to such examples. It is clear to those skilled in the art that various modifications or alterations can be conceived within the scope of the claims, and these will naturally also fall within the technical scope of this disclosure. Furthermore, the components in the above embodiments may be combined in any way without departing from the spirit of the disclosure.

[0229] The specific examples of this disclosure have been described in detail above, but these are merely illustrative and do not limit the scope of the claims. The technologies described in the claims include various modifications and changes to the specific examples described above. [Industrial applicability]

[0230] One embodiment of the present disclosure is suitable for a facial recognition system. [Explanation of Symbols]

[0231] 1. Passenger Management System 10 Gates 11 Cameras 13. Facial Recognition Function Unit 131 Camera Control Unit 132 Face Recognition Processing Unit 14 Person position estimation section 141 Person Tracking Processing Unit 15 Passage management function section 151 Passage Management State Transition Processing Unit 152 History Management Department 153 History DB 16. Face Recognition Server 17. Passage History Management Server

Claims

1. A detection unit that detects a first facial image region included in a first image taken of a person entering a gate from a first direction, and a second facial image region included in a second image taken of the person from a second direction different from the first direction, An estimation unit determines, based on the change in distance between a representative point of the first face image region and a representative point of the second face image region, that the person has moved beyond the position defined at the gate, if the difference between the time the first image was taken and the time the second image was taken is within a predetermined tolerance range. A face matching processing unit that matches the face of the person before the person moves beyond the predetermined position, Equipped with, The system outputs different audio or images depending on whether the face of the person has been successfully matched or whether the person has moved beyond the predetermined position. Information processing device.

2. The detection unit sets an area in the first image where it is predicted that a representative point of the first face image region will first appear in response to the person entering the gate. If the estimation unit determines that the area contains multiple representative points of the first facial image region, it stops estimating the position of the person. The information processing apparatus according to claim 1.

3. The representative point of the first face image region and the representative point of the second face image region are the center of the first face image region and the center of the second face image region, respectively. The information processing apparatus according to claim 1.

4. An authentication device that performs authentication processing of a person using at least one of a first image of a person entering a gate taken from a first direction and a second image of the person taken from a second direction different from the first direction, The first face image region included in the first image and the second face image region included in the second image are detected. If the difference between the time the first image was taken and the time the second image was taken is within a predetermined tolerance range, it is determined that the person has crossed the position defined at the gate based on the change in distance between the representative point of the first face image region and the representative point of the second face image region. The face of the person is verified before the person moves beyond the predetermined position. An information processing device that outputs different sounds or images depending on whether it is determined that the person's face has been successfully matched or that the person has moved beyond the predetermined position, An information processing system equipped with the following features.

5. Information processing device, The system detects a first facial image region included in a first image taken of a person entering the gate from a first direction, and a second facial image region included in a second image taken of the person from a second direction different from the first direction. If the difference between the time the first image was taken and the time the second image was taken is within a predetermined tolerance range, it is determined that the person has crossed the position defined at the gate based on the change in distance between the representative point of the first face image region and the representative point of the second face image region. The face of the person is verified before the person moves beyond the predetermined position. The system outputs different audio or images depending on whether the face of the person has been successfully matched or whether the person has moved beyond the predetermined position. Estimation method.

Images