Information processing device, information processing system, and estimation method

Abstract

To provide an information processing device capable of improving the estimation accuracy of estimating the position of a target which is about to pass through a specific area, an information processing system, and an estimation method.SOLUTION: An information processing device comprises a detection unit and an estimation unit. The detection unit detects a first facial image region included in a first image capturing a person entering a gate from a first direction, and a second facial image region included in a second image capturing a person from a second direction different from the first direction. When the difference between a capture time at which the first image has been captured and a capture time at which the second image has been captured is within a predetermined allowable range, the estimation unit estimates a position of the person at the gate on the basis of a change in the distance between a representative point of the first facial image region and a representative point of the second facial image region.SELECTED DRAWING: Figure 2A

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing system, and an estimation method. [Background technology]

[0002] There is known technology for managing the entry and exit of people passing through gates installed at stations, airports, etc. Patent Document 1 describes a device that, when a person obtains permission to pass through a gate using a wireless card and enters the gate from the entrance, tracks whether the person has passed through the gate (whether the person has returned to the entrance of the gate) based on changes in the position of the wireless card. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 09-330440 Summary of the Invention [Problem to be solved by the invention]

[0004] At gates that control entry and exit, it is desirable to manage which people have passed through the gate. Hereinafter, this management may be abbreviated as "passage management" or "tracking management." There is room for improvement in the accuracy of estimating people's positions in tracking management.

[0005] Non-limiting examples of the present disclosure contribute to providing an information processing device, an information processing system, and an estimation method that can improve the accuracy of estimating the position of an object that is about to pass through a specific area. [Means for solving the problem]

[0006] An information processing device according to one embodiment of the present disclosure includes a detection unit that detects a first facial image area included in a first image obtained by photographing a person entering a gate from a first direction and a second facial image area included in a second image obtained by photographing the person from a second direction different from the first direction, and an estimation unit that, when a difference between the photographing time when the first image was photographed and the photographing time when the second image was photographed is within a predetermined tolerance range, estimates the position of the person at the gate based on a change in distance between a representative point of the first facial image area and a representative point of the second facial image area.

[0007] An information processing system according to one embodiment of the present disclosure includes an authentication device that performs authentication processing of a person entering a gate using at least one of a first image of the person photographed from a first direction and a second image of the person photographed from a second direction different from the first direction; and an information processing device that detects a first facial image area included in the first image and a second facial image area included in the second image, and, when the difference between the photographing time of the first image and the photographing time of the second image is within a predetermined tolerance range, estimates the position of the person at the gate based on a change in distance between a representative point of the first facial image area and a representative point of the second facial image area.

[0008] In an estimation method according to one embodiment of the present disclosure, an information processing device detects a first facial image area included in a first image obtained by photographing a person entering a gate from a first direction and a second facial image area included in a second image obtained by photographing the person from a second direction different from the first direction, and, if the difference between the photographing time of the first image and the photographing time of the second image is within a predetermined tolerance range, estimates the position of the person at the gate based on a change in the distance between a representative point of the first facial image area and a representative point of the second facial image area.

[0009] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. [Effects of the Invention]

[0010] Non-limiting examples of the present disclosure can improve the accuracy of estimating the position of an object passing through a particular area.

[0011] Further advantages and benefits of an embodiment of the present disclosure will become apparent from the specification and drawings. Such advantages and / or benefits may be provided by some of the embodiments and features described in the specification and drawings, respectively, but not necessarily all of them may be provided to obtain one or more identical features. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of a gate according to an embodiment. [Figure 2A] FIG. 1 is a diagram illustrating an example of the concept of the configuration of a currency management system according to an embodiment. [Figure 2B] FIG. 1 is a block diagram illustrating an example of the configuration of a currency management system according to an embodiment. [Figure 3A] Diagram showing an example of zones defined by a gate [Figure 3B] Diagram showing an example of zones defined by a gate [Figure 4] FIG. 10 is a diagram showing an example of face frame detection in one embodiment. [Figure 5A] FIG. 10 is a diagram showing an example of detecting a person's position in an embodiment; [Figure 5B] FIG. 10 is a diagram showing an example of the relationship between the size and center of a face frame relative to the position of a person. [Figure 6A] FIG. 10 is a diagram showing a first example of a transition in the magnitude of a position difference; [Figure 6B] FIG. 10 is a diagram showing a second example of a transition in the magnitude of the position difference; [Figure 7]FIG. 10 is a diagram showing an example of the flow of person tracking processing based on a face frame in one embodiment. [Figure 8] FIG. 1 is a diagram showing an example of an area defined in an image region; [Figure 9] Flowchart showing an example of the flow of currency management [Figure 10] A diagram showing an example of face frame timeout processing [Figure 11] Flowchart showing an example of FFFA multiple face frame elimination processing [Figure 12] Flowchart showing an example of a two-lens face frame position estimation process [Figure 13] A diagram showing examples of defined areas for different camera placements [Figure 14] Flowchart showing a new example of face frame detection using a single camera [Figure 15] Flowchart showing an example of determining whether a vehicle has crossed the charging line using one camera DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functions are designated by the same reference numerals, and redundant description will be omitted.

[0014] (One embodiment) <Knowledge that led to this disclosure> At gates installed at facilities such as train stations and airports to manage access to and from the facility, the use of passage management to accurately manage who has passed through the gate is being considered. If passage management is insufficient, for example, a person who entered through the gate entrance but turned back to the gate entrance instead of going to the gate exit may be mistakenly determined to have passed through, or a person who actually passed through may be mistakenly recognized as not having passed through. Such errors can lead to incorrect billing in services that charge fees to people who pass through a gate, such as ticket gates at train stations.

[0015] To implement passage management, for example, an authentication process for authenticating a person attempting to pass through (which may include, for example, a process for determining that the person cannot be authenticated) and a tracking process for recording the history of the person's movements are carried out. These processes are preferably carried out early in order to ensure time for processing such as recording the passage of a person or regulating the movement of a person, such as opening and closing doors.

[0016] For example, it is being considered to install a camera above the person and the gate (e.g., on the ceiling), photograph the person and the gate with the camera on the ceiling, and then track the person photographed by analyzing the captured image.

[0017] However, when installing cameras on the ceiling, the location where the cameras can be installed may be limited depending on the structure or environment of the installation location. Even if installation is possible, if large-scale construction work is required, the installation costs will increase. This can make it difficult to introduce a passage management system using cameras installed on the ceiling.

[0018] For example, it is conceivable to provide a gate with an arch-shaped or pole-shaped support that extends above the gate and install a camera on the support, but a gate with such a support is larger in height than a gate without a support, which may limit the location where the gate can be installed. Also, from a design perspective, providing a support on the gate may not be desirable.

[0019] In this embodiment, a pass management process including authentication and tracking is carried out by performing tracking using an image captured by a camera used for facial authentication of people attempting to pass through a gate. By using the camera used for authentication also for tracking, there is no need to install a separate device for tracking (for example, a camera on the ceiling, etc.). This makes it possible to suppress increases in the introduction costs of pass management. Furthermore, compared to installing equipment such as a dedicated camera for tracking, restrictions on installation location are relaxed, which increases the degree of freedom in installation location and makes it easier to introduce a pass management system.

[0020] <Gate configuration example> FIG. 1 is a diagram illustrating an example of a gate 10 according to the present embodiment. FIG. 1 is a diagram illustrating the gate 10 as viewed from above, illustrating a situation in which a person h enters through an entrance / exit E1 of the gate 10 and exits through an entrance / exit E2 of the gate 10. Note that, with respect to the gate 10 shown in FIG. 1, a person may enter through an entrance / exit E2 and exit through an entrance / exit E1. In other words, the gate 10 allows people to pass through in both directions.

[0021] Gate 10 has, for example, side walls V facing each other, and a passage L is formed between the side walls V to guide people passing through gate 10. At the top of one of the side walls V, which is about 1 m high, cameras 11 are provided, for example, at two positions closer to entrances E1 and E2 than the center of side wall V, so that a total of four cameras 11 (11-R1, 11-R2, 11-L1, 11-L2) are provided on the two side walls V.

[0022] Cameras 11-R1 and 11-L1 are installed, for example, on a side wall V at a position closer to entrance / exit E2 than the center of gate 10, and are used to photograph people entering gate 10 from entrance / exit E1 on the opposite side and passing through to entrance / exit E2.

[0023] On the other hand, cameras 11-R2 and 11-L2 are installed, for example, on the side wall V at a position closer to entrance / exit E1 than the center of gate 10, and are used to photograph people entering gate 10 from entrance / exit E2 on the opposite side and passing through to entrance / exit E1.

[0024] For example, camera 11-R1 is installed in a position where it can photograph a person entering through entrance E1 from the front right of the person. Camera 11-L1 is installed in a position where it can photograph a person entering through entrance E1 from the front left of the person.

[0025] For example, camera 11-R2 is installed in a position where it can photograph a person from the front right of the person entering through entrance E2 opposite entrance E1. Camera 11-L2 is installed in a position where it can photograph a person from the front left of the person entering through entrance E2 opposite entrance E1.

[0026] Therefore, a person entering gate 10 from entrance / exit E1 and passing through entrance / exit E2 is photographed from two directions (e.g., left and right) by two cameras 11-R1 and 11-L1 installed at positions separated from each other on the top of two side walls V, separated by an aisle L.

[0027] On the other hand, a person passing through gate 10 in the opposite direction, i.e., a person entering gate 10 from entrance / exit E2 and passing through to entrance / exit E1, is photographed from two directions (e.g., left and right) by two cameras 11-R2 and 11-L2 installed at positions separated from each other on the top of two side walls V, separated by an aisle L.

[0028] 1 illustrates an example configuration in which a person can enter through both entrance E1 and entrance E2 of gate 10, but the present disclosure is not limited to this. For example, gate 10 may be configured so that a person can enter through one entrance (e.g., entrance E1) but cannot enter through the other entrance (e.g., entrance E2). If gate 10 is configured so that it does not allow a person to enter through entrance E2, cameras 11-R2 and 11-L2 may not be provided. If gate 10 is configured so that it does not allow a person to enter through entrance E1, cameras 11-R1 and 11-L1 may not be provided.

[0029] The following describes, as an example, a case in which cameras 11-R1 and 11-L1 are used to manage the passage of people entering through entrance E1 and passing through entrance E2 at gate 10 shown in Fig. 1. For convenience, cameras 11-R1 and 11-L1 may be collectively referred to as camera 11.

[0030] Furthermore, camera 11-R1 may be referred to as right camera 11, and an image captured by right camera 11 may be referred to as a "right image." Similarly, camera 11-L1 may be referred to as left camera 11, and an image captured by left camera 11 may be referred to as a "left image."

[0031] Furthermore, a person entering the gate 10 corresponds to a person who is a target of processing including face recognition processing. Hereinafter, a person who is a target of processing will be referred to as a "target person."

[0032] 1 is an example, and the present disclosure is not limited thereto. For example, the gate 10 may be provided with five or more cameras 11, or three or fewer cameras 11. By varying the shooting direction and / or angle of the cameras 11, it is possible to capture images of a wider range of a person's face.

[0033] When two cameras 11 are provided, one may be provided in a position where it photographs a person entering gate 10 from a first direction, and the other may be provided in a position where it photographs a person entering gate 10 from a second direction. Camera 11 may be provided in a position where it photographs the face of a person entering gate 10 from the front, or may be provided in a position where it photographs at least a part of the face (for example, the right half or left half of the face). For example, camera 11 may be provided in a position where it photographs an image where a face frame can be detected by face frame detection, which will be described later.

[0034] The multiple cameras 11 do not have to be identical to each other. For example, the multiple cameras 11 may be configured to capture images with different resolutions, angles of view, and image quality. Furthermore, the installation positions and / or capture directions of the cameras 11 may be fixed or adjustable.

[0035] <System configuration> Fig. 2A is a diagram showing an example of the conceptual configuration of a passage management system according to this embodiment. Fig. 2B is a block diagram showing an example of the configuration of a passage management system according to this embodiment. The passage management system 1 according to this embodiment is a system that manages the passage of people through gates 10 (e.g., entrance gates, ticket gates, etc.) installed at the entrances and exits of facilities such as airports, train stations, and event venues.

[0036] In the passage management system 1 according to this embodiment, for example, management of entrance and exit of users who use a facility is performed by facial recognition. For example, when a user passes through gate 10 to enter the facility, facial recognition determines whether the user is a person permitted to enter the facility. Also, when a user passes through the gate to exit the facility, facial recognition determines which user is exiting the facility. Note that "facial recognition" may be considered a concept included in "matching using a facial image."

[0037] 1, the passage management system 1 includes, for example, a gate 10, cameras 11 (right camera 11 and left camera 11), a face authentication function unit 13, a person position estimation unit 14, a passage management function unit 15, a face authentication server 16, and a passage history management server 17. Note that the passage management system 1 may have one or more gates 10.

[0038] Gate 10 is installed in facilities such as airports, train stations, and event venues. Users authorized to use the facility pass through gate 10 when entering and / or leaving the facility. Gate 10 also controls the passage of people who are not authorized to enter the facility.

[0039] 1, camera 11 is provided, for example, on a side wall V of gate 10. Camera 11 captures a photographic range including the face of a person passing through gate 10 and, if a person is about to pass through gate 10, the face of the person. For example, the photographic range of camera 11 is a range that can capture the frontal view of a person's face.

[0040] The image captured by the camera 11 may be used in a person detection process (or a person tracking process) described later, or in a face authentication process described later.

[0041] The face authentication function unit 13 performs face authentication processing on the image. For example, the face authentication function unit 13 includes a camera control unit 131 and a face matching processing unit 132.

[0042] The camera control unit 131 controls, for example, the timing of shooting by the camera 11. For example, the camera 11 shoots at a speed of about 5 fps under the control of the camera control unit 131. Furthermore, the right camera 11 and the left camera 11 may shoot simultaneously or with a difference in shooting timing within an allowable range under the control of the camera control unit 131. In other words, the shooting timing of the right camera 11 and the shooting timing of the left camera 11 are synchronized under the control of the camera control unit 131, for example.

[0043] The camera control unit 131 detects a face frame, for example, from an image (right image and / or left image) captured by the camera 11. The method for detecting the face frame is not particularly limited, but may be, for example, a method of detecting features (eyes, nose, and mouth) included in a face from an image, and detecting a frame surrounding the face area (face frame) by detecting a boundary between the face area and an area outside the face based on information on the positions and colors of the detected features. For example, when a face frame is detected, the camera control unit 131 outputs information about the detected face frame (face frame information) and the captured image to the face matching processing unit 132.

[0044] The face matching processing unit 132 cuts out a face area included in the image based on, for example, face frame information, and notifies a face matching request including information on the cut-out face area to the face authentication server 16. The information on the face area may be, for example, an image of the face area, or information indicating feature points extracted from the image of the face area.

[0045] For example, facial images of persons who are permitted to pass through the gate 10 are registered in the facial authentication server 16. The facial images registered in the facial authentication server 16 may be referred to as registered facial images. The registered facial images may be associated with information that can uniquely identify or specify a person, such as the ID of the registered person. The registered facial images may also be information that indicates feature points extracted from an image, for example.

[0046] For example, when the face authentication server 16 receives a face matching request from the face matching processing unit 132, it determines whether or not the face of the same person as the face in the face area included in the face matching request is included in the registered face image. The face authentication server 16 notifies the face matching processing unit 132 of the face matching result including the determination result, for example. Note that the face matching result may include, for example, information indicating whether or not the face of the same person as the face in the face area is included in the registered face image (for example, a flag indicating "OK" or "NG"), and, if the face of the same person as the face in the face area is included in the registered face image, information (for example, an ID) of the person associated with the registered face image.

[0047] Matching involves, for example, comparing a registered face image with a face image of a person passing through gate 10 to determine whether a pre-registered registered face image matches the face image of a person passing through gate 10, or whether a pre-registered registered face image and the face image of a person passing through gate 10 are the face images of the same person.

[0048] On the other hand, authentication means proving to an outside party (e.g., gate 10) that a person whose facial image matches a pre-registered facial image is the person in question (in other words, that the person is someone who should be allowed to pass through gate 10).

[0049] However, in this disclosure, "verification" and "authentication" may be used interchangeably.

[0050] For example, the matching process is a process of identifying the identity of a face in image data by comparing feature points of a pre-registered face image with feature points extracted from a detected face area. This matching process may be performed using, for example, a machine learning technique. Furthermore, the matching process may be performed, for example, in the face authentication server 16, but may also be performed in another device such as the gate 10, or may be distributed among multiple devices.

[0051] The face matching processing unit 132 outputs, for example, information including the matching processing result to the passage management function unit 15. The matching processing result may include, for example, information about the registered face image and a matching score. Furthermore, the information output from the face matching processing unit 132 may include, for example, face frame detection information and the shooting time of the face camera image in which the face frame was detected.

[0052] The person position estimation unit 14 performs a person tracking process based on, for example, face frame information. The person position estimation unit 14 includes, for example, a person tracking processing unit 141.

[0053] The person tracking processing unit 141 estimates the position of a person relative to the gate 10, for example, based on face frame information. Then, the person tracking processing unit 141 determines an event that occurs for the person by tracking the estimated position of the person. For example, events that occur for a person include the appearance of a new person, the tracking of a person, and the disappearance of a person. The person tracking processing unit 141 tracks the person by, for example, determining an event based on the person's position and associating the determined event with information such as the person's position and the time of detection.

[0054] The person tracking processing unit 141 outputs, for example, information relating to person tracking to the passage management function unit 15. For example, the information relating to person tracking includes information on the position of the person and information such as the time of detection.

[0055] The passage management function unit 15 manages the status of people located around the gate 10, for example, by associating information output from the face authentication function unit 13 with information output from the person position estimation unit 14. People located around the gate 10 include, for example, people passing through the gate 10, people attempting to pass through, and people passing around the gate 10. Here, a person attempting to pass through the gate 10 is not limited to a person who is permitted to pass through the gate 10 (for example, a person whose face image has been registered in the face authentication server 16), but may also be, for example, a person whose face image is not registered in the face authentication server 16 but who is attempting to pass through. Furthermore, a person passing around the gate 10 may be, for example, a person who is not attempting to pass through the gate 10 but has passed through the capture range of the camera 11, or a person who is not attempting to pass through the gate 10 but has entered the capture range. Furthermore, the status of a person may be, for example, a status related to the movement of a person, such as whether the person is moving or stationary, and the direction of movement if the person is moving.

[0056] The passage management function unit 15 includes, for example, a passage management state transition processing unit 151, a history management unit 152, and a history database (DB) 153.

[0057] For example, in the person passage management process, the passage management state transition processing unit 151 transmits to the gate 10 control information regarding the control of the gate 10 when a person who is permitted to pass through the gate 10 passes through the gate 10. Also, the passage management state transition processing unit 151 transmits to the gate 10 control information regarding the control of the gate 10 when a person who is not permitted to pass through the gate 10 attempts to pass through the gate 10.

[0058] The history management unit 152, for example, stores and manages information (passage history information) indicating the history of a person who has passed through the gate 10. Furthermore, the history management unit 152, for example, stores the passage history information in the history DB 153 and transmits the passage history information to the passage history management server 17. For example, in a railway network, the history management unit 152 may manage local passage history information for each station (or each ticket gate).

[0059] The passage history management server 17 stores and manages, for example, information (passage history information) indicating the history of a person who has passed through the gate 10. For example, the passage history management server 17 may manage the passage history information of a plurality of gates 10. For example, in a large facility with a plurality of entrances and exits, the passage history information of the gates 10 provided at each of the plurality of entrances and exits may be aggregated and managed by the passage history management server 17. Also, for example, in a railway network, the passage history information of each of the gates 10 provided at the ticket gates of a plurality of stations may be aggregated and managed by the passage history management server 17.

[0060] The passage management function unit 15 may output, for example, information related to passage management (passage management information) to a display device (not shown). The passage management information may include, for example, information output from the face authentication function unit 13 and information output from the person position estimation unit 14. The display device may display, for example, the state of the person (for example, the result of face authentication of the person and the direction of movement). For example, the display device may display a right image and / or a left image and superimpose a detected face frame on the right image and / or the left image. Furthermore, the display device may display, for example, information related to the person obtained by face authentication (the person's ID) by superimposing it on the right image and / or the left image.

[0061] The face authentication function unit 13 described above may operate, for example, synchronously with or asynchronously with the passage management function unit 15. In the case of asynchronous operation, for example, the face authentication function unit 13 may operate when a face frame is detected in the camera control unit 131.

[0062] The three components of the face authentication function unit 13, the person position estimation unit 14, and the passage management function unit 15 described above may each have the form of a single information processing device (e.g., a server device), or two or more of the three may be included in a single information processing device. For example, the face authentication function unit 13 may have the form of a single information processing device, and the person position estimation unit 14 and the passage management function unit 15 may be included in a single information processing device. Furthermore, at least one of the face authentication function unit 13, the person position estimation unit 14, and the passage management function unit 15, which have the form of an information processing device, may be included in the gate 10.

[0063] The information processing device described above may include a processor, a memory, and an input / output interface used for transmitting various types of information. The processor is an arithmetic device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The memory is a storage device realized using a RAM (Random Access Memory), a ROM (Read Only Memory), or the like. The processor, memory, and input / output interface are connected to a bus, and various types of information are exchanged via the bus. The processor realizes the functions of the configuration included in the information processing device by reading programs, data, etc. stored in the ROM, for example, into the RAM and executing processing.

[0064] The above-described person position estimation unit 14 and passage management function unit 15 may, for example, define areas (or zones) at the gate 10, and perform person detection and passage management based on the defined zones. An example of a zone defined at the gate 10 will be described below.

[0065] <Gate area management> 3A and 3B are diagrams showing examples of zones defined for gate 10. 3A and 3B show examples of multiple zones when gate 10 is viewed from above. 3A and 3B show an example in which sidewall V of gate 10, which forms passage L, extends in the vertical direction of the paper.

[0066] As shown in FIG. 1, of the entrances E1 and E2 of the gate 10, for example, the upstream side along a specific approach direction (for example, entrance direction) corresponds to the entrance, and the downstream side corresponds to the exit.

[0067] Figure 3A shows an example of a zone defined when a person enters gate 10 from entrance E2, where the person can enter from both entrances E1 and E2. Figure 3B shows an example of a zone defined when a person enters gate E1.

[0068] When gate 10 allows entry in both directions, it is conceivable that whether the movement direction is normal or not will differ depending on the entrance / exit through which the person enters. For example, the movement direction of a person from entrance / exit E1 to entrance / exit E2 may be normal for a person who entered through entrance / exit E1, but may be abnormal for a person who entered through entrance / exit E2. In response to such differences in regulations, the passage management function, for example, defines entrance / exit E1 as the "north side" and entrance / exit E2 as the "south side."

[0069] The expressions "north side" and "south side" are merely examples, and the present disclosure is not limited to these expressions. For example, the expressions "north side" and "south side" do not limit the placement of gate 10 to a placement along the geographic north-south direction. For example, even if the passage L of gate 10 is located along a direction different from the north-south direction, or even if the passage includes a curve, one side may be defined as the "north side" and the other as the "south side."

[0070] For example, Figure 3A shows an example of zones defined when a person enters through entrance / exit E2. In Figure 3A, "Zone outside-S" (south side outside zone area), "Zone A," "Zone B," and "Zone C" are defined for gate 10.

[0071] In contrast, Figure 3B shows an example of zones defined when a person enters through entrance / exit E1. In Figure 3B, "Zone outside-N" (northern outside zone area), "Zone A," "Zone B," and "Zone C" are defined for gate 10.

[0072] Below, each zone will be explained using the example of Figure 3A. The example of Figure 3B is the same as Figure 3A, except that a person enters through entrance / exit E1 and "Zone outside-S" (area outside the south zone) is replaced with "Zone outside-N" (area outside the north zone).

[0073] The boundary between the southern outer zone area and Zone A may be referred to as, for example, the "face recognition start line."

[0074] The "face recognition start line" is used, for example, to determine whether or not to start face recognition processing. For example, when a person crosses the "face recognition start line" and enters gate 10, face recognition processing is started. For example, a face matching request is issued from face frame information, the matching result (face recognition ID) is linked to person detection information, and tracking of the person begins. The "face recognition start line" is sometimes referred to as the "A line."

[0075] The "face authentication start line" may be provided outside the gate 10 (for example, upstream along the path of the gate 10). The "face authentication start line" is not limited to a single line segment, but may have multiple line segments, such as a U-shape. Note that a shape having multiple line segments is not limited to a shape corresponding to a portion of a rectangular shape, such as a U-shape, but may also be a shape corresponding to a portion of a side of another polygonal shape. Alternatively, the "face authentication start line" may have an arc, or may have a shape that combines straight lines and curves. For example, by having the "face authentication start line" have multiple line segments and / or arcs, face authentication processing can be started not only when a person enters from the front of the gate 10, but also when a person enters from a direction that is different from the front, such as from the side.

[0076] The boundary between Zone A and Zone B may be referred to as, for example, the "door closing limit line."

[0077] The "door closing limit line" indicates, for example, the position at which the exit-side gate door will close in response to a door closing command in time for a person to pass through. The "door closing limit line" is determined, for example, taking into consideration the maximum speed at which a person is expected to pass through gate 10 (e.g., 6 km / h; hereinafter, "maximum passable speed") and the time required for the gate door to physically close (e.g., 0.5 seconds). For example, the "door closing limit line" is set a distance before the physical position of the gate door ("gate door position") that is equivalent to the maximum passable speed multiplied by the time required for the gate door to physically close. In this way, if a person who is not permitted to pass through gate 10 passes over the "door closing limit line" and moves at the maximum passable speed, the exit-side gate door will close before the person passes through the exit-side gate door.

[0078] The "closed door limit line" may also be called the "unauthorized intrusion detection line" or the "B line."

[0079] The boundary between Zone B and Zone C may be referred to as the "exit line."

[0080] The "exit line" indicates, for example, the position at which it is determined that the person has exited the gate 10. The "exit line" may be set outside the gate 10, similar to the above-mentioned "face recognition start line." Furthermore, the "exit line" is not limited to, for example, a single line segment, and may have multiple line segments, such as a U-shape. Alternatively, the "exit line" may have an arc. The "exit line" may also be called, for example, a "Z line."

[0081] In passage management, the gate door position may simply be a passage point, in which case the gate door position may be different from or the same as the logically set "exit line." For example, in actual operation, the "gate door position" and the "exit line" may be set to be the same.

[0082] For example, in the case of a gate 10 that charges a fee to people passing through, the "exit line" may correspond to the "charging line."

[0083] For example, if a person who enters gate 10 crosses the charging line (e.g., if they enter zone C from zone B), they will be charged. In other words, if the person has not crossed the charging line (e.g., has not entered zone C), they will not be charged. By providing this charging line, it is possible to avoid the mistake of charging a person who enters gate 10 but turns back before crossing the charging line.

[0084] Although the above example shows that the "charging line" corresponds to the "Z line" ("exit line"), the "charging line" may also correspond to the "B line", for example.

[0085] In the above example, three zones are defined, excluding the area outside the north zone and the area outside the south zone, but the present disclosure is not limited to this. The number, size, position, and shape of the zones may be changed depending on the situation to which the present disclosure is applied.

[0086] By detecting the position of a person relative to the above-mentioned zones, for example, movement between zones can be estimated. In this embodiment, the position of a person is estimated using, for example, images captured by the right camera 11 and the left camera 11.

[0087] FIG. 4 is a diagram showing an example of face frame detection in this embodiment. FIG. 4 shows right images R1 and R2 captured by right camera 11 and left images L1 and L2 captured by left camera 11. Note that right image R1 and left image L1 are images captured at the same time t1, for example. Right image R2 and left image L2 are images captured at the same time t2, for example. Time t2 is a time later than time t1. For example, the position of a person at time t2 is closer to the charging line of gate 10 than the position of the person at time t1.

[0088] The right image R1, the right image R2, the left image L1, and the left image L2 include, for example, a person passing through the gate 10 and a frame (face frame) surrounding the person's face.

[0089] For example, when comparing left images L1 and L2, the face frame in left image L2 is closer to the left edge of the image area than the face frame in left image L1. Also, when comparing right images R1 and R2, the face frame in right image R2 is closer to the right edge of the image area than the face frame in right image R1.

[0090] In other words, by checking the positional relationship between the face frame in the left image and the face frame in the right image (for example, the positional relationship in the left-right direction in the image area), it is possible to detect, for example, the position of a person. Hereinafter, the left-right direction in the image area is defined as the horizontal direction or the X-axis direction.

[0091] Fig. 5A is a diagram showing an example of detecting the position of a person in this embodiment. Fig. 5A shows two images based on the image shown in Fig. 4 and an extracted image in which face frames are extracted from the two images.

[0092] Image T1 in Fig. 5A shows an example of a comparison of the positional relationship of the face frame between left image L1 and right image R1 at time t1 shown in Fig. 4. For example, the right side of image T1 shows a partial region including the face frame of left image L1 shown in Fig. 4, and the left side of image T1 shows a partial region including the face frame of right image R1 shown in Fig. 4. In image T1, the face frame of left image L1 is located to the right of the face frame of right image R1.

[0093] Image T2 in Fig. 5A shows an example of a comparison of the positional relationship of the face frame between left image L2 and right image R2 at time t2 shown in Fig. 4. For example, the right side of image T2 shows a partial region including the face frame of right image R2 shown in Fig. 4, and the left side of image T2 shows a partial region including the face frame of left image L2 shown in Fig. 4. In image T2, the face frame of left image L2 is located to the left of the face frame of right image R2.

[0094] 5A, the positional relationship between the face frame in the right image and the face frame in the left image, which are captured at the same time, changes depending on the position of the person. Therefore, in this embodiment, the position of the person is estimated based on the difference between the position of the face frame in the right image and the position of the face frame in the left image.

[0095] For example, the position of the face frame is represented by a representative point of the face frame. In the following, an example will be described in which the representative point is the center point of the face frame. Since the center point of the face frame does not change significantly even if the size of the face differs, by using the center point of the face frame as the representative point, it is possible to stably estimate the position of a person even if the size of the person's face varies. However, the present disclosure is not limited to this. If the face frame is rectangular, the representative point may be a point indicating a corner of the rectangle. If the face frame is elliptical, the representative point may be the focus of the ellipse.

[0096] As shown in the extracted images U1 and U2, the point P R and point P, which indicates the center of the face frame in the left image. L For example, the distance between point P R Point P starting from R From point P L The horizontal component (component along the X axis) of the vector from the right to the face frame point P R and point P on the face frame in the left image L The distance between the horizontal coordinate position difference and the horizontal coordinate position difference may be referred to as the horizontal coordinate position difference or the position difference.

[0097] For example, in the extracted image U1, point P R is point P LSince the point P is to the left of the point P, the horizontal component of the vector has a positive value. R is point P L , the horizontal component of the vector has a negative value.

[0098] 5B is a diagram showing an example of the relationship between the size and center of the face frame relative to the position of a person. Fig. 5B shows the positional relationship of the center of the face frame for three different distances (far, medium, and close) from the charging line of gate 10, as well as the relationship between the size of the face frame and the direction of the vector defined by the center of the face frame.

[0099] As shown in Figure 5B, when the distance from the charging line is far, comparing the center of the face frame in the right image with the center of the face frame in the left image in the image area, the center of the face frame in the right image is shifted to the left, and the center of the face frame in the left image is shifted to the right. The closer you get to the charging line, the more the center of the face frame in the right image moves to the right, and the center of the face frame in the left image moves to the left. Therefore, the direction of the vector from the center of the face frame in the right image to the center of the face frame in the left image changes from positive to negative, i.e., the polarity of the horizontal distance from the center of the face frame in the right image to the center of the face frame in the left image is reversed.

[0100] The position of the person is estimated by comparing the position difference with a threshold value. For example, if the position difference is equal to or smaller than the threshold value, it is determined that the person is located beyond the charging line.

[0101] Alternatively, when a person passes through a gate, the position difference gradually decreases from a positive value and changes to a negative value. When the position difference changes from a positive value to a negative value (polarity is determined), the positional relationship between the face frame in the left image and the face frame in the right image is reversed. In other words, the face frame in the left image and the face frame in the right image cross each other when the position difference reaches zero. Therefore, when the position difference becomes zero, it may be determined that the person is located beyond the charging line.

[0102] For example, the camera angle of view, gate size, and charging line position may be determined so that the position of the point where the face frame in the left image intersects with the face frame in the right image (cross point) coincides with the charging line.

[0103] The position of the point (cross point) where the face frame of the left image and the face frame of the right image intersect is less affected by the size of the face frame (i.e., the size of the person's face), so if the position of the cross point corresponds to the charging line, the accuracy of determining the charging line can be improved.

[0104] Furthermore, since the center of the face frame is used to calculate the difference in the horizontal coordinate of the center to perform estimation, it is possible to avoid the influence of differences in the size of the face frame due to the size of the face and / or height, and it is possible to suppress or avoid a decrease in estimation accuracy.

[0105] FIG. 6A is a diagram showing a first example of the transition of the magnitude of the position difference. FIG. 6B is a diagram showing a second example of the transition of the magnitude of the position difference. FIGS. 6A and 6B differ from each other in the arrangement of the right camera 11 and the left camera 11. The horizontal axis of FIGS. 6A and 6B indicates the distance along the passage from a position outside the entrance of gate 10 where face authentication is possible, and the vertical axis indicates the magnitude of the position difference. Note that when the position difference is a positive value, the center of the face frame in the left image is located to the right of the face frame in the right image. Note that when the position difference is a negative value, the face frame in the right image is located to the right of the face frame in the left image.

[0106] 6A and 6B, the position difference takes on a zero value at a certain position. When comparing Fig. 6A with Fig. 6B, the position at which the position difference becomes zero may differ depending on the arrangement of the camera 11.

[0107] As described above, if a charging line is defined for gate 10, the position where the position difference is zero can be made to correspond to the charging line by, for example, adjusting the position and / or angle of camera 11. Alternatively, since the position where the position difference is zero is defined by the camera position, the charging line can be adjusted.

[0108] Next, a flow of person tracking based on the detected face frame will be described. Fig. 7 is a diagram showing an example of the flow of person tracking processing based on the face frame in this embodiment.

[0109] The face frame position determination library 201 is a library having a function of, for example, acquiring face frame detection information and detecting the position of a person corresponding to the face frame. The face frame position determination library 201 detects the position of a person from the position of the face frame in an image, for example, by the method described above.

[0110] In addition, the face frame position determination library 201 may assign a new person ID to the face frame detection information if, for example, the acquired face frame detection information has no continuity with previously acquired face frame detection information in terms of time and / or coordinate space.

[0111] Furthermore, if the acquired face frame detection information has continuity with previously acquired face frame detection information in terms of time and / or coordinate space, for example, the face frame position determination library 201 continues tracking of the person based on the face frame detection information that has continuity.

[0112] The face frame position determination library 201 outputs, for example, person tracking information to the passage management library 202 .

[0113] The person tracking information may include, for example, a person ID for identifying a person. The person tracking information may also include, for example, information regarding the position of a person (for example, the appearance of a person, the tracking of a person, the disappearance of a person, etc.). The person tracking information may be compatible between the face frame position determination library 201 and the passage management library 202, for example.

[0114] The passage management library 202 has a function of identifying a passage management event at the gate 10, for example, based on person tracking information. The passage management library 202 also outputs, for example, the passage management event to the passage management processing unit 203. The passage management event includes at least one of a plurality of events, such as an event representing that a person has moved between zones in the multiple zones shown in Figures 3A and 3B, an event representing that a person has crossed a line that defines the zones, an event representing that a person has appeared in a certain zone, and an event representing that a person has disappeared in a certain zone.

[0115] The passage management processing unit 203 outputs passage information indicating whether or not a person has passed through the gate 10 or whether or not a person has passed through a charging line, based on a passage management event, for example.

[0116] The result output unit 204 outputs, for example, the tracking result of the person indicated by the passage information, and for example, displays the result on a display.

[0117] Next, the area defined for the face frame detected in the image will be described.

[0118] Fig. 8 is a diagram showing an example of an area defined in an image region. Fig. 8 shows an image region in which the upper left corner is the origin (0,0), the horizontal length of the frame (X-axis direction) is represented as "XframeMax", and the vertical length is represented as "YframeMax". Fig. 8 also shows, by way of example, a face frame whose side length is wf and whose upper left coordinate is (Xf, Yf), and the center of the face frame is (Xc, Yc). Note that information indicating the face frame may be expressed, for example, by X and Y coordinates representing the upper left point of the face frame and W representing one side of the face frame, as shown in Fig. 8.

[0119] As shown in Fig. 8, three areas may be defined in the image region of the left image. For example, of the three areas, the right area represents an initial appearance area (First Face Frame Area (hereinafter, FFFA)), the central area represents a middle area (Middle Face Frame Area (hereinafter, MFFA)), and the left area represents a passing confirmation area (Passed Face Frame Area (hereinafter, PFFA)).

[0120] Note that while FIG. 8 shows three areas in the image region of the left image, the left and right of the three areas may be reversed in the image region of the right image. In other words, in the image region of the right image, the left area may represent FFFA, and the right area may represent PFFA. Also, the sizes and positions of the three areas in FIG. 8 are merely examples, and the present disclosure is not limited thereto. The sizes and positions of the three areas may differ for each camera 11.

[0121] For example, the face frame position determination library 201 may determine, based on the face frame detection information, which of the three areas of the image region the center of the detected face frame is included in.

[0122] 9 is a flowchart showing an example of the flow of passage management. The flow shown in FIG. 9 may be started, for example, every time information on an image captured by the camera 11 is acquired.

[0123] The face authentication function unit 13 acquires information about an image captured by the camera 11 and performs processing to detect a face frame from the image (S101). The image information acquired from the camera 11 may include information identifying the camera that captured the image (hereinafter referred to as a camera ID), information identifying the captured frame (hereinafter referred to as a frame ID), the date and time of capture, etc. The face authentication function unit 13 generates information about the detected face frame and outputs the information about the face frame and information about the image to the person position estimation unit 14. The information about the face frame may include information indicating the position and size of the face frame. If a face frame is not detected from the image, the processing from S102 onwards need not be executed.

[0124] Next, the person position estimation unit 14 performs the following processes based on the face frame position determination library 201.

[0125] The person position estimation unit 14 performs face frame timeout processing (S102). In the face frame timeout processing, timeout processing is performed on the face frame detection information stored in the face frame information list based on the elapsed time from the timing when the face frame detection information was first acquired. The face frame timeout processing will be described later.

[0126] Next, the person position estimation unit 14 performs a face frame detection information generation process (S103). For example, in the face frame detection information generation process, face frame detection information including information on the camera ID, frame ID, shooting date and time, and coordinates of the detected face frame is generated. The face frame detection information may also include information on the area in which the face frame has been detected. The area in which the face frame has been detected is determined based on area specifications such as those illustrated in FIG. 8. The area specifications may be included in face frame appearance area definition information. Note that, when multiple face frames are detected in one image, the face frame detection information generation process is executed for each of the multiple face frames. Note that the face frame detection information generation process will be described later.

[0127] Next, the person position estimation unit 14 performs FFFA multiple face frame elimination processing (S104). In the FFFA multiple face frame elimination processing, it is determined whether or not multiple face frames exist in FFFA, and if multiple face frames exist in FFFA, the subsequent processing is not executed. The FFFA multiple face frame elimination processing will be described later.

[0128] The person position estimation unit 14 performs a time-series face frame detection evaluation process (S105). In the time-series face frame detection evaluation process, the face frame detection information is evaluated by comparing the latest information in the face frame detection table with the immediately preceding information. For example, if the face frame indicated in the face frame detection information is the face frame of a newly appeared person, a new ID is assigned to that person.

[0129] Next, the person position estimation unit 14 performs a two-lens type face frame position estimation process (S106). The two-lens type face frame position estimation process generates person tracking information. The two-lens type face frame position estimation process will be described later.

[0130] Next, the person position estimation unit 14 outputs the person tracking data to the passage management function unit 15 (S106). Then, the flow of Fig. 9 ends, and the passage management function unit 15 performs passage management processing based on the passage management library.

[0131] <Face frame detection information generation process> In the face frame detection information generation process (S103), the detected position of the face frame is determined. For example, it is determined in which area of ​​the image region the position of the detected face frame is included in each of the right and left images.

[0132] For example, one face frame is extracted from the face frame information, the center of the extracted face frame is calculated, and it is determined in which area of ​​FFFA, MFFA, or PFFA the center exists.

[0133] For example, if the center of a face frame does not exist in any area, the information about that face frame is discarded because there is a high possibility that the face frame belongs to an unauthorized person.

[0134] If the center of the face frame is in one of the three areas, the size of the face frame is compared with the minimum face frame size set for the area where the face frame is located. If the size of the face frame is smaller than the minimum face frame size, the information for the face frame is discarded. If the face frame size is smaller than the minimum face frame size, it is likely to be the face of a person who is unlikely to enter gate 10, or a face drawn on clothing or a poster, or a pattern that is not a face may have been mistakenly detected as a face, which is likely to be the face frame of an unauthorized person.

[0135] When multiple face frames are detected in one image, the above-mentioned determination of the area in which the face frame exists and comparison with the minimum face frame size set for the area in which the face frame exists are performed for each of the multiple face frames.

[0136] A list for performing time-series determination processing from the face frame detection information may be generated. The generated list may be called a face frame information list (or a queue structure list).

[0137] The face frame information list may include information on the center coordinates of the face frame and the appearance area in the image region.

[0138] <Face frame timeout processing> Next, the face frame timeout process (S102) will be described. Fig. 10 is a diagram showing an example of the face frame timeout process.

[0139] In the face frame timeout process (S102), the retention time from the time when the face frame to be determined was first detected (for example, the time when the face frame to be determined first appeared and a new ID was assigned) to the time when the image including the face frame to be determined was captured is compared with the limit time for retaining the face frame (face frame retention limit time). Note that the retention time may be determined, for example, by referring to the face frame information list. The face frame retention limit time is a predetermined threshold value and is stored in a storage unit or the like.

[0140] For example, in the face frame timeout process, when the retention time of face frame information assigned with an ID exceeds a time limit, the face frame information is deleted from the face frame information list.

[0141] This process makes it possible to prevent information about a person whose face frame has been detected but who has not passed through the gate 10 and is out of the range of the camera 11 from remaining in the list.

[0142] 9 shows an example in which the face frame timeout process is executed before the face frame detection information generation process, but the present disclosure is not limited to this. In the face frame timeout process, it may be possible to constantly check whether a timeout has occurred in the face frame information.

[0143] <FFFA Multiple Face Frame Exclusion Process> Next, the FFFA multiple face frame exclusion process (S104) will be described. FIG. 11 is a flowchart showing an example of the FFFA multiple face frame exclusion process. For example, the flow shown in FIG. 11 starts after S103 shown in FIG. 9.

[0144] In the initial setting, the person position estimation unit 14 sets the variable i to the number of detected face frames (S201). Note that i in FIG. 11 is an integer greater than or equal to 1. Also, hereinafter, the case where the number of detected face frames is N (N is an integer greater than or equal to 1) will be described. In this case, the N detected face frames are represented as face frame [1] to face frame [N].

[0145] The person position estimation unit 14 sets the variable "count" to 0 (S202).

[0146] The person position estimation unit 14 determines whether i is greater than 0 (S203).

[0147] If i is greater than 0 (YES in S203), the person position estimation unit 14 determines whether face frame [i] is detected within the FFFA (S204).

[0148] If face frame [i] is detected within the FFFA (YES in S204), the person position estimation unit 14 determines whether the size of face frame [i] is greater than or equal to a predetermined size (S205). Note that the predetermined size may be, for example, the size of a face frame at which face authentication can be started, or a size defined based on the size of a face frame at which face authentication can be started.

[0149] If the size of face frame [i] is greater than or equal to the predetermined size (YES in S205), the person position estimation unit 14 adds 1 to count (S206).

[0150] Then, the person position estimation unit 14 subtracts 1 from i (S207). Then, the process of S203 is executed.

[0151] If the face frame [i] is not detected in the FFFA (NO in S204), or if the size of the face frame [i] is not equal to or larger than a predetermined size (NO in S205), the person position estimation unit 14 subtracts 1 from i (S207). Then, the process of S203 is executed.

[0152] If i is not greater than 0 (NO in S203), for example, when the processes of S204 to S206 are completed for each detected face frame, the person position estimation unit 14 determines whether count is 2 or greater (S208).

[0153] If count is 2 or more (YES in S208), that is, if two or more face frames of a predetermined size or more exist in the FFFA, the person position estimation unit 14 deletes the information of the face frame corresponding to the face frame in the FFFA (S209). Then, the flow shown in FIG. 11 ends.

[0154] If count is not 2 or more (NO in S208), that is, if there are not two or more face frames of a predetermined size or more in the FFFA, the person position estimation unit 14 may end the processing without deleting the face frame information.

[0155] As described above, if the FFFA contains multiple face frames with a size equal to or larger than the threshold for starting face recognition, face recognition processing is not performed for each detected face frame. If face recognition processing is not performed for each detected face frame, the face frame information corresponding to the face frame detected in the FFFA may be deleted. In this case, face recognition processing cannot be performed, so error processing is executed. For example, the output control process of the gate device may be instructed to present a warning message to a person attempting to pass through the gate device via an audio output device and / or a display device.

[0156] This process makes it possible to eliminate fraudulent passage even when the face frames of multiple people are lined up in a vertical line and multiple people are attempting to pass through the gate device all at once fraudulently.

[0157] <Two-eye face frame position estimation processing> Next, the twin-lens type face frame position estimation process (S106) will be described. Fig. 12 is a flowchart showing an example of the twin-lens type face frame position estimation process. For example, the flow shown in Fig. 12 starts after S105 shown in Fig. 9.

[0158] The person position estimation unit 14 determines whether or not a face frame exists in the face frame information list (S301).

[0159] If a detected face frame exists (YES in S301), the person position estimation unit 14 determines whether the difference between the shooting time when the left image was captured by the left camera 11 and the shooting time when the right image was captured by the right camera 11 is within an allowable range (S302). In other words, in S302, it is determined whether the shooting times of the left image and the right image are simultaneous, nearly simultaneous, or have an unacceptable difference. In other words, it is determined whether the shooting times of the left image and the right image are synchronized.

[0160] If the difference in the shooting times is not within the allowable range (NO in S302), the person position estimation unit 14 records a synchronization error in the log (S306).

[0161] If the difference in the shooting times is within the allowable range (YES in S302), the person position estimation unit 14 calculates the difference between the X coordinate of the center of the face frame in the left image and the X coordinate of the center of the face frame in the right image (S303). This difference in the X coordinate of the center of the face frame may be referred to as the "center coordinate difference value" below.

[0162] The person position estimation unit 14 then converts the calculated difference into distance information (S304). For example, a conversion table may be used for this conversion. An example of the conversion table will be described later.

[0163] The person position estimation unit 14 stores the converted result (S305), and the flow then ends.

[0164] If no face frame was detected (NO in S301), or after recording a synchronization error (after S306), the person position estimation unit 14 stores a result indicating that position estimation was not performed (S307). For example, information indicating "no two-eye determination" may be set in the result storage area. Then, the flow ends.

[0165] <Conversion table> Next, an example of the conversion table used in S304 of FIG. 12 will be described.

[0166] For example, in the conversion table, the central coordinate difference value and the estimated position where the target person is located are recorded in association with each other. For example, the numerical information stored in the conversion table may be recorded in pixel units (e.g., in units of one pixel). The difference value may also have a negative number. The estimated position where the target person is located may be represented by the distance from a reference point provided on the gate 10, or may be represented by two-dimensional coordinates (e.g., X coordinate and Y coordinate) from the reference point.

[0167] For example, the conversion table may be stored in memory in order to speed up processing related to currency management.

[0168] Furthermore, if the gate 10 is passable from both directions, a conversion table may be provided for each direction, for example.

[0169] The conversion table may be created when the gate 10 is installed, or may be provided externally.

[0170] As described above, in this embodiment, the center of a face frame indicating the area of ​​a person's face in a right image obtained by photographing the person's face from the front right as the person enters through the entrance to gate 10, and the center of the face frame in a left image obtained by photographing the person's face from the front left, are determined, and the position of the person at gate 10 is estimated based on changes in the positional relationship between the center of the face frame in the right image and the center of the face frame in the left image. This configuration improves the accuracy of estimating the position of a person about to pass through the gate's charging line.

[0171] Furthermore, according to this embodiment, a pass management process including authentication and tracking is performed by performing tracking using an image captured by a camera used for face authentication. This allows the camera used for authentication to also be used for tracking without providing a device for tracking (for example, a camera on the ceiling, etc.), which reduces the introduction cost of pass management and allows the pass management system to be introduced without location restrictions.

[0172] Furthermore, according to this embodiment, the position of a person is estimated based on the distance between the centers of the face frames acquired from the left and right images, respectively. This allows estimation even when there are differences in the installation positions, angle of view, image quality, and resolution between the right and left cameras 11. For example, a stereo camera measures the distance to a subject using the parallax between the two cameras. Therefore, the stereo camera requires precise adjustment of the angles of both cameras, and also requires control processing dedicated to the stereo camera. In this embodiment, the position is estimated using the positional relationship between the face frames detected in the face recognition process, and therefore estimation is easier than with a stereo camera method without requiring precise camera adjustment.

[0173] The configuration shown in this embodiment is merely an example, and the present disclosure is not limited to this. Variations in the position of the camera 11 installed at the gate 10 will be described below.

[0174] 1 shows an example in which the distance from the entrance E1 of gate 10 to the right camera 11 (camera 11-R1) and the distance from the entrance E1 to the left camera 11 (camera 11-L1) are equal, but the present disclosure is not limited to this. For example, one of the right camera 11 and the left camera 11 may be installed at a position closer to the entrance E1 of gate 10 than the other. In other words, in this case, there is a difference in the installation positions of the right camera 11 and the left camera 11 with respect to the entrance E1 of gate 10, and therefore there is a difference in the front-to-back between the shooting range of the right image and the shooting range of the left image.

[0175] For example, if the left camera 11 is located closer to the entrance E1 than the right camera 11, when comparing the face frames in the left and right images, the face frame in the left image will likely move out of the image area before the face frame in the right image, making it unlikely that the face frame will be detected. In such a case, the area defined in the image area can be changed. The following describes an example in which the left camera 11 is located closer to the entrance E1 than the right camera 11.

[0176] Fig. 13 is a diagram showing an example of a defined area for a difference in camera placement, which shows images taken by the right camera 11 and the left camera 11 at two points in time, the face frames detected in each image, and the defined area in each image.

[0177] Each image in FIG. 13 includes person A entering gate 10 and person B entering gate 10 behind person A. As described above, when left camera 11 is installed closer to entrance / exit E1 than right camera 11, the face frame of person A in the left image moves out of the image area before the face frame of person A in the right image. For example, in the example of FIG. 13, because the face frame of person A in the left image moves out of the image area, there is a possibility that the face frame of person B in the left image and the face frame of person A in the right image may be erroneously determined to be the same person. Therefore, as shown in FIG. 13, the PFFA in the image of right camera 11 is set to be narrower in the horizontal direction than the PFFA in the image of left camera 11. This setting, for example, can align the timing at which the center of the face frame moves out of the PFFA between the left image and the right image.

[0178] <Control example 1 when images cannot be acquired from one camera> For example, in the above-mentioned twin-lens system, there may be cases where a face frame cannot be detected in an image captured by one of the two cameras. For example, a case where a face frame cannot be detected may include a case where the face frame detection process fails in an image captured by one of the cameras, or a case where a problem occurs in the capture of one of the cameras (for example, a camera malfunction or a temporary malfunction). Below, an example will be described in which, when a face frame cannot be detected in an image captured by one of the two cameras, passage management is performed based on the face frame detected in an image captured by the other camera. Note that a processing system implemented using images captured by one camera is sometimes referred to as a single-lens system.

[0179] For example, if face frame detection fails in the left image captured by the left camera but face frame detection succeeds in the right image captured by the right camera, the position of the person may be estimated based on the amount of movement of the face frame in the right image. In this person position estimation process, for example, the face frame of a person is detected when a new person appears passing through gate 10 in a time series. An example of detecting the face frame of a new person will be described below.

[0180] Fig. 14 is a flowchart showing an example of new face frame detection using one camera. The flow shown in Fig. 14 may be executed, for example, when information on an image captured by either the right camera 11 or the left camera 11 is acquired and image information is not acquired from the other camera, or when either the right camera 11 or the left camera 11 fails to capture an image.

[0181] The face authentication function unit 13 acquires information about an image captured by the camera 11 and performs processing to detect a face frame from the image (S401).

[0182] The person position estimation unit 14 determines whether the size of the face frame is equal to or larger than a designated size (S402).

[0183] If the face frame size is equal to or larger than the specified size (YES in S402), the person position estimation unit 14 performs a same person determination process (S403). For example, the person position estimation unit 14 determines whether or not the same person as the person in the detected face frame is present among the people in the face frames shown in the past face frame information list. For example, this determination may be performed by comparing the respective feature points, as in the face authentication process.

[0184] As a result of the same person determination process, the person position estimation unit 14 determines whether or not the people are the same person (S404).

[0185] If it is not the same person (YES in S404), the person position estimation unit 14 determines whether the center position of the detected face frame is included in a new appearance zone (for example, FFFA) (S405).

[0186] If the center position of the detected face frame is included in the new appearance zone (YES in S405), the person position estimation unit 14 determines that the detected face frame is the face frame of a newly appeared person, and registers information about the face frame (S406).Then, the flow ends.

[0187] In S406, for example, if the person corresponding to the detected face frame is different from the people corresponding to the face frames detected so far and the face frame is within a zone where it may be determined that a person has newly appeared, the face frame is recognized as the face frame of a new person. For example, the information may be registered in a face frame tracking management table. In this case, a face matching request may be notified.

[0188] If the face frame size is not equal to or larger than the specified size (NO in S402), or if the center position of the detected face frame is not included in the new appearance zone (NO in S405), the person position estimation unit 14 determines that the detected face frame is a face frame that is not subject to management (S407), and the flow ends.

[0189] If they are the same person (NO in S404), the person position estimation unit 14 executes the tracking process for the same person (S408), and the flow ends.

[0190] In the above flow, it is determined that an initial face frame for a new person has appeared when the center point of the face frame is in the new appearance zone, but for example, if a face frame is detected in the new appearance zone a predetermined number of times in succession, it may be determined that the person corresponding to the detected face frame is a new person. By making a determination based on successive detections, it is possible to avoid registering a face frame that appears once and then disappears the next time.

[0191] The same person determination process is not particularly limited. For example, it may be determined whether or not the two images are the same person based on the amount of movement of the face frame. For example, a predetermined range of the amount of movement set based on the shooting interval (a range in which a person can move relative to the shooting interval (e.g., 60 msec)) is compared with the amount of movement of the face frame between the two images, and if the amount of movement of the face frame is within the predetermined range, it may be determined that the two images are the same person.

[0192] <Control example 2 when images cannot be acquired from one camera> The above example has shown a method for detecting the face frame of a newly appearing person when a face frame is not detected in an image captured by one of the two cameras in a twin-lens system (a system in which processing is performed using images captured by two cameras). Below, an example will be described in which, in a twin-lens system, a face frame is not detected in an image captured by one of the two cameras, and it is determined whether a person passing through gate 10 has crossed the charging line.

[0193] Fig. 15 is a flowchart showing an example of determining whether a vehicle has crossed the charging line using one camera. The flow shown in Fig. 15 may be executed, for example, when image information captured by either the right camera 11 or the left camera 11 is acquired but image information is not acquired from the other camera, or when either the right camera 11 or the left camera 11 cannot capture an image.

[0194] The person position estimation unit 14 calculates the center position of the face frame (S501).

[0195] The person position estimation unit 14 determines whether the center position of the face frame is within a charging zone (for example, PFFA) (S502).

[0196] If the center position of the face frame does not exist within the charging zone (for example, PFFA) (NO in S502), the person position estimation unit 14 may end the process.

[0197] If the center position of the face frame is within the charging zone (YES in S502), the person position estimation unit 14 determines whether the initial appearance of the face frame is in a zone other than the new appearance zone (e.g., FFFA) (S503). For example, it refers to the information on the face frame that is first detected among the information on the face frame that indicates that the person is the same as the face frame whose center position is within the charging zone, and determines whether the zone where the referenced face frame was detected is the new appearance zone.

[0198] If the initial appearance of the central position is in a zone other than the new appearance zone (YES in S503), for example, if the person in the face frame is not a person about to pass through gate 10 but a person passing around gate 10 (for example, another gate adjacent to gate 10), the flow ends because no charge will be made to that person. In this case, a determination result indicating that no charge will be made to that person may be recorded.

[0199] If the initial appearance of the central position is not in a zone other than the new appearance zone (e.g., FFFA) (NO in S503), the person position estimation unit 14 determines whether the size of the face frame is equal to or larger than the face frame size, which is the threshold defined in the charging zone (S504).

[0200] If the size of the face frame is not equal to or larger than the face frame size defined in the charging zone (NO in S504), the person position estimation unit 14 may end the process.

[0201] If the size of the face frame is equal to or larger than the face frame size defined in the charging zone (YES in S504), the person position estimation unit 14 determines whether the size of the face frame is increasing (S505). For example, the person position estimation unit 14 may refer to information about the face frame indicating that the face frame is the same person as the face frame being determined, and determine whether the size of the face frame is increasing as the shooting time at which the image was captured progresses.

[0202] If the size of the face frame is not increasing (NO in S505), for example, if the size of the face frame has not changed (if the person has not moved for a certain period of time) or if the face frame size is decreasing (if the person is moving away from the charging zone), the person position estimation unit 14 may terminate since no charge will be made to the person. In this case, a determination result indicating that no charge was made may be recorded. If the same determination result continues for the person for a certain period of time, this corresponds to the person remaining within the gate 10 for a certain period of time, and a warning may be displayed from the gate 10.

[0203] If the size of the face frame is increasing (YES in S505), the person position estimation unit 14 determines that the person has entered a charging zone (e.g., zone C) at gate 10 (S506). The person position estimation unit 14 notifies a server (e.g., face authentication server 16 and / or passage history management server 17 in FIG. 2A) of the determination result, thereby charging the person. Then, the flow ends.

[0204] The above-described processing makes it possible to determine whether a person has entered the charging zone using an image captured by one of the cameras 11, so that even if one of the cameras 11 malfunctions or fails to capture an image due to an obstruction or the like, it is possible to properly charge a person who has passed through the gate 10.

[0205] In the above-described embodiment, the FFFA, MFFA, and PFFA zones are described as not overlapping with each other, but this is not limited thereto. Two or more zones may overlap with each other. In this case, a threshold value for the size of the face frame may be set for each zone, and if the center position of the face frame belongs to the corresponding zone and the size of the face frame exceeds the threshold value of the corresponding zone, the face frame may be determined to be located in the corresponding zone. By performing this determination in the order of PFFA, with the largest face frame size, followed by MFFA and FFFA, it is possible to appropriately determine which zone the face frame belongs to even if the center position of the face frame is located in an overlapping area of ​​the zones. For example, if the center position of the face frame belongs to both PFFA and MFFA (located in a zone where PFFA and MFFA overlap), and the size of the face frame exceeds the threshold value associated with PFFA, the face frame is determined to belong to PFFA without determining whether the size of the face frame exceeds the threshold value associated with MFFA. Furthermore, if the size of the face frame is equal to or smaller than the threshold associated with PFFA, and if the size of the face frame exceeds the threshold associated with MFFA, the face frame is determined to belong to MFFA. Similarly, if the center position of the face frame belongs to both MFFA and FFFA, it is first determined whether the face frame belongs to MFFA, and if it is determined that the face frame does not belong to MFFA, it is determined whether the face frame belongs to FFFA. Generally, the size of the face frame is largest for PFFA and smallest for FFFA, so the thresholds associated with each zone may be set to values ​​corresponding to this size relationship. However, other values ​​may be used, such as setting the same thresholds associated with all zones.

[0206] In the above-described embodiment, when a face frame is not detected in an image captured by one of the two cameras in the twin-lens system, the charging line crossing determination is performed using one camera (single-lens system). However, this is not limited to this. For example, even if the twin-lens system is operating normally, the single-lens system may be used. In this case, by controlling the system so that charging is performed when a person crosses the charging line in both systems and not charging when a person does not cross the charging line in either system, charging can be more accurately determined. Furthermore, by controlling the system so that charging is performed when a person crosses the charging line in either system, charging can be reliably performed even if the determination in either system is incorrect. Furthermore, only the single-lens system may be used.

[0207] Although the above-described embodiment describes a system for managing the passage of a person passing through gate 10, the present disclosure is not limited thereto. For example, the present disclosure may be applied to a case where there are no side walls of the passageway or a restricting portion (e.g., a door) restricting the passage of a person. For example, the present disclosure may be applied to a movement path from one zone to another zone where a person is permitted to enter in accordance with authentication processing. In this case, a camera for capturing images of the face of a person passing through the movement path may be installed, for example, on a support portion provided on the movement path. Furthermore, while the above-described embodiment illustrates an example in which a fee is charged to a person passing through gate 10, the present disclosure is not limited thereto. For example, the present disclosure may be applied to recording and / or managing passage without charging. Furthermore, for example, in a case in which entry is recorded at the time of entry, such as at a station ticket gate, and a fee is charged based on the entry record at the time of exit, the present disclosure may be applied to both entry and exit.

[0208] Furthermore, in the embodiment, the determination is made using the right and left images captured at the same time, but this is not limited to this. For example, the right and left images captured at synchronized times, but not at the same time, may be used. In this case, by determining whether the distance between the face frames is equal to or less than the threshold using the right and left images captured at the closest times to each other, a determination result similar to that of the above-described embodiment can be obtained.

[0209] Furthermore, in the present embodiment, an example in which the authentication target is a person has been described, but the present disclosure is not limited thereto. For example, the present disclosure may be applied to moving objects such as animals and vehicles. Furthermore, in the present embodiment, an example in which face authentication is performed has been described, but the present disclosure is not limited thereto. For example, the present disclosure may be applied to authentication using an ID card indicating permission to pass through a gate, and other authentication methods such as biometric authentication.

[0210] Furthermore, facial recognition may be used in combination with other authentication methods. Even if the face recognition disclosed in the above-described embodiments does not permit passage, passage may be permitted exceptionally if ID card information is entered.

[0211] Furthermore, in the above-described embodiment, it has been described that camera 11 is used for both authentication processing and tracking processing, but the present disclosure is not limited to this. For example, a configuration may be adopted in which an image captured by camera 11 is used for person tracking processing (position estimation processing) but not for face authentication processing. For example, if authentication using an ID card indicating gate passage permission or other authentication such as biometric authentication is used instead of face authentication processing, a configuration may be adopted in which an image captured by camera 11 is used for person tracking processing (position estimation processing) but not for face authentication processing.

[0212] Furthermore, in the above-described embodiment, the camera 11 is not limited to being provided on the side wall V. For example, the camera 11 may be attached to a support provided on the gate 10. The support may be, for example, a pole extending vertically from the gate 10 or an arch-shaped member provided to cover the side wall of the gate 10. Furthermore, the gate 10 is not limited to being provided with a passage formed by two side walls V. For example, the side wall V may be omitted. When the side wall V is not present, the camera 11 may be placed at a desired position using a member such as a pole. Even in a configuration without the side wall V, tracking of a person is performed based on the positional relationship of a face frame detected from an image captured by the camera 11, as in the above-described embodiment. Furthermore, person tracking using a camera capturing images from a different angle may be used in combination with the configuration described in the above-described embodiment. For example, person tracking using a ceiling camera may be used in combination. By using person tracking in combination, the accuracy of estimating the position and movement direction of a person can be improved. For example, a surveillance camera installed in a station may be used in combination as the ceiling camera.

[0213] Furthermore, in the above-described embodiment, the success or failure of the face authentication process by the face authentication function unit 13 and the success or failure of the determination of passage through the exit line by the passage management function unit 15 may be output by sound and / or image. In this case, different sounds and / or images may be output depending on the success or failure of the face authentication process and the success or failure of the determination of passage through the exit line. In this way, a person attempting to pass through gate 10 can be made aware that determinations are being made for both face authentication and passage through the charge line. Furthermore, if a person fails to pass through gate 10, it is possible to distinguish and notify whether the failure occurred at the stage of obtaining permission to pass through gate 10 (success or failure of face authentication process) or at the stage of confirming passage through the gate (success or failure of passage).

[0214] Furthermore, in the above-described embodiment, a door is used as a means for restricting passage through gate 10, but passage may be restricted directly or indirectly by other means. For example, an alarm may be sounded or a warning light may be turned on to notify a person attempting to pass through gate 10 that passage through gate 10 has been restricted. Furthermore, a notification may be sent to a terminal or the like owned by an employee near gate 10, allowing the employee to restrict passage.

[0215] Furthermore, whether or not to perform control to prevent passage, or the means for restricting passage, may be switched depending on the congestion situation. For example, in an environment where it is dangerous to prevent or restrict people's passage, such as when a large number of people are entering or exiting, passage through gate 10 may not be prevented, and information indicating that an unauthorized passage has occurred may be recorded. In this case, a facial image or facial recognition results of the person who has illegally passed through may be recorded in association with the information indicating the unauthorized passage. This makes it possible to later track the person who has illegally passed through and claim compensation for the right of passage, etc.

[0216] Furthermore, in the above-described embodiment, the passage management system 1 managed both entry to and exit from a facility at the entrances and exits of facilities such as airports, train stations, and event venues, but it is also possible to manage either entry to or exit from a facility at the entrance or exit, and not manage the other.

[0217] The present disclosure can be realized in software, hardware, or software in conjunction with hardware.

[0218] Each functional block used in the description of the above embodiments may be partially or entirely realized as an LSI, which is an integrated circuit, and each process described in the above embodiments may be partially or entirely controlled by a single LSI or a combination of LSIs. The LSI may be composed of individual chips, or may be composed of a single chip that includes some or all of the functional blocks. The LSI may have data input and output. Depending on the degree of integration, the LSI may be called an IC, system LSI, super LSI, or ultra LSI.

[0219] The integrated circuit method is not limited to LSI, but may be realized by a dedicated circuit, a general-purpose processor, or a dedicated processor. Also, a field programmable gate array (FPGA) that can be programmed after LSI manufacturing, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells within the LSI, may be used. The present disclosure may be realized as digital processing or analog processing.

[0220] Furthermore, if an integrated circuit technology that can replace LSI emerges due to advances in semiconductor technology or other derivative technologies, it is natural that such technology may be used to integrate functional blocks. The application of biotechnology, etc. is also a possibility.

[0221] The present disclosure may be implemented in any type of apparatus, device, or system (collectively referred to as a communications apparatus) that has a communications function. The communications apparatus may include a wireless transceiver and processing / control circuitry. The wireless transceiver may include a receiver and a transmitter, or both functions. The wireless transceiver (transmitter and receiver) may include a radio frequency (RF) module and one or more antennas. The RF module may include an amplifier, an RF modulator / demodulator, or the like. Non-limiting examples of communication devices include telephones (e.g., cell phones, smartphones), tablets, personal computers (PCs) (e.g., laptops, desktops, notebooks), cameras (e.g., digital still / video cameras), digital players (e.g., digital audio / video players), wearable devices (e.g., wearable cameras, smartwatches, tracking devices), game consoles, digital book readers, telehealth / telemedicine devices, communication-enabled vehicles or mobile transportation (e.g., cars, airplanes, ships), and combinations of the above devices.

[0222] Communications equipment is not limited to portable or mobile equipment, but also includes non-portable or fixed equipment, devices, and systems of any kind, such as smart home devices (such as appliances, lighting equipment, smart meters or metering devices, control panels, etc.), vending machines, and any other "things" that may exist on an IoT (Internet of Things) network.

[0223] Furthermore, in recent years, in the field of IoT (Internet of Things) technology, CPS (Cyber ​​Physical Systems) has been attracting attention as a new concept that creates new added value by linking information between physical space and cyberspace. This CPS concept can also be adopted in the above-mentioned embodiments.

[0224] That is, as a basic configuration of a CPS, for example, an edge server located in physical space and a cloud server located in cyberspace can be connected via a network, and processing can be distributed and performed by processors installed on both servers. Here, it is preferable that each piece of processing data generated on the edge server or cloud server is generated on a standardized platform, and the use of such a standardized platform can improve the efficiency of building a system that includes a variety of sensor groups and IoT application software.

[0225] Communications include data communications via cellular systems, wireless LAN systems, communications satellite systems, etc., as well as data communications via combinations of these.

[0226] A communications apparatus also includes devices such as controllers and sensors connected or coupled to a communications device that performs the communications functions described in this disclosure, such as controllers and sensors that generate control and data signals used by the communications device to perform the communications functions of the communications apparatus.

[0227] The communication apparatus also includes infrastructure facilities, such as base stations, access points, and any other apparatus, device, or system that communicates with or controls the various apparatuses listed above, but are not limited to these.

[0228] Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications or alterations within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure. Furthermore, the components of the above-described embodiments may be combined in any manner without departing from the spirit of the disclosure.

[0229] Although specific examples of the present disclosure have been described in detail above, these are merely examples and do not limit the scope of the claims. The technology described in the claims includes various modifications and alterations of the specific examples exemplified above. [Industrial Applicability]

[0230] An embodiment of the present disclosure is suitable for a face authentication system. [Explanation of symbols]

[0231] 1. Currency Management System 10 Gates 11 Camera 13 Face recognition function unit 131 Camera control unit 132 Face matching processing unit 14 Person position estimation section 141 Person tracking processing unit 15 Passage management function section 151 Passing management state transition processing unit 152 History Management Department 153 History DB 16 Face Recognition Server 17 Passing history management server

Claims

1. a detection unit that detects a first facial image area included in a first image obtained by photographing a person entering a gate from a first direction and a second facial image area included in a second image obtained by photographing the person from a second direction different from the first direction; an estimation unit that determines that the person has passed a position defined by the gate when a difference between the photographing time of the first image and the photographing time of the second image is within a predetermined allowable range, and when a horizontal distance between a representative point of the first face image area and a representative point of the second face image area becomes equal to or less than a threshold, or when polarity of the horizontal distance from the representative point of the first face image area to the representative point of the second face image area is inverted; An information processing device comprising:

2. a face matching processing unit that matches the face of the person before the person passes the specified position; The information processing device includes: outputting different sounds or images depending on whether the face of the person has been successfully matched or whether it has been determined that the person has passed the specified position; The information processing device according to claim 1 .

3. the detection unit sets an area in the first image where a representative point of the first face image area is predicted to first appear in response to the person entering the gate; When the estimation unit determines that a plurality of representative points of the first face image region are included in the area, the estimation unit stops estimating the position of the person. The information processing device according to claim 1 .

4. the representative point of the first face image area and the representative point of the second face image area are the centers of the first face image area and the second face image area, respectively; The information processing device according to claim 1 .

5. an authentication device that performs authentication processing of a person entering a gate using at least one of a first image obtained by photographing the person from a first direction and a second image obtained by photographing the person from a second direction different from the first direction; Detecting a first face image area included in the first image and a second face image area included in the second image; an information processing device that determines that the person has passed a position defined by the gate when a difference between the photographing time when the first image and the photographing time when the second image are photographed is within a predetermined allowable range, and when a horizontal distance between a representative point of the first face image area and a representative point of the second face image area becomes equal to or less than a threshold, or when polarity of the horizontal distance from the representative point of the first face image area to the representative point of the second face image area is inverted; An information processing system comprising:

6. The information processing device detecting a first facial image area included in a first image obtained by photographing a person entering a gate from a first direction and a second facial image area included in a second image obtained by photographing the person from a second direction different from the first direction; When the difference between the photographing time when the first image was photographed and the photographing time when the second image was photographed is within a predetermined allowable range, if the horizontal distance between the representative point of the first face image area and the representative point of the second face image area becomes equal to or less than a threshold value, or if the polarity of the horizontal distance from the representative point of the first face image area to the representative point of the second face image area is inverted, it is determined that the person has passed the position defined by the gate. Estimation method.

Images