Information processing device, person verification system, and information processing method

Abstract

Provided is an information processing device comprising a processor and a communication device. The communication device receives a captured image of a person passing through a prescribed passageway. The processor uses the captured image to track the person, and when the person is positioned in a verification zone that is set on the entrance side of the passageway, the processor uses the captured image of the person to verify the person. When a person for whom verification has failed is positioned in a passage determination zone set on the exit side of the passageway, the processor displays, on a prescribed monitor, information indicating that the verification has failed. When a person for whom verification has failed has exited to the exit side from the passage determination zone, the processor displays, on the monitor, information indicating that said person has made unauthorized passage.

Description

Information processing device, person matching system, and information processing method

[0001] The present disclosure relates to an information processing device, a person matching system, and an information processing method.

[0002] There is known a technique for detecting people from an image captured by a camera and tracking each of the detected people (for example, Patent Document 1).

[0003] Japanese Patent Application Publication No. 2024-018522

[0004] Patent Literature 1 discloses that when a person who fails verification is located in a passage judgment zone, a message indicating unauthorized passage is displayed. However, there are cases where an innocent person sees the unauthorized passage message displayed in the passage judgment zone, recognizes that they do not have the right to pass, and makes a U-turn or other such action. On the other hand, there are also cases where a malicious person ignores the unauthorized passage message and attempts to pass through. For this reason, it is expected that, for example, a monitor will hesitate to immediately take action such as stopping the person when the unauthorized passage message is displayed. In other words, it is difficult for a monitor to determine when to start taking action such as stopping the unauthorized passerby.

[0005] An object of the present disclosure is to provide a technology that makes it easy to understand, for example, the timing when an observer should begin taking action to stop an unauthorized pedestrian.

[0006] One aspect of the present disclosure provides an information processing device including a processor and a communication device, wherein the communication device receives a photographed image of a person passing through a specified passageway, the processor uses the photographed image to track the person, and if the person is located in a matching zone set on the entrance side of the passageway, the processor matches the person using the photographed image of the person, and if the person who fails the matching is located in a passage determination zone set on the exit side of the passageway, the processor displays information indicating that the matching has failed on a specified monitor, and if the person who fails the matching exits from the passage determination zone to the exit side, the processor displays information indicating that the person is an unauthorized passerby on the monitor.

[0007] One aspect of the present disclosure provides a person matching system including a photographing device that generates a photographed image of a person passing through a specified passageway, a monitor, and an information processing device, wherein the information processing device receives the photographed image from the photographing device, tracks the person using the photographed image, and if the person is located in a matching zone set on the entrance side of the passageway, matches the person using the photographed image of the person, and if the person who has failed the matching is located in a passage determination zone set on the exit side of the passageway, displays information indicating that the matching has failed on the monitor, and if the person who has failed the matching exits from the passage determination zone to the exit side, displays information indicating that the person is an unauthorized passerby on the monitor.

[0008] One aspect of the present disclosure provides an information processing method that receives a photographed image of a person passing through a specified passageway, tracks the person using the received photographed image, and if the person is located in a matching zone set on the entrance side of the passageway, matches the person using the photographed image of the person, and if the person who fails the matching is located in a passage determination zone set on the exit side of the passageway, displays information indicating that the matching has failed on a specified monitor, and if the person who fails the matching exits from the passage determination zone to the exit side, displays information indicating that the person is an unauthorized passerby on the monitor.

[0009] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0010] According to the present disclosure, for example, it becomes easier to understand the timing when an observer should start taking action to stop an unauthorized pedestrian.

[0011] FIG. 7 is a diagram for explaining an overview of a gate system according to the first embodiment; FIG. 8 is a diagram showing an example of display of a matching result on the first display device or the second display device according to the first embodiment; FIG. 9 is a block diagram showing an example of the configuration of a gate system according to the first embodiment; FIG. 10 is a diagram for explaining a face image and a body image according to the first embodiment; FIG. 1 is a diagram illustrating a face matching zone and a passage determination zone in a detection area where a person is detected; FIG. 2 is a diagram illustrating an example of a display pattern in a first display device according to embodiment 1; FIG. 3 is a diagram illustrating a method for detecting, from a person information table, personal information that is presumed to be the same person as the person indicated by body detection information according to embodiment 1; FIG. 4 is a diagram illustrating a method for detecting, from a person information table, personal information that is presumed to be the same person as the person indicated by body detection information according to embodiment 1;FIG. 1 is a diagram for explaining a case where an unauthorized passerby image is generated without processing. FIG. 2 is a diagram for explaining a first method for processing a prohibited person area according to the second embodiment. FIG. 3 is a diagram for explaining a modified example of the first method for processing a prohibited person area according to the second embodiment. FIG. 4 is a diagram for explaining a processing method when a plurality of authorized person areas overlap at least a part below a prohibited person area according to the second embodiment. FIG. 1 is a diagram showing an example of displaying a determination result of a person passing through a zone according to the third embodiment. FIG. 2 is a diagram showing an example of the movement of a person making a U-turn in a zone according to the third embodiment. FIG. 3 is a diagram showing an example of displaying a determination result of a person making a U-turn in a zone according to the third embodiment.

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with appropriate reference to the drawings. However, more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0013] (First Embodiment) <Outline of Gate System> Fig. 1 is a diagram for explaining an outline of a gate system 9 according to the first embodiment. Fig. 2 is a diagram showing an example of display of a matching result on the first display device 31 or the second display device 32 according to the first embodiment.

[0014] The gate system 9 is a system that collates a facial image of a person 1 and determines whether the person 1 has the right to pass through the gate. The gate system 9 includes a first gate 11 and a second gate 12. The first gate 11 and the second gate 12 are installed a predetermined distance apart in a passage 2 through which the person 1 moves. In this embodiment, the direction from the second gate 12 toward the first gate 11 is defined as a first direction, and the direction from the first gate 11 toward the second gate 12 is defined as a second direction. In this embodiment, the area between the first gate 11 and the second gate 12 in the passage 2 is referred to as a person detection area 3. Note that, although the first embodiment describes a configuration in which the gate system 9 includes two gates, the gate system 9 may also be configured to include only one gate (e.g., the first gate 11) and to collate a person 1 heading in one direction (e.g., the first direction).

[0015] In this embodiment, for convenience of explanation, the axis along the first or second direction is referred to as the Y-axis, the axis perpendicular to the Y-axis and extending in the height direction is referred to as the Z-axis, and the axis perpendicular to the Y-axis and the Z-axis is referred to as the X-axis. The direction of the X-axis may also be referred to as the width direction of the gate.

[0016] The first gate 11 includes a first imaging device 21 capable of capturing an image of at least a portion of the detection area 3, and a first display device 31 having a screen facing the second gate 12. In FIG. 1 , the first gate 11 includes three first imaging devices 21, but the first gate 11 may include one or two first imaging devices 21, or four or more first imaging devices 21. Also, in FIG. 1 , the first gate 11 includes one first display device 31, but the first gate 11 may include two or more first display devices 31.

[0017] The second gate 12 is equipped with a second imaging device 22 capable of capturing an image of at least a portion of the detection area 3, and a second display device 32 having a screen facing the first gate 11. In Fig. 1, the second gate 12 is equipped with three second imaging devices 22, but the second gate 12 may be equipped with one or two second imaging devices 22, or may be equipped with four or more second imaging devices 22. Furthermore, in Fig. 1, the second gate 12 is equipped with one second display device 32, but the second gate 12 may be equipped with two or more second display devices 32.

[0018] In this embodiment, the first direction is the direction of entry into a facility or the like where the person has the right to enter or exit, and the second direction is the direction of exit from the facility or the like. However, the first direction may also be the direction of exit from a facility or the like, and the second direction may also be the direction of entry into the facility or the like.

[0019] The gate system 9 compares the facial image of the person 1 moving in the first direction captured by the first photographing device 21 at the first gate 11 with each pre-registered facial image for comparison, and displays the comparison result on the first display device 31, as illustrated in Figure 2.

[0020] The gate system 9 compares the facial image of the person 1 moving in the second direction captured by the second photographing device 22 at the second gate 12 with each pre-registered facial image for comparison, and displays the comparison result on the second display device 32, as illustrated in Figure 2.

[0021] The gate width of the first gate 11 and the second gate 12, or the width in the X direction of the detection area 3, may be wide enough for multiple people 1 to pass through side by side. As a result, the gate system 9 according to this embodiment allows more people 1 to enter and exit smoothly compared to a gate system with a gate width wide enough for only one person to pass through. For example, in a gate system with a gate width wide enough for one person, if the verification of a person 1 ahead fails, the following people 1 will be held up. However, in a gate system 9 according to this embodiment with a gate width wide enough for multiple people 1 to pass through side by side, even if the verification of a person 1 ahead fails, the following people 1 can simply move around the person 1 ahead who failed to be verified, and thus can pass through the gate smoothly without being held up. In other words, the gate system 9 according to this embodiment can prevent congestion when entering or exiting a facility, etc. The gate system 9 according to this embodiment will be described in detail below.

[0022] <Configuration of Gate System> Fig. 3 is a block diagram showing an example of the configuration of the gate system 9 according to embodiment 1. Fig. 4 is a diagram for explaining face images and body images according to embodiment 1.

[0023] The gate system 9 includes a first gate 11 and a second gate 12. The first gate 11 includes a first imaging device 21 and a first display device 31. The second gate 12 includes a second imaging device 22 and a second display device 32. The first gate 11, the first imaging device 21, the first display device 31, the second gate 12, the second imaging device 22, and the second display device 32 may be collectively referred to as a gate device 10.

[0024] The gate system 9 further includes a first edge device 41, a second edge device 42, a management device 50, and a face matching device 60. The first image capturing device 21 is connected to the first edge device 41 via an electrical cable (e.g., a USB cable or an Ethernet (registered trademark) cable). The first display device 31 is connected to the first edge device 41 via an electrical cable (e.g., an HDMI (registered trademark) cable). The second image capturing device 22 is connected to the second edge device 42 via an electrical cable (e.g., a USB cable or an Ethernet cable). The second display device 32 is connected to the second edge device 42 via an electrical cable (e.g., an HDMI cable). The first edge device 41 and the second edge device 42 are connected to the management device 50 via a communication network 4. The management device 50 is connected to the face matching device 60 via the communication network 4. Examples of the communication network 4 include a wired LAN, a wireless LAN, a mobile communication network (e.g., LTE, 4G, 5G), the Internet, and / or a VPN.

[0025] The first imaging device 21 captures an image of the detection area 3 to generate a captured image, and transmits the captured image to the first edge device 41. The second imaging device 22 captures an image of the detection area 3 to generate a captured image, and transmits the captured image to the second edge device 42. The captured image may be a still image or a moving image.

[0026] As shown in FIGS. 4( a) and 4(b), the first edge device 41 detects the entire body of the person 1 from the photographed image received from the first image capturing device 21. In this embodiment, the image of the entire body of the person 1 detected from the photographed image is referred to as a body image 200. The first edge device 41 transmits the detected body image 200 to the management device 50. Furthermore, as shown in FIG. 4(a), the first edge device 41 detects the face portion of the person 1 from the photographed image received from the first image capturing device 21. In this embodiment, the image of the face portion of the person 1 detected from the photographed image is referred to as a face image 220. The first edge device 41 transmits the detected face image 220 to the management device 50.

[0027] As shown in Figures 4(a) and 4(b), the second edge device 42 detects the entire body of the person 1 from the captured image received from the second image capturing device 22. The second edge device 42 transmits a body image 200 detected from the captured image to the management device 50. Also, as shown in Figure 4(a), the second edge device 42 detects the face portion of the person 1 from the captured image received from the second image capturing device 22. The second edge device 42 transmits a face image 220 detected from the captured image to the management device 50. Details of the processing performed by the first edge device 41 and the second edge device 42 will be described later.

[0028] The management device 50 generates person information 100 (see FIG. 6 ) corresponding to the person 1 located in the detection area 3 based on the body image 200 or face image 220 received from the first edge device 41, or the body image 200 or face image 220 received from the second edge device 42. The management device 50 may generate person information 100 for each person 1 detected in the detection area 3. Details of the person information 100 will be described later (see FIG. 6 ). The management device 50 transmits a face matching request including the face image 220 to the face matching device 60.

[0029] The face matching device 60 stores in advance a face image for matching of a person 1 who has the right to pass through a gate (for example, to enter or exit). For example, the face matching device 60 stores a face image for matching of a person 1 who has already registered to enter the facility. The face matching device 60 may manage the face image for matching by linking it to the name, age, sex, credit card information, etc. of the person 1.

[0030] Furthermore, the face matching device 60 matches the face image 220 received from the management device 50 with each of the face images for matching. If the face matching device 60 is successful in matching with any of the registered face images for matching, it generates face matching result information including a successful matching result and transmits it to the management device 50. At this time, the face matching device 60 may perform billing in real time using credit card information linked to the successfully matched face image for matching. If the face matching device 60 fails to match with any of the registered face images for matching, it generates face matching result information including a failed matching result and transmits it to the management device 50. Note that the matching between the face image 220 and the face images for matching may be achieved by a known face image matching technology. Also, "matching" may be read as "authentication."

[0031] The management device 50 receives face matching result information from the face matching device 60. The management device 50 transmits person matching information including face matching result information for the face image 220 of the person 1 moving in the first direction to the first edge device 41, and transmits person matching information including face matching result information for the face image 220 of the person 1 moving in the second direction to the second edge device 42.

[0032] When the first edge device 41 receives person matching information from the management device 50, it displays the face matching result information included in the person matching information on the first display device 31, as shown in Fig. 2. This allows a person 1 located in the detection area 3 and moving in the first direction to view the screen of the first display device 31 located in the direction of movement and know his / her own matching result. In other words, the first edge device 41 allows a person 1 moving in the first direction to visually recognize the matching result of the person 1.

[0033] When the second edge device 42 receives person matching information from the management device 50, it displays the face matching result information included in the person matching information on the second display device 32, as shown in FIG. 2. This allows person 1 located in the detection area 3 and moving in the second direction to view the screen of the second display device 32 located at the end of the moving direction and know his / her matching result. In other words, the second edge device 42 allows person 1 moving in the second direction to visually confirm the matching result for that person 1. Below, the processes performed by the first edge device 41, the second edge device 42, and the management device 50 will be described in detail.

[0034] <Detection Process in First Edge Device> Fig. 5 is a flowchart showing an example of the detection process performed by the first edge device 41 according to embodiment 1. Note that, for the detection process performed by the second edge device 42 according to embodiment 1, in the following description with reference to Fig. 5, the "first image capturing device 21" can be read as the "second image capturing device 22."

[0035] The first edge device 41 receives a captured image from the first image capturing device 21 (S101).

[0036] The first edge device 41 receives a distance image from the first image capturing device 21 (S102). In this way, the first image capturing device 21 may generate a distance image indicating the distance to the person 1 or the like present in the detection area 3. The same applies to the second image capturing device 22. The distance image may be generated using any known technology, such as a stereo camera or a distance measuring sensor (e.g., LiDAR (Light Detection And Ranging)). Note that in this embodiment, the processing of step S102 is performed after the processing of step S101, but this is not limiting and the procedures may be reversed.

[0037] 4A, the first edge device 41 detects a facial image 220 of the person 1 from the captured image (S103). The detection of the facial image 220 of the person may be realized using any known technology, such as pattern matching or AI (Artificial Intelligence). The AI ​​may also be an image recognition technology using deep learning (e.g., CNN (Convolutional Neural Network)).

[0038] The first edge device 41 determines whether or not the face image 220 was detected from the captured image in step S103 (S104). If the face image 220 was not detected (S104: NO), the first edge device 41 proceeds to step S108.

[0039] If the face image 220 is detected (S104: YES), the first edge device 41 uses the distance image to calculate three-dimensional coordinates (hereinafter referred to as face coordinates 221 (see FIG. 4)) of the detected face image 220 (S105). For example, as shown in FIG. 4(a), the first edge device 41 calculates the center point of the face image 220 as the face coordinates 221.

[0040] The first edge device 41 generates face detection information (S106), for example, including an ID (hereinafter referred to as a photographing device ID) for identifying the first photographing device 21 that is the sender of the photographed image, the face image 220, the size of the face image 220 (hereinafter referred to as a face image size), face coordinates 221, and the photographing time of the photographed image.

[0041] The first edge device 41 transmits the generated face detection information to the management device 50 (S107), and the process proceeds to the next step S108.

[0042] As shown in Fig. 4(a) or 4(b), the first edge device 41 detects a body image 200 of the person 1 from the captured image (S108). The detection of the body image 200 of the person 1 may be realized using any known technology, such as pattern matching or AI. The AI ​​may also be an image recognition technology using deep learning (e.g., CNN).

[0043] The first edge device 41 determines whether or not the body image 200 was detected from the captured image in step S108 (S109). If the body image 200 was not detected (S109: NO), the first edge device 41 returns the process to step S101.

[0044] If the body image 200 is detected (S109: YES), the first edge device 41 uses the distance image to calculate three-dimensional coordinates (hereinafter referred to as body coordinates 201 (see FIG. 4)) of the detected body image 200 (S110). For example, as shown in FIGS. 4(a) and 4(b), the first edge device 41 calculates the center point of the body image 200 as the body coordinates 201.

[0045] The first edge device 41 generates body detection information (S111). For example, the first edge device 41 generates body detection information including an image capturing device ID, a body ID for identifying the body image 200, the body image 200, body coordinates 201, and the image capturing time of the captured image.

[0046] The first edge device 41 transmits the generated body detection information to the management device 50 (S112), and the process returns to step S101. Note that in this embodiment, the process of detecting a face image from the captured image (S103 to S107) is performed first, followed by the process of detecting a body image from the captured image (S108 to S112), but this is not limiting, and the steps may be reversed.

[0047] In this way, by performing the process of detecting the facial image 220 and / or the body image 200 of the person 1 from the captured image and the process of calculating the facial coordinates 221 and / or the body coordinates 201 at the first edge device 41 or the second edge device 42, the processing load can be distributed compared to when these processes are performed at the management device 50. Note that if distribution of the processing load is not a consideration, the gate system 9 does not need to include the first edge device 41 and the second edge device 42. In this case, the first image capturing device 21, the first display device 31, the second image capturing device 22, and the second display device 32 may be directly connected to the management device 50. Then, the management device 50 may perform the process of detecting the facial image 220 and / or the body image 200 of the person 1 from the captured image and the process of calculating the facial coordinates 221 and / or the body coordinates 201.

[0048] <Personal Information> FIG. 6 shows an example of the configuration of personal information 100 according to the first embodiment.

[0049] The person information 100 is generated for each person 1 detected in the detection area 3. The person information 100 includes, as information or data, a tracking ID 101, a first image capture time 102, an image update time 103, a body image 104, a body image update time 105, body coordinates 106, a face image 107, a face image update time 108, face coordinates 109, a face matching ID 110, a matching status 111, and a movement direction 112.

[0050] The tracking ID 101 is set to an ID used to track the same person in the detection area 3. The tracking ID 101 can also be said to be an ID for identifying the person information 100.

[0051] The first image capturing time 102 is set to the time when the body image 104 or face image 107 included in the personal information 100 was first captured. The first image capturing time can also be said to be the time when the personal information 100 was first generated.

[0052] The image update time 103 is set to the time when the body image 104 or the face image 107 included in the personal information 100 was last updated. The image update time 103 can also be said to be the time when the personal information 100 was last updated.

[0053] The body image 104 is set to a body image 200 detected from a photographed image of the person 1 indicated by the personal information 100. The body image 104 may be updated as needed to the body image 200 detected from the latest photographed image.

[0054] The body image update time 105 is set to the time when the body image 104 included in the personal information 100 was last updated.

[0055] The body coordinates 106 are set to the body coordinates 201 calculated based on the body image 104. The body coordinates 106 may be expressed as three-dimensional coordinates (X coordinate, Y coordinate, Z coordinate) in the detection area 3.

[0056] The face image 107 is set to a face image 220 detected from a photographed image of the person 1 indicated by the personal information 100. The face image 107 may be updated as needed to the face image 220 detected from the latest photographed image.

[0057] The face image update time 108 is set to the time when the face image 107 included in the person information 100 was last updated.

[0058] The face coordinates 221 calculated based on the face image 107 are set as the face coordinates 109. The face coordinates 109 may be expressed as three-dimensional coordinates (X coordinate, Y coordinate, Z coordinate) in the detection area 3.

[0059] An ID assigned to a successfully matched face image for matching is set in face matching ID 110. In other words, face matching ID 110 can also be said to be an ID that certifies person 1 corresponding to face image 107. If no successfully matched face image for matching exists (i.e., if face matching fails), NULL may be set in face matching ID 110.

[0060] The matching status 111 is set to the matching status for the face image 107. For example, the matching status 111 is set to one of "before matching request", "matching in progress", "matching successful", and "matching failed".

[0061] The movement direction 112 is set to the movement direction of the person 1 indicated by the person information 100. For example, the movement direction is set to either the first direction or the second direction.

[0062] The management device 50 manages a plurality of pieces of personal information 100 as a personal information table (not shown). The personal information table may be stored in a memory 1002 or a storage 1003 (see FIG. 29 ) provided in the management device 50, or may be stored in a device different from the management device 50.

[0063] <Overall Processing in Management Device> FIG. 7 is a flowchart showing an example of overall processing performed by the management device 50 according to the first embodiment.

[0064] The management device 50 determines whether or not face detection information (see step S107 in FIG. 5) has been received from the first edge device 41 or the second edge device 42 (S201).

[0065] When the management device 50 receives face detection information (S201: YES), it executes face detection information processing (S202). Note that the face detection information processing will be described in detail later (see FIG. 8).

[0066] If the management device 50 has not received face detection information (S202: NO), it determines whether it has received body detection information (see step S112 in Figure 5) from the first edge device 41 or the second edge device 42 (S203).

[0067] When the management device 50 receives the body detection information (S203: YES), the management device 50 executes body detection information processing (S204). Details of the body detection information processing will be described later (see FIG. 9).

[0068] If the management device 50 has not received the body detection information (S203: NO), it determines whether or not it has received face matching result information (S205). The face matching result information is information returned from the face matching device 60 in response to a face matching request transmitted from the management device 50 to the face matching device 60 in Fig. 8, which will be described later, and includes the result of face matching (for example, matching success or failure).

[0069] When the management device 50 receives the face matching result information (S205: YES), it executes face matching result information processing (S206). Note that the face matching result information processing will be described in detail later (see FIG. 11).

[0070] If the management device 50 has not received the face matching result information (S205: NO), the process returns to step S201.

[0071] After executing the face detection information processing (S202) or the body detection information processing (S204), the management device 50 executes post-movement processing (S210) and returns the processing to step S201. Details of the post-movement processing will be described later (see FIG. 10 ). Note that in this embodiment, the processing of steps S201, S203, and S205 is performed in stages, but this is not limiting, and the processing of steps S201, S203, and S205 may be interchanged or may be performed simultaneously in parallel.

[0072] <Face Detection Information Processing in Management Apparatus> FIG. 8 is a flowchart showing an example of the face detection information processing (S202) shown in FIG.

[0073] The management device 50 detects, from the person information table, the person information 100 corresponding to the face detection information received from the first edge device 41 or the second edge device 42 (S301). In this embodiment, the person information 100 corresponding to the face detection information is the person information 100 that is presumed to be the same person as person 1 indicated by the face detection information. Note that the method for detecting, from the person information table, the person information 100 that is presumed to be the same person as person 1 indicated by the face detection information will be described in detail later (see FIGS. 15A and 15B ).

[0074] The management device 50 determines whether or not the personal information 100 has been detected from the personal information table in step S301 (S302).

[0075] Next, a case where it is determined in step S302 that the personal information 100 cannot be detected from the personal information table (S302: NO) will be described.

[0076] The management device 50 generates new personal information 100 based on the face detection information (S303). At this time, the management device 50 sets the following contents in the new personal information 100: - A new tracking ID is set in the tracking ID 101 of the personal information 100 - A face image 220 included in the face detection information is set in the face image 107 of the personal information 100 - Face coordinates 221 included in the face detection information are set in the face coordinates 109 of the personal information 100 - The shooting time included in the earliest face detection information (i.e., the earliest acquired shooting time) is set in the first image shooting time 102 of the personal information 100 - The matching status 111 of the personal information 100 is set to "before matching request"

[0077] The management device 50 detects the movement direction of the person 1 corresponding to the face detection information, and sets the detected movement direction as the movement direction 112 of the person information 100 generated in step S303 (S304). Note that the method for detecting the movement direction of a person will be described in detail later (see FIG. 16).

[0078] The management device 50 newly registers the personal information 100 generated and set in steps S303 and S304 in the personal information table (S305).

[0079] The management device 50 transmits a face matching request including the face image 107 and the tracking ID 101 to the face matching device 60, and updates the matching status 111 of the person information 100 to "matching in progress" (S306). Then, the management device 50 completes this process.

[0080] Next, a case where it is determined in step S302 that the personal information 100 has been detected from the personal information table (S302: YES) will be described.

[0081] The management device 50 updates the content of the person information 100 detected in step S301 based on the face detection information (S310). For example, the management device 50 updates the detected person information 100 as follows: Updates the face coordinates 109 of the person information 100 to the face coordinates 221 included in the face detection information. Updates the image update time 103 and face image update time 108 of the person information 100 to the shooting time included in the face detection information.

[0082] The management device 50 determines whether the matching status 111 of the detected person information 100 is "before matching request" or "matching failed" (S311). If the matching status 111 is not "before matching request" or "matching failed" (S311: NO), the management device 50 completes this process.

[0083] If the matching status 111 is "before matching request" or "matching failed" (S311: YES), the management device 50 transmits a face matching request including the face image 107 and the tracking ID 101 to the face matching device 60, and updates the matching status 111 of the person information 100 to "matching in progress" (S312). Then, the management device 50 completes this process.

[0084] <Body Detection Information Processing in Control Device> FIG. 9 is a flowchart showing an example of the body detection information processing (S204) shown in FIG.

[0085] The management device 50 detects, from the person information table, personal information 100 corresponding to the body detection information received from the first edge device 41 or the second edge device 42 (S401). In this embodiment, the personal information 100 corresponding to the body detection information is personal information 100 that is presumed to be the same person as person 1 indicated by the body detection information. Details of a method for detecting, from the person information table, personal information 100 that is presumed to be the same person as person 1 indicated by the body detection information will be described later (see FIGS. 15A and 15B ).

[0086] The management device 50 determines whether or not the personal information 100 has been detected from the personal information table in step S401 (S402).

[0087] Next, a case where it is determined in step S402 that the personal information 100 cannot be detected from the personal information table (S402: NO) will be described.

[0088] The management device 50 generates new personal information 100 based on the received body detection information (S403). At this time, the management device 50 sets the following contents in the new personal information 100: - Sets a new tracking ID in the tracking ID 101 of the personal information 100 - Sets a body image 200 included in the body detection information in the body image 104 of the personal information 100 - Sets body coordinates 202 included in the body detection information in the body coordinates 106 of the personal information 100 - Sets the shooting time included in the earliest body detection information (i.e., the earliest acquired shooting time) in the first image shooting time 102 of the personal information 100

[0089] The management device 50 detects the moving direction of the person 1 corresponding to the body detection information, and sets the detected moving direction as the moving direction 112 of the person information 100 generated in step S403 (S404). Note that the method for detecting the moving direction of the person 1 will be described in detail later (see FIG. 16 ).

[0090] The management device 50 newly registers the personal information 100 generated and set in steps S403 and S404 in the personal information table (S405), and the management device 50 then completes this process.

[0091] Next, a case where it is determined in step S402 that the personal information 100 has been detected from the personal information table (S402: YES) will be described.

[0092] The management device 50 updates the content of the person information 100 detected in step S401 based on the body detection information (S410). For example, the management device 50 updates the detected person information 100 as follows: Updates the body coordinates 106 of the person information 100 to the body coordinates 202 included in the body detection information. Updates the image update time 103 and body image update time 105 of the person information 100 to the body image capture time included in the body detection information.

[0093] Then, the management device 50 completes this process.

[0094] <Post-Movement Processing in Management Apparatus> FIG. 10 is a flowchart showing an example of the post-move processing (S210) shown in FIG.

[0095] The management device 50 generates person movement information based on the target person information (S501). In this embodiment, the target person information is the person information 100 detected from the person information table or the newly generated person information 100 in the face detection information processing shown in Fig. 8, or the person information 100 detected from the person information table or the newly generated person information 100 in the body detection information processing shown in Fig. 9. At this time, the management device 50 sets the following information in the person movement information: Tracking ID Zone information Face image (however, if there is no face image, a body image) Matching status Face matching ID

[0096] Here, the zone information is information indicating whether the person 1 corresponding to the target person information is located in the face matching zone 301 or the passage determination zone 302 in the detection area 3. The face matching zone 301 and the passage determination zone 302 will be described in detail later (see FIG. 13 ).

[0097] The management device 50 determines whether the movement direction of the target person information is the first direction or the second direction (S502).

[0098] If the movement direction is the first direction (S502: first direction), the management device 50 transmits the person movement information generated in step S501 to the first edge device 41 (S503), and completes this process.

[0099] If the movement direction is the second direction (S502: second direction), the management device 50 transmits the person movement information generated in step S501 to the second edge device 42 (S504), and completes this process.

[0100] <Face Matching Result Information Processing in Control Device> FIG. 11 is a flowchart showing an example of the face matching result information processing (S206) shown in FIG.

[0101] The management device 50 sets the content of the face matching result information received from the face matching device 60 to the personal information 100 corresponding to the face matching result information in the personal information table (S601). For example, the management device 50 sets the content of the face matching result information to the personal information 100 as follows: - Sets the face matching ID of the face matching result information to the face matching ID 110 of the personal information 100 - Sets the matching result of the face matching result information (matching success or matching failure) to the matching status 111 of the personal information 100

[0102] The management device 50 generates person matching information based on the person information 100 set in step S601 (S602). At this time, the management device 50 sets the following information in the person matching information: Tracking ID Zone information Face image Matching status Face matching ID

[0103] The management device 50 determines whether the movement direction 112 of the person information 100 set in step S601 is the first direction or the second direction (S603).

[0104] If the movement direction 112 is the first direction (S602: first direction), the management device 50 transmits the person matching information generated in step S602 to the first edge device 41 (S604), and completes this process.

[0105] If the movement direction 112 is the second direction (S602: second direction), the management device 50 transmits the person matching information generated in step S602 to the second edge device 42 (S605), and completes this process.

[0106] <Display Processing in First Edge Device> Fig. 12 is a flowchart showing an example of display processing performed by the first edge device 41 according to embodiment 1. Note that, for the display processing performed by the second edge device 42 according to embodiment 1, in the following description with reference to Fig. 12, "first edge device 41" may be read as "second edge device 42," "first display device 31" may be read as "second display device 32," and "first direction" may be read as "second direction."

[0107] The first edge device 41 determines whether or not it has received person movement information or person matching information from the management device 50 (S701). If the first edge device 41 has not received either person movement information or person matching information (S701: NO), it returns the process to step S701. If the first edge device 41 has received either person movement information or person matching information (S701: YES), it performs the following process.

[0108] The first edge device 41 determines whether a face image or a body image corresponding to the tracking ID included in the received person movement information or person matching information is displayed on the first display device 31 (S702).

[0109] If the face image or body image corresponding to the tracking ID is not displayed on the first display device 31 (S702: NO), the first edge device 41 newly displays the face image or body image and the matching status included in the person movement information or person matching information on the first display device 31 (S703). Then, the first edge device 41 returns the process to step S701.

[0110] If a face image or body image corresponding to the tracking ID is displayed on the first display device 31 (S702: YES), the first edge device 41 updates the displayed face image or body image and matching status to the face image or body image and matching status included in the person movement information or person matching information and displays them (S704).Then, the first edge device 41 returns the process to step S701.

[0111] 2, a matching result image showing the face image or body image of each person 1 moving in the first direction and the matching status (e.g., matching success or failure, etc.) is displayed on the first display device 31. Therefore, each person 1 moving in the first direction can visually confirm their own matching result from the first display device 31 installed at the destination in the direction of movement.

[0112] 2, the second display device 32 displays a matching result image showing the face image or body image of each person 1 moving in the second direction and the matching status (e.g., matching success or failure, etc.) Therefore, each person 1 moving in the second direction can visually confirm their own matching result from the second display device 32 installed at the destination in the direction of movement.

[0113] The details of the above-mentioned display process can be expressed as follows: That is, the management device 50 detects the movement direction of each person 1 moving between the first gate 11 and the second gate 12, and displays the matching result of the facial image 220 of the person 1 moving in the first direction on the first display device 31 but not on the second display device 32, and displays the matching result of the facial image 220 of the person 1 moving in the second direction on the second display device 32 but not on the first display device 31.

[0114] In addition, if the gate system 9 is configured to include the first photographing device 21 and the first display device 31, but not the second photographing device 22 and the second display device 32, the management device 50 displays the matching results of the facial image 220 of the person 1 moving in the first direction on the first display device 31, and does not need to display the matching results of the facial image 220 of the person 1 moving in the second direction (for example, an error message such as illegal passage or the back view of a passerby, etc.) on the first display device 31.

[0115] As a result, the matching result for person 1 is displayed on the display device installed in the direction of movement of person 1, and is not displayed on the display device installed opposite the direction of movement of person 1. Therefore, the management device 50 can appropriately show the matching result to each of person 1 moving in the first direction along the passage 2 and person 1 moving in the second direction.

[0116] 13 is a diagram showing face matching zone 301 and passage determination zone 302 in detection area 3 according to embodiment 1. Note that, although the following description will be given using first gate 11 and first direction as an example, the same applies to second gate 12 and second direction.

[0117] The image captured by the first photographing device 21 at the first gate 11 is used for face matching of the person 1 moving in the first direction. In the detection area 3, a predetermined area close to the second gate 12 is set as a face matching zone 301, and a predetermined area close to the first gate 11 is set as a passage determination zone 302. The first edge device 41 may change the display pattern on the first display device 31 depending on whether the person 1 is located in the face matching zone 301 or the passage determination zone 302. Next, the display pattern will be described.

[0118] 14 is a diagram for explaining an example of a display pattern in the first display device 31 according to embodiment 1. Note that, with regard to the display pattern in the second display device 32, in the following description with reference to FIG. 14, the "first edge device 41" may be read as the "second edge device 42," and the "first display device 31" may be read as the "second display device 32."

[0119] If the person 1 is located outside the detection area 3 (outside the zone), the first edge device 41 does not display the matching result image of the person 1 on the first display device 31 .

[0120] When person 1 is located in the face matching zone 301, the first edge device 41 controls the display of the first display device 31 according to the following patterns (A0) to (A3): (A0) When the face image 220 of person 1 exists and the matching status 111 is "matching in progress," the first edge device 41 displays a matching result image including the face image 220 of person 1 and a message "matching in progress" on the first display device 31. (A1) When the face image 220 of person 1 exists and the matching status 111 is "matching successful," the first edge device 41 displays a matching result image including the face image 220 of person 1 and a message "matching successful" on the first display device 31. At this time, the first edge device 41 may output a matching success sound from a speaker provided in the first display device 31. (A2) When the facial image 220 of person 1 exists and the matching status 111 is "matching failed," the first edge device 41 displays a matching result image including the facial image 220 of person 1 and a message "matching failed" on the first display device 31. At this time, the first edge device 41 may output a matching failure sound from a speaker provided in the first display device 31. (A3) When the facial image 220 of person 1 does not exist but the body image 200 exists, the first edge device 41 displays a matching result image including the body image 200 of person 1 and a message "warning" on the first display device 31. At this time, the first edge device 41 may output a warning sound from a speaker provided in the first display device 31.

[0121] When person 1 is located in the passage determination zone 302, the first edge device 41 controls the display of the first display device 31 according to the following patterns (A4) to (A7). (A4) When the face image 220 of person 1 is present and the matching status 111 is "matching in progress," the first edge device 41 displays a matching result image including the face image 220 of person 1 and a message "matching in progress" on the first display device 31. (A5) When the face image 220 of person 1 is present and the matching status 111 is "matching successful," the first edge device 41 displays a matching result image including the face image 220 of person 1 and a message "passed" on the first display device 31. At this time, the first edge device 41 may output a passing sound from a speaker provided in the first display device 31. (A6) When the facial image 220 of person 1 is present and the matching status 111 is "matching failed," the first edge device 41 displays a matching result image including the facial image 220 of person 1 and a message "unauthorized passage" on the first display device 31. At this time, the first edge device 41 may output an unauthorized passage sound from a speaker provided in the first display device 31. (A7) When the facial image 220 of person 1 is not present but the body image 200 is present, the first edge device 41 displays a matching result image including the body image 200 of person 1 and a message "unauthorized passage" on the first display device 31. At this time, the first edge device 41 may output an unauthorized passage sound from a speaker provided in the first display device 31.

[0122] The display priority may be set as follows: The passage determination zone 302 has a higher display priority than the face matching zone 301. The "passing" message has the highest display priority, followed by the "matching successful" message. If a face image 220 is present, the "matching in progress" message has the lowest display priority. The presence of a face image 220 has a higher display priority than the presence of only a body image 200.

[0123] For example, the display priorities of the above-mentioned patterns (A0) to (A7) may be set as follows, from lowest to highest: (A3)<(A7)<(A0)<(A4)<(A2)<(A6)<(A1)<(A5).

[0124] 15A and 15B are diagrams for explaining a method for detecting, from the personal information table, personal information 100 that is estimated to belong to the same person as person 1 indicated by the body detection information in accordance with Embodiment 1. In FIG. 15 , the description will be given assuming that person 1 is moving in a first direction. In the case where person 1 is moving in a second direction, in the following description referring to FIG. 15 , the "first image capturing device 21" should be read as the "second image capturing device 22," and the "second image capturing device 22" should be read as the "first image capturing device 21."

[0125] A body image 200 of person 1 moving in a first direction captured by the first camera 21 shows the front view of the body of person 1. Hereinafter, the X and Y coordinates of body coordinates 201A calculated from the front view of body image 200 will be referred to as the X coordinate 201Ax and Y coordinate 201Ay of the front of the body, respectively. Furthermore, a face image 220 of person 1 moving in the first direction captured by the first camera 21 shows the front view of the face of person 1. Hereinafter, the X and Y coordinates of face coordinates 221A calculated from the front view of face image 220 will be referred to as the X coordinate 221Ax and Y coordinate 221Ay of the front of the face, respectively.

[0126] The body image 200 of the person 1 moving in the first direction captured by the second image capturing device 22 shows the back of the body of the person 1. Hereinafter, the X coordinate and the Y coordinate of the body coordinate 201B calculated from the back body image 200 will be referred to as the X coordinate 201Bx and the Y coordinate 201By of the back of the body, respectively.

[0127] As shown in FIG. 15A , the Y coordinate 201Ay of the front of the body and the Y coordinate 201By of the back of the body are separated by the thickness of the person 1 in the front-to-back direction. Therefore, if the positional difference between the Y coordinate 201Ay of the front of the body and the Y coordinate 201By of the back of the body is smaller than a predetermined threshold W based on the body thickness (hereinafter referred to as the body thickness threshold), it can be assumed that the Y coordinate 201Ay of the front of the body and the Y coordinate 201By of the back of the body correspond to the Y coordinates of the same person. The body thickness threshold W may be a value slightly larger than the average body thickness. Similarly, if the positional difference between the Y coordinate 221Ay of the front of the face and the Y coordinate 201By of the back of the body is smaller than the body thickness threshold W, it can be assumed that the Y coordinate 221Ay of the front of the face and the Y coordinate 201By of the back of the body correspond to the Y coordinates of the same person.

[0128] 15B , if the amount of change in the X component between X-coordinate 201Ax of body coordinate 201A included in the received body detection information (i.e., the current X-coordinate) and X-coordinate 201Ax of body coordinate 201A included in personal information 100 (i.e., the past X-coordinate) is equal to or less than a predetermined threshold value (hereinafter referred to as the body width threshold value) W2, it can be estimated that X-coordinate 201Ax of the current body coordinate and X-coordinate 201Ax of the past body coordinate are the X-coordinates of the same person. Note that face coordinate 221A may be used instead of body coordinate 201A.

[0129] In other words, if the management device 50 can detect from the personal information table personal information 100 such that the positional difference between the X coordinate of the body coordinate 201 (or face coordinate 221) included in the received body detection information and the X coordinate of the body coordinate 201 (or face coordinate 221) included in the personal information 100 is less than the body width threshold W2, and the positional difference between the Y coordinate of the body coordinate 201 included in the received body detection information and the Y coordinate of the body coordinate 201 or the Y coordinate of the face coordinate 221 included in the personal information 100 is less than the body thickness threshold W, the management device 50 may estimate that the detected personal information 100 is the same person as the person 1 indicated by the body detection information.

[0130] Similarly, if the management device 50 can detect from the personal information table personal information 100 in which the positional difference between the X coordinate of the body coordinate 201 (or face coordinate 221) included in the received body detection information and the X coordinate of the body coordinate 201 (or face coordinate 221) included in the personal information 100 is smaller than the body width threshold W2, and the positional difference between the Y coordinate of the face coordinate 221 included in the received face detection information and the Y coordinate of the body coordinate 201 or the Y coordinate of the face coordinate 221 included in the personal information 100 is smaller than the body thickness threshold W, the management device 50 may estimate that the detected personal information 100 is the same person as the person 1 indicated by the face detection information.

[0131] The above-described conditions regarding the X-coordinate positional difference and the body width threshold W2, and the Y-coordinate positional difference and the body thickness threshold W are met when the time difference between the face photographing time included in the face detection information or the body photographing time included in the body detection information and the face image update time 108 or the body image update time 105 included in the person information 100 is sufficiently small (i.e., when the time difference is smaller than a predetermined time threshold T). Because person 1 is moving, if the time difference is large, person information 100 that satisfies the above-described conditions regarding the X-coordinate positional difference and the body width threshold W2, and the Y-coordinate positional difference and the body thickness threshold W is likely to belong to a different person who was in that position in the past.

[0132] 8 may perform the following process: The management device 50 detects, from the personal information table, personal information 100 for which the time difference between the face capture time included in the face detection information and the face image update time 108 or the body image update time 105 included in the personal information 100 is less than a predetermined time threshold T, the positional difference between the X coordinate of the face coordinate 221 included in the face detection information and the X coordinate of the face coordinate 109 included in the personal information 100 is smaller than a body width threshold W2, and the positional difference between the Y coordinate of the face coordinate 221 included in the face detection information and the Y coordinate of the face coordinate 109 or the Y coordinate of the body coordinate 106 included in the personal information 100 is smaller than a body thickness threshold W.

[0133] 9 may perform the following process: The management device 50 detects, from the personal information table, personal information 100 in which the time difference between the body image capture time included in the body detection information and the face image update time 108 or the body image update time 105 included in the personal information 100 is less than a predetermined time threshold T, the positional difference between the X coordinate of the body coordinate 201 included in the body detection information and the X coordinate of the body coordinate 106 included in the personal information 100 is smaller than a body width threshold W2, and the positional difference between the X coordinate of the body coordinate 201 included in the body detection information and the Y coordinate of the face coordinate 109 or the Y coordinate of the body coordinate 106 included in the personal information 100 is smaller than a body thickness threshold W.

[0134] The above content can be expressed as follows: That is, if the difference between the position (e.g., body coordinates 201 or face coordinates 221) of person 1 included in a first image (e.g., body image 200 or face image 220) captured by first image capture device 21 and the position (e.g., body coordinates 201 or face coordinates 221) of person 1 included in a second image (e.g., body image 200 or face image 220) captured by second image capture device 22 is smaller than a predetermined threshold W based on the thickness of the person's body, management device 50 estimates that the position of person 1 included in the first image and the position of person 1 included in the second image represent the positions of the same person.

[0135] This allows the management device 50 to estimate, for example, whether a first image capturing the front of person 1 and a second image capturing the back of person 1 are of the same person.

[0136] <Method for Detecting Person's Moving Direction> FIG. 16 is a diagram for explaining a method for detecting the moving direction of person 1 according to the first embodiment.

[0137] The management device 50 detects the moving direction of the person 1 by, for example, the following first, second, or third method.

[0138] <<First Method>> The management device 50 detects the movement direction 112 of the person 1 based on the movement direction over time of the face coordinates 109 or the body coordinates 106 included in the person information 100. For example, when the management device 50 detects that the Y coordinate of the face coordinates 109 or the Y coordinate of the body coordinates 106 included in the person information 100 is moving in the negative direction of the Y axis over time, the management device 50 detects the movement direction 112 of the person 1 as a first direction. When the management device 50 detects that the Y coordinate of the face coordinates 109 or the Y coordinate of the body coordinates 106 included in the person information 100 is moving in the positive direction of the Y axis over time, the management device 50 detects the movement direction 112 of the person 1 as a second direction.

[0139] <<Second Method>> The management device 50 determines whether the image of the person 1 detected for the first time was captured by the first camera 21 or the second camera 22, based on the camera device ID. If the image of the person 1 detected for the first time was captured by the first camera 21, the management device 50 detects the movement direction 112 of the person 1 as the first direction. This is because the person 1 entering the detection area 3 from the second gate 12 is first photographed by the first camera 21 provided at the first gate 11. If the image of the person 1 detected for the first time was captured by the second camera 22, the management device 50 detects the movement direction 112 of the person 1 as the second direction. This is because the person 1 entering the detection area 3 from the first gate 11 is first photographed by the second camera 22 provided at the second gate 12.

[0140] 16 , when the face coordinates 109 or body coordinates 106 of a person 1 detected for the first time are included in a predetermined area 312 of the detection area 3 that is closer to the second gate 12, the management device 50 detects the movement direction 112 of the person 1 as a first direction. This is because the person 1 entering the detection area 3 from the second gate 12 first enters the predetermined area 312 closer to the second gate 12. When the face coordinates 109 or body coordinates 106 of a person 1 detected for the first time are included in a predetermined area 311 of the detection area 3 that is closer to the first gate 11, the management device 50 detects the movement direction 112 of the person 1 as a second direction. This is because the person 1 entering the detection area 3 from the first gate 11 first enters the predetermined area 311 closer to the first gate 11.

[0141] The management device 50 may detect the movement direction 112 of person 1 as the first direction when the face image 107 or body image 104 of person 1 detected for the first time is included in a predetermined area 312 of the detection area 3 that is closer to the second gate 12 (or the second line) using the image captured by the first image capturing device 21, without using the image captured by the second image capturing device 22. Alternatively, the management device 50 may detect the movement direction 112 of person 1 as the second direction when the face image 107 or body image 104 of person 1 detected for the first time is included in a predetermined area 311 of the detection area 3 that is closer to the first gate 11 (or the first line). In other words, the gate system 9 may detect the movement direction 112 of person 1 using the first image capturing device 21 and first display device 31 in one direction.

[0142] (Embodiment 2) Fig. 17 is a diagram for explaining an overview of a gate system 9 according to embodiment 2. Fig. 18 is a block diagram of the gate system 9 according to embodiment 2. Fig. 19 is a diagram showing an example of a display of an unauthorized passerby image 500 on a surveillance monitor 70 according to embodiment 2.

[0143] As shown in Fig. 17 , the gate system 9 according to the second embodiment includes a surveillance monitor 70 in addition to the devices shown in Fig. 3 . In the second embodiment, in addition to the processing described in the first embodiment, the management device 50 performs processing to display an image (hereinafter referred to as an unauthorized pedestrian image 500) of a person who does not have the right to pass through the gate device 10 (hereinafter referred to as an unauthorized pedestrian) on the surveillance monitor 70. For example, the surveillance monitor 70 is installed near the exit of the gate device 10, and an observer 80 monitors the surveillance monitor 70 and people passing through the gate device 10 near the exit of the gate device 10. In this way, when the unauthorized pedestrian image 500 is displayed on the surveillance monitor 70, the observer 80 can quickly find the unauthorized pedestrian by comparing the unauthorized pedestrian image 500 with the person passing through the gate device 10.

[0144] The second embodiment will be described in detail below. The methods for matching and tracking a person's face may be the same as those described in the first embodiment. The second embodiment will describe a case in which the first gate 11 in the first embodiment is an exit gate, the second gate 12 is an entrance gate, and a person moving in a first direction is judged as having illegal passage. However, the contents of the second embodiment also apply to a case in which the first gate 11 in the first embodiment is an entrance gate, the second gate 12 is an exit gate, and a person moving in a second direction is judged as having illegal passage. In the following description, a person whose face has been successfully matched may be referred to as an authorized person 401, and a person whose face has not been matched may be referred to as an unauthorized person 402.

[0145] <Flowchart> FIG. 20 is a flowchart showing an example of processing performed by the management device 50 according to the second embodiment.

[0146] The management device 50 detects and tracks a person from an image captured by the first image capturing device 21, and stores the image in the memory 1002 or the storage 1003 (see FIG. 29 ) (S1001). The memory 1002 or the storage 1003 stores a plurality of captured images in chronological order.

[0147] The management device 50 performs face matching on the detected face image of the person (S1002). The face matching may be performed by the method described in the first embodiment.

[0148] The management device 50 sets a permitted person area 411 for an area in the captured image that shows a person whose face has been successfully matched (a permitted person), and sets a non-permitted person area 412 for an area in the captured image that shows a person whose face has not been matched (a non-permitted person) (S1003). At this time, the management device 50 may set the area so as to include as much of the person's entire body as possible. Furthermore, the shapes of the permitted person area 411 and the non-permitted person area 412 may be rectangular.

[0149] The management device 50 determines whether or not the permitted person area 411 and the non-permitted person area 412 overlap each other (S1004).

[0150] If the permitted person area 411 and the non-permitted person area 412 do not overlap at all (S1004: NO), the management device 50 proceeds to step S1007.

[0151] If the permitted person area 411 and the non-permitted person area 412 partially overlap (S1004: YES), the management device 50 determines whether the permitted person area 411 overlaps at least a part of the upper part of the non-permitted person area 412 (S1005). For example, if the permitted person is located behind the non-permitted person as viewed from the first image capture device 21, the permitted person area 411 overlaps at least a part of the upper part of the non-permitted person area 412.

[0152] If it is determined in step S1005 that the permitted person area 411 overlaps at least a portion of the upper part of the non-permitted person area 412 (S1005: YES), the management device 50 proceeds to step S1007.

[0153] If it is determined in step S1005 that the permitted person area 411 does not overlap at least a portion of the upper part of the prohibited person area 412 (S1005: NO), that is, if the permitted person area 411 overlaps at least a portion of the prohibited person area 412 other than the upper part (for example, at least a portion of the lower part), the management device 50 performs the following process. That is, the management device 50 processes the image of the permitted person in the prohibited person area 412 so that it is not visible, and generates the unauthorized passerby image 500 (S1006). Details of this processing will be described later. The process then proceeds to step S1008.

[0154] In step S1007, the management device 50 generates an unauthorized passerby image 500 based on the image of the unauthorized person area 412 (S1007). For example, the management device 50 generates the unauthorized passerby image 500 by directly using the image of the unauthorized person area 412. Details of this process will be described later. Then, the process proceeds to step S1008.

[0155] In step S1008, the management device 50 displays the unauthorized passerby image 500 generated in step S1006 or step S1007 on the surveillance monitor 70, and returns the process to step S1001. When the management device 50 detects an unauthorized person 402 in the face matching zone 301 shown in FIG. 13 , the management device 50 may display the unauthorized passerby image 500 at the timing when the unauthorized person 402 enters the passage determination zone 302. However, the timing when the management device 50 displays the unauthorized passerby image 500 is not limited to this. For example, when the management device 50 detects an unauthorized person 402, the management device 50 may immediately display the unauthorized passerby image 500. Note that a method for displaying the unauthorized passerby image 500 on the surveillance monitor 70 will be described in detail later.

[0156] Through the above process, the unauthorized passerby image 500 corresponding to the unauthorized passerby is displayed on the surveillance monitor 70. Therefore, the surveillance staff 80 can quickly find the unauthorized passerby by looking at the surveillance monitor 70.

[0157] <When an authorized person area overlaps at least a portion of a prohibited person area other than the upper portion (for example, at least a portion of the lower portion)> Figure 21 is a diagram illustrating a case where an unauthorized passerby image is generated without processing when an authorized person area 411 overlaps at least a portion of the lower portion of a prohibited person area 412. Note that "above" the prohibited person area 412 may be a predetermined range including the upper edge of the prohibited person area 412. "Below" the prohibited person area 412 may be below the predetermined range including the upper edge of the prohibited person area 412.

[0158] When the permitted person area 411 overlaps at least a portion of the lower part of the prohibited person area 412, if the image of the prohibited person area 412 is used as an unauthorized passerby image without any processing, the unauthorized passerby image will include not only the facial image of the prohibited person but also the facial image of the authorized person, as shown in image 601 in Figure 21. If such an unauthorized passerby image is displayed on the surveillance monitor 70, the surveillance officer 80 may be confused about which of the facial images included in the unauthorized passerby image is the unauthorized passerby. Furthermore, it is undesirable to display the facial image of a person who is not an unauthorized passerby as an unauthorized passerby image on the surveillance monitor 70.

[0159] 20, the management device 50 performs processing so that the facial image of the authorized person is not displayed in the unauthorized passerby image 500, thereby generating the unauthorized passerby image 500. For example, the management device 50 performs either the following first processing method or second processing method.

[0160] Fig. 22A is a diagram for explaining a first processing method for a non-permitted person region according to Embodiment 2. Hereinafter, the processing of step S1006 in Fig. 20 will be described in detail with reference to Fig. 22A.

[0161] If the permitted person area 411 overlaps at least a portion of the lower part of the prohibited person area 412, that is, if a permitted person is present in front of the prohibited person as viewed from the exit side, the management device 50 performs processing to cut out the portion of the image of the prohibited person area 412 that is overlapped by the permitted person area 411, as shown in image 602A, to generate unauthorized passerby image 500A. As a result, the facial image of the permitted person is not included in unauthorized passerby image 500A.

[0162] Alternatively, if the permitted person area 411 overlaps at least a portion of the lower part of the unauthorized person area 412, that is, if an authorized person is present in front of the unauthorized person as viewed from the exit side, the management device 50 performs processing to mask the portion of the image of the unauthorized person area 412 that overlaps with the permitted person area 411, as shown in image 603A, to generate unauthorized passerby image 500B. As a result, the facial image of the authorized person becomes invisible in unauthorized passerby image 500B.

[0163] Through the above process, the surveillance monitor 70 can display the unauthorized passerby image 500 (500A, 500B) without displaying the face image of the authorized person.

[0164] 22B is a diagram for explaining a modification of the first processing method of the unauthorized person area according to Embodiment 2. The management device 50 may apply the processing method shown in FIG. 22B instead of the processing method shown in FIG. 22A.

[0165] When the permitted person area 411 overlaps at least a portion of the lower part of the unauthorized person area 412, that is, when viewed from the exit side, when a permitted person is present in front of the unauthorized person, the management device 50 may perform processing to cut out the portion of the facial image included in the permitted person area 411 from the image of the unauthorized person area 412, as shown in image 602B, to generate unauthorized passerby image 500A. As a result, the facial image of the permitted person is not included in unauthorized passerby image 500A.

[0166] Alternatively, if the permitted person area 411 overlaps at least a portion of the lower part of the unauthorized person area 412, that is, if an authorized person is present in front of the unauthorized person when viewed from the exit side, the management device 50 performs processing to mask the portion of the facial image included in the permitted person area 411 of the image of the unauthorized person area 412, as shown in image 603B, to generate unauthorized passerby image 500B. As a result, the facial image of the authorized person cannot be seen in unauthorized passerby image 500B.

[0167] Through the above process, the surveillance monitor 70 can display the unauthorized passerby image 500 (500A, 500B) without displaying the face image of the authorized person.

[0168] FIG. 23 is a diagram for explaining a processing method in the case where a plurality of permitted person regions 411 overlap at least partly below a non-permitted person region 412 according to the second embodiment.

[0169] When multiple permitted person regions 411 overlap at least a portion below the prohibited person region 412, that is, when multiple permitted persons are present in front of the prohibited person as viewed from the exit side, the management device 50 generates an unauthorized passerby image 500C by cutting out the overlapping portions of the permitted person regions 411 from the image of the prohibited person region 412, as shown in image 604. Note that, as in FIG. 22B , the management device 50 may also generate an unauthorized passerby image 500C by cutting out the facial image portions of the permitted person regions 411 from the image of the prohibited person region 412.

[0170] Alternatively, when multiple permitted person regions 411 overlap at least a portion below the prohibited person region 412, that is, when multiple permitted persons are present in front of the prohibited person as viewed from the exit side, the management device 50 performs processing to mask the portions of the image of the prohibited person region 412 that are overlapped by the permitted person regions 411, as shown in image 605, to generate an unauthorized passerby image 500D. Note that, as in FIG. 22B , the management device 50 may also perform processing to mask the facial image portions of the permitted person regions 411 from the image of the prohibited person region 412 to generate an unauthorized passerby image 500D.

[0171] 24 is a diagram for explaining a second processing method for the unauthorized person area 412 according to Embodiment 2. The process of step S1006 in FIG. 20 will be described in detail below with reference to FIG.

[0172] If the permitted person area 411 overlaps at least a portion of the lower part of the prohibited person area 412, that is, if a permitted person is present in front of the prohibited person when viewed from the exit side, the management device 50 generates an unauthorized passerby image 500E by cutting out the portion below the upper edge Z1 where the permitted person area 411 overlaps from the image of the prohibited person area 412, as shown in image 606. As a result, the unauthorized passerby image 500E does not include the facial image of the permitted person.

[0173] Alternatively, if the permitted person area 411 overlaps at least a portion of the lower part of the unauthorized person area 412, that is, if an authorized person is present in front of the unauthorized person when viewed from the exit side, the management device 50 generates an unauthorized passerby image 500F by masking the portion of the image of the unauthorized person area 412 below the upper edge Z1 where the permitted person area 411 overlaps, as shown in image 607. This makes the facial image of the authorized person invisible in the unauthorized passerby image 500F.

[0174] Through the above processing, the surveillance monitor 70 can display the unauthorized passerby image 500 (500E, 500F) in which the face image of the authorized person 401 is not displayed.

[0175] <When an authorized person area overlaps at least a part of the upper portion of a prohibited person area> Fig. 25 is a diagram for explaining a method for generating an unauthorized passerby image 500 when an authorized person area 411 overlaps at least a part of the upper portion of a prohibited person area 412 according to Embodiment 2. Hereinafter, the processing of step S1007 in Fig. 20 will be described in detail with reference to Fig. 25 .

[0176] When the permitted person area 411 overlaps at least a portion of the upper part of the prohibited person area 412, that is, when viewed from the exit side, when the permitted person 401 is located behind the prohibited person 402, the management device 50 may use the image of the prohibited person area 412 as the unauthorized passerby image 500G as is, as shown in image 608. This is because even if the image of the prohibited person area 412 is used as the unauthorized passerby image 500G as is, the facial image of the permitted person 401 will not be included in the unauthorized passerby image 500G.

[0177] Through the above processing, the surveillance monitor 70 can display the unauthorized passerby image 500 (500G) in which the face image of the authorized person 401 is not displayed.

[0178] <Details of Display of Unauthorized Passerby Images> The first image capturing device 21 continuously captures images of people passing through the passage 2, thereby generating multiple captured images in chronological order. Accordingly, multiple unauthorized passerby images 500 of unauthorized passerby 402 are also generated in chronological order and stored in the memory 1002 or the storage 1003. The management device 50 may preferentially display on the surveillance monitor 70, among the multiple generated unauthorized passerby images 500, those that satisfy a predetermined condition A. For example, the management device 50 may determine which unauthorized passerby image 500 to display preferentially based on one of the following conditions A1, A2, or A3.

[0179] <<Condition A1>> The management device 50 preferentially displays, on the surveillance monitor 70, the unauthorized passerby image 500 generated using the most recently captured image among the multiple unauthorized passerby images 500. This allows the surveillance staff 80 to check the unauthorized passerby image 500 that includes the most recently captured facial image of the unauthorized person 402.

[0180] <<Condition A2>> The management device 50 preferentially displays the unauthorized passerby image 500 with the smallest proportion of the cut-out or masked area among the multiple unauthorized passerby images 500 on the surveillance monitor 70. This allows the surveillance staff 80 to confirm the unauthorized passerby image 500 in which the entire body of the unauthorized person 402 is more clearly visible.

[0181] <<Condition A3>> The management device 50 preferentially displays, among the multiple unauthorized passerby images 500, the unauthorized passerby image 500 having the largest ratio of the maximum vertical length d2 to the maximum horizontal length d1 (see FIGS. 22, 23, and 24) on the surveillance monitor 70. This allows the surveillance staff 80 to confirm the unauthorized passerby image 500 in which the entire body of the unauthorized person 402 is more clearly visible.

[0182] Furthermore, the plurality of unauthorized passerby images 500 under the above-mentioned condition A may be selected as unauthorized passerby images 500 that satisfy a predetermined condition B. For example, the management device 50 may select unauthorized passerby images 500 from the entire unauthorized passerby images 500 based on any one of the following conditions B1, B2, B3, and B4, and set them as the plurality of unauthorized passerby images 500 under the above-mentioned condition A.

[0183] <<Condition B1>> The management device 50 selects unauthorized passerby images 500 generated using images captured in the face matching zone 301 from the entire unauthorized passerby images 500. In this way, unauthorized passerby images 500 that are highly likely to show the entire body of the unauthorized person 402 are selected.

[0184] <<Condition B2>> The management device 50 selects, from all unauthorized passerby images 500, unauthorized passerby images 500 generated using images captured in the passage determination zone 302. In this way, unauthorized passerby images 500 that show the unauthorized person 402 after unauthorized passage has been confirmed are selected.

[0185] <<Condition B3>> The management device 50 selects unauthorized passerby images 500 that include the facial image of the unauthorized person 402 from all unauthorized passerby images 500. As a result, unauthorized passerby images 500 that do not include the facial image of the unauthorized person 402 are excluded.

[0186] <<Condition B4>> The management device 50 selects unauthorized passerby images in which the face of the unauthorized person 402 is facing forward within a predetermined range (for example, within a range from −α degrees to +α degrees) from the entire unauthorized passerby image 500. This selects unauthorized passerby images 500 in which the face of the unauthorized person 402 is facing more forward.

[0187] The management device 50 may select the unauthorized passerby image 500 based on a combination of two or more of the above-mentioned conditions B1 to B4.

[0188] FIG. 26 is a diagram showing an example in which the display mode of the unauthorized passerby image 500 is changed depending on the position of the unauthorized person 402 on the passage 2 according to the second embodiment.

[0189] The management device 50 may change the display position of the unauthorized passerby image 500 on the surveillance monitor 70 in conjunction with the position of the unauthorized person 402 in the width direction of the passage 2 .

[0190] For example, as shown in (a) of Figure 26, when the unauthorized person 402 is located at the right end of the passage determination zone 302 as viewed from the exit side of passage 2, the management device 50 displays the unauthorized passerby image 500 on the right side of the surveillance monitor 70. Then, as shown in (b) and (c) of Figure 26, when the unauthorized person 402 moves diagonally from the right end of the passage determination zone 302 to the left end as viewed from the exit side of passage 2, the management device 50 accordingly changes the display position of the unauthorized passerby image 500 from the right side to the left side of the surveillance monitor 70. This allows the monitor 80 to see the position of the unauthorized passerby image 500 displayed on the surveillance monitor 70 and grasp the position in passage 2 of the unauthorized passerby 402 corresponding to the unauthorized passerby image 500. This can improve the efficiency with which the monitor 80 captures the unauthorized person 402.

[0191] FIG. 27 is a diagram showing an example in which a plurality of unauthorized passerby images 500 according to the second embodiment are displayed.

[0192] When displaying one unauthorized passerby image 500, the management device 50 may display one unauthorized passerby image 500 using the entire screen of the surveillance monitor 70, as shown in FIG. 27(a).

[0193] When displaying a plurality of unauthorized passerby images 500 (500H, 500I), the management device 50 may increase the number of unauthorized passerby images 500 displayed by vertically dividing the screen of the surveillance monitor 70 as shown in FIG. 27(b). In this case, the management device 50 may display the unauthorized passerby image 500 with the most recent photographed time on the top row (or bottom row). The management device 50 may also increase the number of unauthorized passerby images 500 displayed by horizontally dividing the screen of the surveillance monitor 70. In this case, the management device 50 may display the unauthorized passerby image 500 with the most recent photographed time on the far right (or far left).

[0194] Furthermore, the management device 50 may prevent the display of multiple unauthorized passerby images 500 of the same unauthorized person 402. For example, the management device 50 may update a previous unauthorized passerby image 500 of the same unauthorized person 402 to a new unauthorized passerby image 500 captured at the time of capture, rather than displaying an additional unauthorized passerby image 500 of the same unauthorized person 402.

[0195] This allows the monitor 80 to view the surveillance monitor 70 and recognize the timing in chronological order when each of the multiple unauthorized persons 402 was detected. This allows the monitor 80 to, for example, prioritize capturing an older unauthorized person 402 in chronological order, or prioritize capturing a newer unauthorized person 402 in chronological order. In other words, the monitor 80 can view the surveillance monitor 70 and determine which unauthorized person 402 should be prioritized for capture.

[0196] When the matching of an unauthorized person 402 in the passage judgment zone 302 is successful (i.e., when the unauthorized person 402 is determined to be a permitted person 401), the management device 50 may automatically hide the display of the unauthorized passerby image 500 of the unauthorized person 402. This prevents the face image of a person who failed to match in the face matching zone 301 and was determined to be an unauthorized person 402 but was successfully matched in the passage judgment zone 302 (permitted person 401) from being continuously displayed on the surveillance monitor 70.

[0197] FIG. 28 is a diagram showing an example of displaying a plurality of images 500 of unauthorized passersby taken in time series like a moving image according to the second embodiment.

[0198] The management device 50 may display a plurality of unauthorized passerby images 500 captured in chronological order on the surveillance monitor 70, and may repeat the display in chronological order. For example, as shown in Fig. 28 , the management device 50 may sequentially display an unauthorized passerby image 500 captured at time t = 1, an unauthorized passerby image 500 captured at time t = 2, an unauthorized passerby image 500 captured at time t = 3, and an unauthorized passerby image 500 captured at time t = 4, and then repeatedly return to displaying the unauthorized passerby image 500 captured at time t = 1.

[0199] Furthermore, even after the unauthorized person 402 has passed through the exit gate, the management device 50 may repeatedly display on the surveillance monitor 70 in chronological order as described above, multiple unauthorized passerby images 500 taken while the unauthorized person 402 was passing through passage 2.

[0200] As a result, the unauthorized passerby image 500 is displayed on the surveillance monitor 70 like a moving image, which makes it easier for the surveillance officer 80 to identify the unauthorized person 402 passing through the passage 2, compared to when the unauthorized passerby image 500 is displayed as a single still image. In addition, such a moving display attracts the attention of people who may be attempting to pass unauthorizedly, which helps to deter unauthorized passage.

[0201] (Embodiment 3) <Indication of Confirmation of Unauthorized Passage> In embodiment 2, the monitor 80 recognizes a person illegally passing through the passageway by looking at the image of the illegal passerby displayed on the surveillance monitor 70 and, for example, stops the person. However, the gate system 9 may temporarily fail to match the person but later succeed, and a failed match does not necessarily mean illegal passage. Therefore, simply displaying a failed match by the gate system 9 makes it difficult to determine the timing for the monitor 80 to initiate action to stop the illegal passerby. The following describes a method for clarifying the timing for the monitor 80 to initiate action to stop the illegal passerby. Note that the components of the gate system 9 according to embodiment 3 are generally the same as those in embodiment 1 or 2, and therefore common reference numbers are used and their description may be omitted.

[0202] Fig. 29 is a diagram showing an example of the movement of a person passing through a zone according to embodiment 3. Fig. 30 is a diagram showing an example of a display of a determination result of a person passing through a zone according to embodiment 3.

[0203] For example, as shown in FIG. 29, suppose that person 1 first enters face matching zone 301 from the IN side, then enters passage determination zone 302, and finally moves out of passage determination zone 302 to the OUT side.

[0204] When person 1 is a permitted person (person who has been successfully authenticated) and moves through the zone as indicated by the dotted line 700 in Figure 29, the first display device 31 and / or the surveillance monitor 70 displays the information shown in the column of permitted person in Figure 30(a). That is, when person 1 is located in the face matching zone 301 (for example, when person 1 is located at point 701 in Figure 29), the first display device 31 and / or the surveillance monitor 70 displays "matching successful" when person 1 is located in the passage judgment zone 302 (for example, when person 1 is located at point 702 in Figure 29). When person 1 is located outside the zone (for example, outside the passage judgment zone 302) (for example, when person 1 is located at point 703 in Figure 29) or when tracking of person 1 is lost, the first display device 31 and / or the surveillance monitor 70 displays "passing record." When displaying the "passing record", the management device 50 records the photographed image and face matching ID of the person 1 being tracked as a passing record in the memory 1002 or the storage 1003.

[0205] When person 1 is a person not permitted to pass (person who failed authentication) and moves through the zone as indicated by the dotted line 700 in Figure 29, the first display device 31 and / or the surveillance monitor 70 display the information shown in the column of (b) Person Not Permitted to Pass in Figure 30. That is, when person 1 is located in the face matching zone 301 (for example, when person 1 is located at point 701 in Figure 29), the first display device 31 and / or the surveillance monitor 70 display "Matching Failed" when person 1 is located in the passage judgment zone 302 (for example, when person 1 is located at point 702 in Figure 29). When person 1 is located outside the zone (for example, outside the passage judgment zone 302) (for example, when person 1 is located at point 703 in Figure 29) or when tracking of person 1 is lost, the first display device 31 and / or the surveillance monitor 70 display "Unauthorized Passage." As a result, the monitor 80 can start taking action to stop the person 1 (unauthorized passerby) whose face image is displayed at the same time as the message "unauthorized passage" is displayed on the surveillance monitor 70. In other words, the display method shown in Fig. 30 makes it clear when the monitor 80 should start taking action to stop the unauthorized passerby.

[0206] The display time for "Unauthorized Passage" may be longer than the display time for "Matching Success" or "Passage Record." When displaying "Unauthorized Passage," the management device 50 may record the captured image and tracking log of the person 1 being tracked as an unauthorized passage record in the memory 1002 or the storage 1003.

[0207] <U-Turn Confirmation Display> In addition to people passing through the passage, there may also be people making a U-turn in the middle of the passage. For example, a person who sees the display indicating that matching has failed and realizes that they cannot pass may make a U-turn, or a person who has successfully matched may make a U-turn due to an urgent matter. Below, a display method that can also handle such U-turns will be described.

[0208] Fig. 31 is a diagram showing an example of the movement of a person making a U-turn in a zone according to Embodiment 3. Fig. 32 is a diagram showing an example of a display of a determination result of a person making a U-turn in a zone according to Embodiment 3.

[0209] When person 1 is a permitted person (person who has been successfully authenticated) and makes a U-turn through the zone as indicated by the dotted line 710 in Fig. 31 , the first display device 31 and / or the surveillance monitor 70 displays the information shown in the column of permitted person in Fig. 32 (a) in Fig. 32 . That is, when person 1 is located in the face matching zone 301 (for example, when person 1 is located at point 711 in Fig. 31 ), the first display device 31 and / or the surveillance monitor 70 displays "matching successful" when person 1 is located in the passage judgment zone 302 (for example, when person 1 is located at point 712 in Fig. 31 ). When person 1 makes a U-turn from the passage judgment zone 302 and is located in the face matching zone 301 (for example, when person 1 is located at point 713 in Fig. 31 ), the first display device 31 and / or the surveillance monitor 70 displays "matching successful (U-turn)." When person 1 makes a U-turn and is located outside the zone (for example, outside face matching zone 301) (for example, when the person is located at point 714 in FIG. 31 ) or when tracking of person 1 who has made a U-turn is lost, first display device 31 and / or surveillance monitor 70 display "Matching successful (U-turn)." This allows other passing people 1 and surveillance officer 80 to visually recognize that person 1 in the face image displayed together with "Matching successful (U-turn)" is the person who has made a U-turn.

[0210] When person 1 is a non-permitted person (authentication failed person) and makes a U-turn in the zone as indicated by the dotted line 710 in Fig. 31 , the first display device 31 and / or the surveillance monitor 70 displays the information shown in the column of (b) Non-permitted Person in Fig. 32 . That is, when person 1 is located in the face matching zone 301 (e.g., when located at point 711 in Fig. 31 ), the first display device 31 and / or the surveillance monitor 70 displays "Matching failed" when person 1 is located in the passage judgment zone 302 (e.g., when located at point 712 in Fig. 31 ). When person 1 makes a U-turn from the passage judgment zone 302 and is located in the face matching zone 301 (e.g., when located at point 713 in Fig. 31 ), the first display device 31 and / or the surveillance monitor 70 displays "Matching failed (U-turn)." The first display device 31 and / or the surveillance monitor 70 display "U-turn" when the person 1 makes a U-turn and is located outside the zone (e.g., outside the face matching zone 301 and / or the passage determination zone 302) (e.g., when the person 1 is located at point 714 in Figure 31) or when tracking of the person 1 who made the U-turn is lost. When displaying "U-turn," the management device 50 may erase records, such as photographed images, of the person 1 being tracked. This is because the person 1 who made the U-turn did not engage in illegal passage. This prevents the log of the person 1 who made the U-turn from being recorded as a possible illegal passage. Furthermore, the person 1 who made the U-turn can rest assured by seeing the "Matching failed (U-turn)" or "U-turn" display that they will not be identified or recorded as an illegal pedestrian. Furthermore, other passing people 1 and the surveillance officer 80 can recognize that the person 1 whose face image is displayed with "Matching failed (U-turn)" or "U-turn" is not an illegal pedestrian but a person who made a U-turn.

[0211] <Flowchart> Next, the process of the management device 50 when the illegal passage confirmation display and the U-turn confirmation display are performed will be described.

[0212] FIG. 33 is a flowchart illustrating an example of processing performed by the management device according to the third embodiment.

[0213] The management device 50 cooperates with the first edge device 41 to perform the person detection process (S1201) as described in the first embodiment, thereby detecting each person 1 within the zone.

[0214] The management device 50, in cooperation with the first edge device 41, determines the traveling direction of the detected person 1 (S1202), as described in the first embodiment. The management device 50 performs the following process for each detected person 1.

[0215] The management device 50 determines the zone in which person 1 is currently located (S1203).

[0216] If it is determined that person 1 is located in face matching zone 301 (S1203: face matching zone), management device 50 executes face matching zone processing (S1204) and returns the processing to step S1203. Details of the face matching zone processing will be described later (see FIG. 34).

[0217] If it is determined that person 1 is located in the passage determination zone 302 (S1203: passage determination zone), the management device 50 executes passage determination zone processing (S1205) and returns the processing to step S1203. The passage determination zone processing will be described in detail later (see FIG. 35).

[0218] If it is determined that person 1 is located in another zone or that tracking of person 1 has been lost (S1203: other zone or lost), the management device 50 executes other zone processing (S1206) and returns the processing to step S1203. For example, if person 1 is located in neither the face matching zone 301 nor the passage determination zone 302, or if tracking of person 1 has been lost (lost), the management device 50 determines that person 1 is located in another zone. Details of the other zone processing will be described later (see FIG. 36 ).

[0219] <Face Matching Zone Processing> Fig. 34 is a flowchart showing an example of face matching zone processing according to embodiment 3. The processing shown in Fig. 34 corresponds to details of the processing in step S1204 in Fig. 33.

[0220] The management device 50 determines whether or not person 1 has entered the passage determination zone 302 (S1301). That is, the management device 50 determines whether person 1, who is currently located in the face matching zone 301, has made a U-turn from the passage determination zone 302.

[0221] First, we will explain the processing when person 1 has never entered passage determination zone 302 (S1301: NO), that is, when person 1 located in face matching zone 301 is not a person who has made a U-turn from passage determination zone 302.

[0222] The management device 50 determines whether or not face matching of person 1 has been successful (S1302).

[0223] If face matching of person 1 has been successful (S1302: YES), the management device 50 displays "Matching successful" on the first display device 31 (S1303), and ends this process (return to the process of FIG. 33).

[0224] If face matching of person 1 has failed (S1302: NO), the management device 50 displays "Matching failed" on the first display device 31 (S1304), and ends this process (returns to the process of FIG. 33).

[0225] Next, the processing when person 1 has entered passage determination zone 302 (S1301: YES), that is, when person 1 located in face matching zone 301 is a person who has made a U-turn from passage determination zone 302, will be described.

[0226] The management device 50 determines whether or not the face matching of person 1 has been successful (S1305).

[0227] If face matching for person 1 is successful (S1305: YES), the management device 50 displays "Matching successful (U-turn)" on the first display device 31 (S1306). If "Matching failed" or "Illegal passage" for person 1 is displayed on the surveillance monitor 70, the management device 50 erases the display (S1307) and completes this processing (return to the processing in FIG. 33). This makes it possible to prevent the surveillance monitor 70 from continuing to display information about a person who was once determined to have failed matching but later determined to have succeeded.

[0228] If face matching for person 1 has failed (S1305: NO), the management device 50 displays "Matching failed (U-turn)" on the first display device 31 (S1308). The management device 50 displays "Matching failed (U-turn)" for person 1 on the surveillance monitor 70 (S1309), and completes this processing (return to the processing in FIG. 33). This allows the monitor 80 to look at the surveillance monitor 70 and recognize that person 1, whose face image is displayed along with "Matching failed (U-turn)," is making a U-turn, and can call out to him if necessary.

[0229] <Passage Determination Zone Processing> Fig. 35 is a flowchart showing an example of the passage determination zone processing according to Embodiment 3. The processing shown in Fig. 35 corresponds to details of the processing in step S1205 in Fig. 33 .

[0230] The management device 50 determines whether or not face matching of person 1 located in the passage determination zone 302 has been successful (S1401).

[0231] If face matching for person 1 is successful (S1401: YES), the management device 50 displays "Matching successful" on the first display device 31 (S1402). If "Matching failed" or "Unauthorized passage" or the like is displayed for person 1 on the surveillance monitor 70, the management device 50 erases the display (S1403) and completes this processing (return to the processing in FIG. 33). This makes it possible to prevent the surveillance monitor 70 from continuing to display information about a person who was once determined to have failed matching but later determined to have succeeded.

[0232] If face matching of person 1 has failed (S1401: NO), the management device 50 displays "Matching failed" on the first display device 31 (S1404). The management device 50 displays "Matching failed" on the surveillance monitor 70 (S1405), and ends this process (return to the process of FIG. 33).

[0233] <Other Zone Processing> Fig. 36 is a flowchart showing an example of other zone processing according to embodiment 3. The processing shown in Fig. 36 corresponds to details of the processing in step S1206 in Fig. 33.

[0234] The management device 50 determines the zone in which person 1 was last located (S1501).

[0235] First, a case where the zone where person 1 was last located was the face matching zone (S1501: face matching zone), that is, a case where person 1 makes a U-turn and exits from the IN side of face matching zone 301 will be described.

[0236] The management device 50 determines whether person 1 is in a state of successful matching (S1502).

[0237] If person 1 has been successfully matched (S1502: YES), the management device 50 displays "U-turn" on the first display device 31 (S1503) and completes this process (returning to the process in FIG. 33). This prevents the surveillance monitor 70 from continuing to display a person who was once determined to have failed matching but later determined to have been successfully matched.

[0238] If person 1 is not in a state of successful matching (S1502: NO), that is, if person 1 is in a state of failed matching, the management device 50 displays "U-turn" on the first display device 31 (S1505). The management device 50 displays "Matching failed (U-turn)" for that person 1 on the surveillance monitor 70 (S1506). The management device 50 erases records of photographed images of that person 1 (S1507) and completes this processing (return to the processing in FIG. 33). This allows the monitor 80 to look at the surveillance monitor 70 and recognize that person 1, whose face image is displayed along with "Matching failed (U-turn)," is making a U-turn, and can call out to him or her as necessary.

[0239] Next, a case where the zone where person 1 was last located is the passage determination zone (S1501: passage determination zone), that is, a case where person 1 exits from the OUT side of the passage determination zone 302, will be described.

[0240] The management device 50 determines whether person 1 has been successfully matched (S1508).

[0241] If person 1 has been successfully matched (S1508: YES), the management device 50 displays a "passing record" on the first display device 31 and records the photographed image of person 1, the face matching ID, and the like as a passing record in the memory 1002 or the storage 1003 (S1509). Then, the management device 50 completes this process (returns to the process in FIG. 33). This records the passage of the permitted person through the gate.

[0242] If person 1 is not in a state of successful matching (S1508: NO), that is, if person 1 is in a state of unsuccessful matching, the management device 50 displays "Unauthorized Passage" on the first display device 31 and records the photographed image of person 1 and the face matching ID, etc. as an unauthorized passage record in the memory 1002 or storage 1003 (S1510). This records the passage of the unauthorized person through the gate. The management device 50 displays "Unauthorized Passage" along with the face image of person 1 on the surveillance monitor 70 (S1511). This allows the monitor 80 to initiate action such as stopping person 1 (the unauthorized passerby) whose face image is displayed together with "Unauthorized Passage" when it is displayed on the surveillance monitor 70. The management device 50 then completes this process (returning to the process of FIG. 33).

[0243] <When determining whether a person is passing by using an image capturing device installed on the OUT side> Fig. 37 is a diagram for explaining an example of when determining whether a person is passing by using an image capturing device 23 installed on the OUT side according to embodiment 3. Note that image capturing device 23 has the same configuration as first image capturing device 21 or second image capturing device 22, and may be read as, for example, a camera device.

[0244] 37, the image capture device 23 is installed on the OUT side so that the camera angle of view includes at least the IN side of the aisle. An IN entry judgment zone is set in a predetermined range on the IN side of the aisle, and an OUT entry judgment zone is set in a predetermined range on the OUT side of the aisle so that they fit within the camera angle of view of the image capture device 23. For example, the management device 50 determines that the person first detected in the IN entry judgment zone is a person who entered from the IN side, and determines that the person first detected in the OUT entry judgment zone is a person who entered from the OUT side.

[0245] As shown in Figure 37, the management device 50 may perform different processing on a person determined to have entered from the IN side (hereinafter referred to as IN-side entering person 1A) and a person determined to have entered from the OUT side (hereinafter referred to as OUT-side entering person 1B).

[0246] 37 , the management device 50 may set the face matching zone 301 and the passage determination zone 302 for the IN-side approaching person 1A, as described above. This allows the management device 50 to perform face matching and passage determination for the IN-side approaching person 1A, as described above.

[0247] 37 , the management device 50 does not need to set a passage determination zone 302 and a face matching zone 301 for the OUT-side approaching person 1B. In other words, the management device 50 does not need to photograph, match, or track the OUT-side approaching person 1B. This allows the OUT-side approaching person 1B to move freely along the passage from the OUT side to the IN side.

[0248] <Relationship Between the Viewing Angle of the Camera Device and the Zone> FIG. 38 is a diagram for explaining an example of the relationship between the viewing angle of the camera device 23 installed on the OUT side and the zone according to the third embodiment.

[0249] As shown in FIG. 38 , the camera 23 is installed on the OUT side so that at least the IN-side face matching zone 301 is included in the camera's angle of view (photography range) of the camera 23. In this case, part of the passage determination zone 302 may not be included in the camera's angle of view. In this case, the camera 23 cannot capture a person 1 located in an area 741 that is not included in the camera's angle of view of the passage determination zone 302. Therefore, the management device 50 may determine in step S1203 of FIG. 33 that a person 1 who exits from an area 740 included in the camera's angle of view of the passage determination zone 302 to an area 741 that is not included in the camera's angle of view of the passage determination zone 302 belongs to another zone. In other words, the entire face matching zone 301 is included in the camera's angle of view of the camera 23, but the passage determination zone 302 may be set to match the camera's angle of view of the camera 23.

[0250] (Summary of Third Embodiment) The above description of the third embodiment discloses the following techniques.

[0251] <Technology 1> In an information processing device including a processor and a communication device, the communication device receives a photographed image of a person passing through a predetermined passageway, the processor uses the photographed image to track the person, and if the person is located in a verification zone set on the entrance side of the passageway, the processor performs verification of the person using the photographed image of the person, and if the person who fails the verification is located in a passage determination zone set on the exit side of the passageway, the processor displays information indicating that the verification has failed on a predetermined monitor, and if the person who fails the verification exits the passage determination zone toward the exit side, the processor displays information indicating that the person is an unauthorized passerby on the monitor. This allows, for example, an observer 80 to know from the monitor display when unauthorized passage is confirmed. In other words, it becomes clear when the observer 80 should begin taking action to stop the unauthorized passerby.

[0252] <Technology 2> In the information processing device described in Technology 1, when the person who has been successfully matched enters the passage determination zone and then returns to the match zone, the processor displays on the monitor an image of the person together with information indicating that the person is making a U-turn. This allows, for example, an observer 80 to know that the person is making a U-turn from the image displayed on the monitor and the information indicating the U-turn.

[0253] <Technology 3> In the information processing device described in Technology 1 or 2, when a person who failed the verification enters the passage determination zone and then returns to the verification zone, the processor displays on the monitor an image of the person along with information indicating a U-turn. This allows, for example, an observer 80 to know the person making a U-turn from the image displayed on the monitor and the information indicating a U-turn. Furthermore, a person who failed the verification and made a U-turn can be assured that they will not be judged as passing illegally from the image displayed on the monitor and the information indicating a U-turn.

[0254] <Technology 4> In the information processing device described in Technology 3, the processor erases the record of the photographed image of a person who failed the verification when the person enters the passage determination zone, returns to the verification zone, and exits the verification zone. This prevents the photographed image of a person who failed the verification and made a U-turn from being recorded as an illegal passage. Furthermore, the person who failed the verification and made a U-turn can rest assured that the photographed image will not be recorded as an illegal passage.

[0255] (Hardware Configuration) FIG. 39 is a diagram illustrating an example of a hardware configuration of an information processing device according to the present disclosure.

[0256] The above-described first edge device 41, second edge device 42, management device 50, and face matching device 60 may be configured to include an information processing device 1000 shown in Fig. 29. Furthermore, the processes performed by the above-described first edge device 41, second edge device 42, management device 50, and face matching device 60 may be realized by the information processing device 1000 shown in Fig. 29 executing a computer program.

[0257] The information processing device 1000 includes a processor 1001, a memory 1002, a storage 1003, an input device 1004, an output device 1005, a communication device 1006, a graphics processing unit (GPU) 1007, a reading device 1008, and a bus 1009. The processor 1001, the memory 1002, the storage 1003, the input device 1004, the output device 1005, the communication device 1006, the GPU 1007, and the reading device 1008 are connected to the bus 1009 and can transmit and receive data bidirectionally via the bus 1009.

[0258] The processor 1001 executes computer programs stored in the memory 1002 to realize the processing performed by each of the above-mentioned devices. Therefore, the processing performed by each of the above-mentioned devices can be interpreted as processing mainly performed by the processor 1001. Examples of the processor 1001 include a central processing unit (CPU), a micro processing unit (MPU), a controller, a large scale integration (LSI), an application specific integrated circuit (ASIC), a programmable logic device (PLD), and a field-programmable gate array (FPGA).

[0259] The memory 1002 stores computer programs and data used by the information processing device 1000. The memory 1002 may include a read-only memory (ROM) and a random access memory (RAM). The memory 1002 may be configured as either a volatile storage medium or a non-volatile storage medium.

[0260] The storage 1003 is configured as a nonvolatile storage medium and stores computer programs and data handled by the information processing apparatus 1000. Examples of the storage 1003 include a hard disk drive (HDD), a solid state drive (SSD), and a flash memory.

[0261] The input device 1004 accepts data to be input to the processor 1001. Examples of the input device 1004 include a keyboard, a mouse, a touchpad, and a microphone.

[0262] The output device 1005 outputs data generated by the processor 1001. For example, the output device 1005 outputs display data generated by the processor 1001 to the first display device 31 or the second display device 32. Alternatively, the output device 1005 may be a display or monitor that displays the display data itself. Alternatively, the output device 1005 outputs audio data generated by the processor 1001 to a speaker.

[0263] The communication device 1006 transmits and receives data to and from other devices via the communication network 4. The communication device 1006 may include a transmitter that transmits data and a receiver that receives data. The communication device 1006 may support either wired communication or wireless communication.

[0264] The GPU 1007 is a device that processes image rendering at high speed. The GPU 1007 may be used for AI processing (e.g., machine learning or deep learning).

[0265] The reading device 1008 is a device that reads data from an external recording medium such as a Digital Versatile Disk Read Only Memory (DVD-ROM) or a Universal Serial Bus (USB) memory.

[0266] Although the embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications, alterations, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also fall within the technical scope of the present disclosure. Furthermore, the components in the above-described embodiments may be combined in any manner without departing from the spirit of the invention.

[0267] This application is based on a Japanese patent application (Patent Application No. 2024-092889) filed on June 7, 2024, the contents of which are incorporated herein by reference.

[0268] The techniques of this disclosure are useful for monitoring unauthorized persons.

[0269] 1 Person 1A Face-registered person 1B Face-unregistered person 1C Tailgating person 2 Passage 3 Detection area 4 Communication network 9 Gate system 10 Gate device 11 First gate 12 Second gate 21 First image capturing device 22 Second image capturing device 31 First display device 32 Second display device 41 First edge device 42 Second edge device 50 Management device 60 Face matching device 70 Surveillance monitor 100 Person information 102 First image capturing time 103 Image update time 104 Body image 105 Body image update time 106 Body coordinates 107 Face image 108 Face image update time 109 Face coordinates 110 Face matching ID 111 Matching status 112 Movement direction 200 Body image 201, 201A, 201B Body coordinates 202 Body coordinates 220 Facial image 221, 221A Facial coordinates 301 Facial matching zone 302 Passage determination zone 303 Tailgating determination zone 311 Predetermined area closer to first gate 312 Predetermined area closer to second gate 401 Permitted person 402 Unpermitted person 411 Permitted person area 412 Unpermitted person area 1000 Information processing device 1001 Processor 1002 Memory 1003 Storage 1004 Input device 1005 Output device 1006 Communication device 1007 GPU 1008 Reading device 1009 Bus

Claims

1. An information processing device comprising a processor and a communication device, wherein the communication device receives a photographed image of a person passing through a specified passageway, the processor uses the photographed image to track the person, and if the person is located in a matching zone set on the entrance side of the passageway, performs matching on the person using the photographed image of the person, and if the person who has failed the matching is located in a passage determination zone set on the exit side of the passageway, displays information indicating that the matching has failed on a specified monitor, and if the person who has failed the matching exits from the passage determination zone to the exit side, displays information indicating that the person is an unauthorized passerby on the monitor.

2. The information processing device of claim 1, wherein the processor, when a person who has been successfully matched enters the passage determination zone and then returns to the match zone, displays on the monitor an image of the person along with information indicating a U-turn.

3. An information processing device as described in claim 1 or 2, wherein the processor, when a person who fails the match enters the passage determination zone and then returns to the match zone, displays on the monitor an image of the person together with information indicating a U-turn.

4. The information processing device of claim 3, wherein the processor erases the record of the photographed image of a person who failed the matching when that person enters the passage determination zone, returns to the matching zone, and then exits the matching zone.

5. A person matching system comprising a photographing device that generates photographed images of people passing through a specified passageway, a monitor, and an information processing device, wherein the information processing device: receives the photographed images from the photographing device; tracks the person using the photographed images; if the person is located in a matching zone set on the entrance side of the passageway, matches the person using the photographed image of the person; if the person who has failed to be matched is located in a passage judgment zone set on the exit side of the passageway, displays information on the monitor indicating that the match has failed; and if the person who has failed to be matched exits the passage judgment zone towards the exit, displays information on the monitor indicating that the person is an unauthorized passerby.

6. An information processing method comprising: receiving a photographed image of a person passing through a specified passageway; tracking the person using the received photographed image; if the person is located in a matching zone set on the entrance side of the passageway, matching the person using the photographed image of the person; if the person who fails to be matched is located in a passage judgment zone set on the exit side of the passageway, displaying information indicating that the matching has failed on a specified monitor; if the person who fails to be matched exits the passage judgment zone towards the exit, displaying information indicating that the person is an unauthorized passerby on the monitor.

Images