Information processing apparatus, person collation system and image display method

Abstract

To appropriately display an image of each person even when a plurality of persons overlap in an imaged image.SOLUTION: An information processing apparatus sets an allowed person area that includes an image of a person who is allowed for passage of a route and an un-allowed person area that includes an image of a person who is not allowed for passage of a route for an imaged image in which a plurality of persons passing through a predetermined route are imaged; generates an illegal pedestrian image, in which an image of at least a part included in the allowed person area is not displayed and an image included in the un-allowed person area is displayed when the allowed person area and the un-allowed person area partially overlap; and displays the illegal pedestrian image on a predetermined monitor.SELECTED DRAWING: Figure 17

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, a person matching system, and an image display method. [Background technology]

[0002] There is known a technique for detecting multiple people from an image captured by a camera and tracking each of the detected people. Patent Document 1 discloses a technique for generating an image of a person in the background, where the person in the background is partially hidden by a person in the foreground, by excluding the portion hidden by the person in the foreground and complementing the image of the excluded portion with the image of the remaining portion of the person in the background. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-128824 Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology disclosed in Patent Document 1, the interpolation cannot accurately reproduce the hidden parts of the person in the background, so the interpolated image of the person may give a different (wrong) impression from the actual person. On the other hand, if the person in the background is extracted and displayed from the captured image without interpolation, part of the person in the foreground will also be displayed.

[0005] An object of the present disclosure is to provide a technology for appropriately displaying an image of each person even when multiple people overlap in a captured image. [Means for solving the problem]

[0006] An information processing device according to one aspect of the present disclosure is an information processing device having a processor and a memory, wherein the processor, in cooperation with the memory, sets, for a captured image of a plurality of persons passing through a specified passageway, an allowed person area including images of persons who are permitted to pass through the passageway and an unallowed person area including images of persons who are not permitted to pass through the passageway, and when the allowed person area and the unallowed person area partially overlap, generates an unauthorized passerby image in which at least a portion of the image included in the allowed person area is not displayed and the image included in the unallowed person area is displayed, and displays the unauthorized passerby image on a specified monitor.

[0007] A person matching system according to one aspect of the present disclosure includes a photographing device that generates a photographed image of multiple persons passing through a specified passageway, a management device that sets, in the photographed image, an allowed person area including images of persons who are permitted to pass through the passageway and an unallowed person area including images of persons who are not permitted to pass through the passageway, and generates an unauthorized passerby image in which, when the allowed person area and the unallowed person area partially overlap, at least a portion of the image included in the allowed person area is not displayed and the image included in the unallowed person area is displayed, and a monitor that displays the unauthorized passerby image.

[0008] An image display method according to one aspect of the present disclosure sets, for a captured image of multiple people passing through a specified passageway, an allowed person area containing images of people who are permitted to pass through the passageway and an unallowed person area containing images of people who are not permitted to pass through the passageway, and if the allowed person area and the unallowed person area partially overlap, generates an unauthorized passerby image in which at least a portion of the image contained in the allowed person area is not displayed and the image contained in the unallowed person area is displayed, and displays the unauthorized passerby image on a specified monitor.

[0009] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. [Effects of the Invention]

[0010] According to the present disclosure, even when multiple people overlap in a captured image, the images of each person can be displayed appropriately. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram for explaining an overview of a gate system according to a first embodiment. [Figure 2] FIG. 10 is a diagram showing an example of displaying a matching result on the first display device or the second display device according to the first embodiment; [Figure 3] A block diagram showing a configuration example of a gate system according to a first embodiment. [Figure 4] FIG. 1 is a diagram for explaining a face image and a body image according to the first embodiment. [Figure 5] 1 is a flowchart showing an example of a detection process performed by a first edge device according to the first embodiment; [Figure 6] FIG. 1 shows an example of the configuration of person information according to the first embodiment. [Figure 7] 1 is a flowchart showing an example of overall processing performed by a management device according to a first embodiment; [Figure 8] Flowchart showing an example of face detection information processing (S202) shown in FIG. 7 [Figure 9] Flowchart showing an example of the body detection information processing (S204) shown in FIG. 7 [Figure 10] A flowchart showing an example of post-movement processing (S210) shown in FIG. 7. [Figure 11] Flowchart showing an example of face matching result information processing (S206) shown in FIG. 7 [Figure 12] 1 is a flowchart showing an example of a display process performed by a first edge device according to the first embodiment; [Figure 13] FIG. 1 is a diagram showing a face matching zone and a passage determination zone in a detection area according to the first embodiment; [Figure 14] FIG. 1 is a diagram for explaining an example of a display pattern in a first display device according to embodiment 1; [Figure 15A] FIG. 1 is a first diagram illustrating a method for detecting, from a personal information table, personal information that is presumed to belong to the same person as the person indicated by the body detection information according to the first embodiment; [Figure 15B] FIG. 2 is a second diagram for explaining a method for detecting, from the personal information table, personal information that is presumed to belong to the same person as the person indicated by the body detection information according to the first embodiment; [Figure 16] FIG. 1 is a diagram illustrating a method for detecting a person's moving direction according to the first embodiment. [Figure 17] FIG. 10 is a diagram for explaining an overview of a gate system according to a second embodiment. [Figure 18] Block diagram of a gate system according to a second embodiment [Figure 19] FIG. 10 is a diagram showing an example of displaying an image of an unauthorized passerby on a surveillance monitor according to the second embodiment. [Figure 20] 10 is a flowchart showing an example of processing performed by a management device according to a second embodiment. [Figure 21] FIG. 10 is a diagram illustrating a case where an unauthorized passerby image is generated without processing when an authorized person area overlaps at least a portion below an unauthorized person area; [Figure 22A] FIG. 10 is a diagram for explaining a first processing method for a non-permitted person area according to the second embodiment; [Figure 22B] FIG. 10 is a diagram for explaining a modification of the first processing method for the unauthorized person area according to the second embodiment; [Figure 23] FIG. 10 is a diagram for explaining a processing method when a plurality of permitted person regions overlap at least a part below a non-permitted person region according to the second embodiment; [Figure 24] FIG. 10 is a diagram for explaining a second processing method for a non-permitted person area according to the second embodiment; [Figure 25] FIG. 10 is a diagram for explaining a method for generating an unauthorized passerby image when an authorized person area overlaps at least a part of the upper part of a unauthorized person area according to the second embodiment. [Figure 26] FIG. 10 shows an example in which the display mode of an unauthorized passerby image is changed depending on the position of the unauthorized passerby on the passage according to the second embodiment. [Figure 27]FIG. 10 is a diagram showing an example of displaying a plurality of unauthorized passerby images according to the second embodiment. [Figure 28] FIG. 10 is a diagram showing an example of displaying a plurality of images of unauthorized passersby taken in time series in a video format according to the second embodiment. [Figure 29] FIG. 1 is a diagram illustrating an example of a hardware configuration of an information processing device according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with appropriate reference to the drawings. However, more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0013] (Embodiment 1) <Gate system overview> Fig. 1 is a diagram illustrating an overview of a gate system 9 according to embodiment 1. Fig. 2 is a diagram illustrating an example of how a matching result is displayed on the first display device 31 or the second display device 32 according to embodiment 1.

[0014] The gate system 9 is a system that collates a facial image of a person 1 and determines whether the person 1 has the right to pass through the gate. The gate system 9 includes a first gate 11 and a second gate 12. The first gate 11 and the second gate 12 are installed a predetermined distance apart in a passage 2 through which the person 1 moves. In this embodiment, the direction from the second gate 12 toward the first gate 11 is defined as a first direction, and the direction from the first gate 11 toward the second gate 12 is defined as a second direction. In this embodiment, the area between the first gate 11 and the second gate 12 in the passage 2 is referred to as a person detection area 3. Note that, although the first embodiment describes a configuration in which the gate system 9 includes two gates, the gate system 9 may also be configured to include only one gate (e.g., the first gate 11) and to collate a person 1 heading in one direction (e.g., the first direction).

[0015] For ease of explanation, the present embodiment will be described assuming that the axis along the first or second direction is the Y-axis, the axis perpendicular to the Y-axis and extending in the height direction is the Z-axis, and the axis perpendicular to the Y-axis and the Z-axis is the X-axis. The X-axis direction may also be referred to as the width direction of the gate.

[0016] The first gate 11 is equipped with a first imaging device 21 capable of capturing an image of at least a portion of the detection area 3, and a first display device 31 having a screen facing the second gate 12. In FIG. 1, the first gate 11 is shown equipped with three first imaging devices 21, but the first gate 11 may be equipped with one or two first imaging devices 21, or four or more first imaging devices 21. In addition, in FIG. 1, the first gate 11 is shown equipped with one first display device 31, but the first gate 11 may be equipped with two or more first display devices 31.

[0017] The second gate 12 is equipped with a second imaging device 22 capable of capturing an image of at least a portion of the detection area 3, and a second display device 32 having a screen facing the first gate 11. In FIG. 1, the second gate 12 is shown equipped with three second imaging devices 22, but the second gate 12 may be equipped with one or two second imaging devices 22, or four or more second imaging devices 22. In addition, in FIG. 1, the second gate 12 is shown equipped with one second display device 32, but the second gate 12 may be equipped with two or more second display devices 32.

[0018] In this embodiment, the first direction is the direction of entry into a facility or the like where the person has the right to enter or exit, and the second direction is the direction of exit from the facility or the like. However, the first direction may also be the direction of exit from a facility or the like, and the second direction may also be the direction of entry into the facility or the like.

[0019] The gate system 9 compares the facial image of the person 1 moving in the first direction captured by the first photographing device 21 at the first gate 11 with each pre-registered facial image for comparison, and displays the comparison result on the first display device 31, as shown in FIG. 2.

[0020] The gate system 9 compares the facial image of the person 1 moving in the second direction captured by the second photographing device 22 at the second gate 12 with each pre-registered facial image for comparison, and displays the comparison result on the second display device 32, as illustrated in FIG. 2.

[0021] The gate width of the first gate 11 and the second gate 12, or the width in the X direction of the detection area 3, may be a width that allows multiple people 1 to pass through side by side. As a result, the gate system 9 according to this embodiment allows more people 1 to enter and exit smoothly compared to a gate system with a gate width that allows only one person to pass through. For example, in a gate system with a gate width wide enough for one person, if the verification of a person 1 ahead fails, the following people 1 will be held up. However, in a gate system 9 according to this embodiment with a gate width that allows multiple people 1 to pass through side by side, even if the verification of a person 1 ahead fails, the following people 1 can simply move around the person 1 ahead who failed to be verified, and thus can pass through the gate smoothly without being held up. In other words, the gate system 9 according to this embodiment can prevent congestion when entering or exiting a facility, etc. The gate system 9 according to this embodiment will be described in detail below.

[0022] <Gate system configuration> Fig. 3 is a block diagram showing an example of the configuration of the gate system 9 according to the embodiment 1. Fig. 4 is a diagram for explaining face images and body images according to the embodiment 1.

[0023] The gate system 9 includes a first gate 11 and a second gate 12. The first gate 11 includes a first imaging device 21 and a first display device 31. The second gate 12 includes a second imaging device 22 and a second display device 32. The first gate 11, the first imaging device 21, the first display device 31, the second gate 12, the second imaging device 22, and the second display device 32 may be collectively referred to as a gate device 10.

[0024] The gate system 9 further includes a first edge device 41, a second edge device 42, a management device 50, and a face matching device 60. The first image capturing device 21 is connected to the first edge device 41 via an electrical cable (e.g., a USB cable or an Ethernet (registered trademark) cable). The first display device 31 is connected to the first edge device 41 via an electrical cable (e.g., an HDMI (registered trademark) cable). The second image capturing device 22 is connected to the second edge device 42 via an electrical cable (e.g., a USB cable or an Ethernet cable). The second display device 32 is connected to the second edge device 42 via an electrical cable (e.g., an HDMI cable). The first edge device 41 and the second edge device 42 are connected to the management device 50 via a communication network 4. The management device 50 is connected to the face matching device 60 via the communication network 4. Examples of the communication network 4 include a wired LAN, a wireless LAN, a mobile communication network (e.g., LTE, 4G, 5G), the Internet, and / or a VPN.

[0025] The first image capturing device 21 captures an image of the detection area 3 to generate a captured image, and transmits the captured image to the first edge device 41. The second image capturing device 22 captures an image of the detection area 3 to generate a captured image, and transmits the captured image to the second edge device 42. The captured image may be a still image or a moving image.

[0026] As shown in FIGS. 4(a) and 4(b), the first edge device 41 detects the entire body of the person 1 from the photographed image received from the first image capturing device 21. In this embodiment, the image of the entire body of the person 1 detected from the photographed image is referred to as a body image 200. The first edge device 41 transmits the detected body image 200 to the management device 50. Furthermore, as shown in FIG. 4(a), the first edge device 41 detects the face portion of the person 1 from the photographed image received from the first image capturing device 21. In this embodiment, the image of the face portion of the person 1 detected from the photographed image is referred to as a face image 220. The first edge device 41 transmits the detected face image 220 to the management device 50.

[0027] As shown in FIGS. 4(a) and 4(b), the second edge device 42 detects the entire body of the person 1 from the photographed image received from the second image capturing device 22. The second edge device 42 transmits a body image 200 detected from the photographed image to the management device 50. Also, as shown in FIG. 4(a), the second edge device 42 detects the face portion of the person 1 from the photographed image received from the second image capturing device 22. The second edge device 42 transmits a face image 220 detected from the photographed image to the management device 50. Details of the processes performed by the first edge device 41 and the second edge device 42 will be described later.

[0028] The management device 50 generates person information 100 (see FIG. 6) corresponding to the person 1 located in the detection area 3 based on the body image 200 or face image 220 received from the first edge device 41, or the body image 200 or face image 220 received from the second edge device 42. The management device 50 may generate person information 100 for each person 1 detected in the detection area 3. Details of the person information 100 will be described later (see FIG. 6). The management device 50 transmits a face matching request including the face image 220 to the face matching device 60.

[0029] The face matching device 60 stores in advance a face image for matching of a person 1 who has the right to pass through a gate (for example, to enter or exit). For example, the face matching device 60 stores a face image for matching of a person 1 who has already registered to enter the facility. The face matching device 60 may manage the face image for matching by linking it to the name, age, sex, credit card information, etc. of the person 1.

[0030] Furthermore, the face matching device 60 matches the face image 220 received from the management device 50 with each of the face images for matching. If the face matching device 60 is successful in matching with any of the registered face images for matching, it generates face matching result information including a successful matching result and transmits it to the management device 50. At this time, the face matching device 60 may perform billing in real time using credit card information linked to the successfully matched face image for matching. If the face matching device 60 fails to match with any of the registered face images for matching, it generates face matching result information including a failed matching result and transmits it to the management device 50. Note that the matching between the face image 220 and the face images for matching may be achieved by a known face image matching technology. Also, "matching" may be read as "authentication."

[0031] The management device 50 receives face matching result information from the face matching device 60. The management device 50 transmits person matching information including face matching result information for the face image 220 of the person 1 moving in the first direction to the first edge device 41, and transmits person matching information including face matching result information for the face image 220 of the person 1 moving in the second direction to the second edge device 42.

[0032] When the first edge device 41 receives person matching information from the management device 50, it displays the face matching result information included in the person matching information on the first display device 31, as shown in Fig. 2. This allows a person 1 located in the detection area 3 and moving in the first direction to know his / her own matching result by looking at the screen of the first display device 31 located in the direction of movement. In other words, the first edge device 41 allows a person 1 moving in the first direction to visually recognize the matching result of the person 1.

[0033] When the second edge device 42 receives person matching information from the management device 50, it displays the face matching result information included in the person matching information on the second display device 32, as shown in FIG. 2. This allows a person 1 located in the detection area 3 and moving in the second direction to view the screen of the second display device 32 located at the end of the moving direction to know his / her own matching result. In other words, the second edge device 42 allows a person 1 moving in the second direction to visually recognize the matching result of the person 1. Below, the processes performed by the first edge device 41, the second edge device 42, and the management device 50 will be described in detail.

[0034] <Detection process in first edge device> 5 is a flowchart showing an example of the detection process performed by the first edge device 41 according to embodiment 1. Note that, for the detection process performed by the second edge device 42 according to embodiment 1, in the following description with reference to FIG. 5, the "first image capturing device 21" can be read as the "second image capturing device 22."

[0035] The first edge device 41 receives a captured image from the first image capturing device 21 (S101).

[0036] The first edge device 41 receives a distance image from the first image capturing device 21 (S102). In this way, the first image capturing device 21 may generate a distance image indicating the distance to the person 1 or the like present in the detection area 3. The same applies to the second image capturing device 22. The distance image may be generated using any known technology, such as a stereo camera or a distance measuring sensor (e.g., LiDAR (Light Detection And Ranging)). Note that in this embodiment, the processing of step S102 is performed after the processing of step S101, but this is not limiting and the order may be reversed.

[0037] As shown in Fig. 4(a), the first edge device 41 detects a facial image 220 of the person 1 from the captured image (S103). The detection of the facial image 220 of the person may be realized using any known technology, such as pattern matching or AI (Artificial Intelligence). The AI ​​may also be an image recognition technology using deep learning (e.g., CNN (Convolutional Neural Network)).

[0038] The first edge device 41 determines whether or not the face image 220 was detected from the captured image in step S103 (S104). If the face image 220 was not detected (S104: NO), the first edge device 41 proceeds to step S108.

[0039] If the face image 220 is detected (S104: YES), the first edge device 41 uses the distance image to calculate three-dimensional coordinates (hereinafter referred to as face coordinates 221 (see FIG. 4)) for the detected face image 220 (S105). For example, as shown in FIG. 4(a), the first edge device 41 calculates the center point of the face image 220 as the face coordinates 221.

[0040] The first edge device 41 generates face detection information (S106). For example, the first edge device 41 generates face detection information including an ID (hereinafter referred to as a photographing device ID) for identifying the first photographing device 21 that is the sender of the photographed image, a face image 220, a size of the face image 220 (hereinafter referred to as a face image size), face coordinates 221, and a photographing time of the photographed image.

[0041] The first edge device 41 transmits the generated face detection information to the management device 50 (S107), and the process proceeds to the next step S108.

[0042] As shown in FIG. 4(a) or 4(b), the first edge device 41 detects a body image 200 of the person 1 from the captured image (S108). The detection of the body image 200 of the person 1 may be realized using any known technology, such as pattern matching or AI. The AI ​​may also be an image recognition technology using deep learning (e.g., CNN).

[0043] The first edge device 41 determines whether or not the body image 200 was detected from the captured image in step S108 (S109). If the body image 200 was not detected (S109: NO), the first edge device 41 returns the process to step S101.

[0044] If the body image 200 is detected (S109: YES), the first edge device 41 uses the distance image to calculate three-dimensional coordinates (hereinafter referred to as body coordinates 201 (see FIG. 4)) of the detected body image 200 (S110). For example, the first edge device 41 calculates the center point of the body image 200 as the body coordinates 201, as shown in FIGS. 4(a) and 4(b).

[0045] The first edge device 41 generates body detection information (S111). For example, the first edge device 41 generates body detection information including an imaging device ID, a body ID for identifying the body image 200, the body image 200, body coordinates 201, and the imaging time of the captured image.

[0046] The first edge device 41 transmits the generated body detection information to the management device 50 (S112), and the process returns to step S101. Note that in this embodiment, the process of detecting a face image from the captured image (S103 to S107) is performed before the process of detecting a body image from the captured image (S108 to S112), but this is not limiting and the steps may be reversed.

[0047] In this way, by performing the process of detecting the face image 220 and / or the body image 200 of the person 1 from the captured image and the process of calculating the face coordinates 221 and / or the body coordinates 201 in the first edge device 41 or the second edge device 42, the processing load can be distributed compared to when these processes are performed in the management device 50. Note that if distribution of the processing load is not a consideration, the gate system 9 does not need to include the first edge device 41 and the second edge device 42. In this case, the first image capturing device 21, the first display device 31, the second image capturing device 22, and the second display device 32 may be directly connected to the management device 50. Then, the management device 50 may perform the process of detecting the face image 220 and / or the body image 200 of the person 1 from the captured image and the process of calculating the face coordinates 221 and / or the body coordinates 201.

[0048] <Person information> FIG. 6 shows an example of the configuration of the person information 100 according to the first embodiment.

[0049] Person information 100 is generated for each person 1 detected in the detection area 3. The person information 100 includes, as information or data, a tracking ID 101, a first image capture time 102, an image update time 103, a body image 104, a body image update time 105, body coordinates 106, a face image 107, a face image update time 108, face coordinates 109, a face matching ID 110, a matching status 111, and a movement direction 112.

[0050] The tracking ID 101 is set to an ID used to track the same person in the detection area 3. The tracking ID 101 can also be said to be an ID for identifying the person information 100.

[0051] The first image capturing time 102 is set to the time when the body image 104 or face image 107 included in the personal information 100 was first captured. The first image capturing time can also be said to be the time when the personal information 100 was first generated.

[0052] The image update time 103 is set to the time when the body image 104 or face image 107 included in the personal information 100 was last updated. The image update time 103 can also be said to be the time when the personal information 100 was last updated.

[0053] The body image 104 is set to a body image 200 detected from a photographed image of the person 1 indicated by the personal information 100. The body image 104 may be updated as needed to the body image 200 detected from the latest photographed image.

[0054] The body image update time 105 is set to the time when the body image 104 included in the personal information 100 was last updated.

[0055] The body coordinates 201 calculated based on the body image 104 are set as the body coordinates 106. The body coordinates 106 may be expressed as three-dimensional coordinates in the detection area 3 (X coordinate, Y coordinate, Z coordinate).

[0056] A face image 220 detected from a photographed image of person 1 indicated by the personal information 100 is set as the face image 107. The face image 107 may be updated as needed to the face image 220 detected from the latest photographed image.

[0057] The face image update time 108 is set to the time when the face image 107 included in the person information 100 was last updated.

[0058] The face coordinates 109 are set to face coordinates 221 calculated based on the face image 107. The face coordinates 109 may be expressed as three-dimensional coordinates (X coordinate, Y coordinate, Z coordinate) in the detection area 3.

[0059] An ID assigned to a successfully matched face image for matching is set in face matching ID 110. In other words, face matching ID 110 can also be said to be an ID that certifies person 1 corresponding to face image 107. If there is no successfully matched face image for matching (i.e., if face matching fails), NULL may be set in face matching ID 110.

[0060] The matching status 111 is set to the matching status for the face image 107. For example, the matching status 111 is set to one of "before matching request", "matching in progress", "matching successful", and "matching failed".

[0061] The moving direction 112 is set to the moving direction of the person 1 indicated by the person information 100. For example, the moving direction is set to either the first direction or the second direction.

[0062] The management device 50 manages a plurality of pieces of personal information 100 as a personal information table (not shown). The personal information table may be stored in a memory 1002 or a storage 1003 (see FIG. 29 ) provided in the management device 50, or may be stored in a device different from the management device 50.

[0063] <Overall processing in the management device> FIG. 7 is a flowchart showing an example of the overall process performed by the management device 50 according to the first embodiment.

[0064] The management device 50 determines whether or not face detection information (see step S107 in FIG. 5) has been received from the first edge device 41 or the second edge device 42 (S201).

[0065] When the management device 50 receives face detection information (S201: YES), the management device 50 executes face detection information processing (S202). Details of the face detection information processing will be described later (see FIG. 8).

[0066] If the management device 50 has not received face detection information (S202: NO), it determines whether or not it has received body detection information (see step S112 in FIG. 5) from the first edge device 41 or the second edge device 42 (S203).

[0067] When the management device 50 receives the body detection information (S203: YES), the management device 50 executes body detection information processing (S204). Details of the body detection information processing will be described later (see FIG. 9).

[0068] If the management device 50 has not received the body detection information (S203: NO), it determines whether or not it has received face matching result information (S205). The face matching result information is information returned from the face matching device 60 in response to a face matching request transmitted from the management device 50 to the face matching device 60 in Fig. 8, which will be described later, and includes the result of face matching (for example, matching success or matching failure).

[0069] When the management device 50 receives the face matching result information (S205: YES), it executes face matching result information processing (S206). The face matching result information processing will be described in detail later (see FIG. 11).

[0070] If the management device 50 has not received the face matching result information (S205: NO), the processing returns to step S201.

[0071] After executing face detection information processing (S202) or body detection information processing (S204), the management device 50 executes post-movement processing (S210) and returns the processing to step S201. Details of the post-movement processing will be described later (see FIG. 10). Note that in this embodiment, the processing of steps S201, S203, and S205 is performed in stages, but this is not limiting, and the processing of steps S201, S203, and S205 may be interchanged or may be performed simultaneously in parallel.

[0072] <Face detection information processing in the management device> FIG. 8 is a flowchart showing an example of the face detection information processing (S202) shown in FIG.

[0073] The management device 50 detects, from the person information table, the person information 100 corresponding to the face detection information received from the first edge device 41 or the second edge device 42 (S301). In this embodiment, the person information 100 corresponding to the face detection information is person information 100 that is presumed to be the same person as person 1 indicated by the face detection information. Note that a method for detecting, from the person information table, the person information 100 that is presumed to be the same person as person 1 indicated by the face detection information will be described in detail later (see FIGS. 15A and 15B).

[0074] The management device 50 determines whether or not the personal information 100 has been detected from the personal information table in step S301 (S302).

[0075] Next, a case where it is determined in step S302 that the personal information 100 cannot be detected from the personal information table (S302: NO) will be described.

[0076] The management device 50 generates new personal information 100 based on the face detection information (S303). At this time, the management device 50 sets the following content in the new personal information 100. Set a new tracking ID to tracking ID 101 of person information 100 The face image 220 included in the face detection information is set to the face image 107 of the person information 100. Set the face coordinates 221 included in the face detection information to the face coordinates 109 of the person information 100 The first image capture time 102 of the person information 100 is set to the capture time included in the earliest face detection information (i.e., the earliest acquired capture time). Set "Before matching request" to the matching status 111 of the person information 100

[0077] The management device 50 detects the moving direction of the person 1 corresponding to the face detection information, and sets the detected moving direction as the moving direction 112 of the person information 100 generated in step S303 (S304). Note that the method for detecting the moving direction of a person will be described in detail later (see FIG. 16).

[0078] The management device 50 newly registers the personal information 100 generated and set in steps S303 and S304 in the personal information table (S305).

[0079] The management device 50 transmits a face matching request including the face image 107 and the tracking ID 101 to the face matching device 60, and updates the matching status 111 of the person information 100 to "matching in progress" (S306). Then, the management device 50 completes this process.

[0080] Next, a case where it is determined in step S302 that the personal information 100 has been detected from the personal information table (S302: YES) will be described.

[0081] The management device 50 updates the content of the person information 100 detected in step S301 based on the face detection information (S310). For example, the management device 50 updates the detected person information 100 as follows. Update the face coordinates 109 of the person information 100 to the face coordinates 221 included in the face detection information The image update time 103 and face image update time 108 of the person information 100 are updated to the shooting time included in the face detection information.

[0082] The management device 50 determines whether the matching status 111 of the detected person information 100 is "before matching request" or "matching failed" (S311). If the matching status 111 is not "before matching request" or "matching failed" (S311: NO), the management device 50 completes this process.

[0083] If the matching status 111 is "before matching request" or "matching failed" (S311: YES), the management device 50 transmits a face matching request including the face image 107 and the tracking ID 101 to the face matching device 60, and updates the matching status 111 of the person information 100 to "matching in progress" (S312). Then, the management device 50 completes this process.

[0084] <Body detection information processing in control devices> FIG. 9 is a flowchart showing an example of the body detection information processing (S204) shown in FIG.

[0085] The management device 50 detects, from the person information table, the person information 100 corresponding to the body detection information received from the first edge device 41 or the second edge device 42 (S401). In this embodiment, the person information 100 corresponding to the body detection information is the person information 100 that is presumed to be the same person as the person 1 indicated by the body detection information. Details of a method for detecting, from the person information table, the person information 100 that is presumed to be the same person as the person 1 indicated by the body detection information will be described later (see FIGS. 15A and 15B).

[0086] The management device 50 determines whether or not the personal information 100 has been detected from the personal information table in step S401 (S402).

[0087] Next, a case where it is determined in step S402 that the personal information 100 cannot be detected from the personal information table (S402: NO) will be described.

[0088] The management device 50 generates new personal information 100 based on the received body detection information (S403). At this time, the management device 50 sets the following content in the new personal information 100. Set a new tracking ID to tracking ID 101 of person information 100 The body image 200 included in the body detection information is set to the body image 104 of the person information 100. The body coordinates 202 included in the body detection information are set to the body coordinates 106 of the person information 100. The first image capture time 102 of the person information 100 is set to the capture time included in the earliest body detection information (i.e., the earliest acquired capture time).

[0089] The management device 50 detects the moving direction of the person 1 corresponding to the body detection information, and sets the detected moving direction as the moving direction 112 of the person information 100 generated in step S403 (S404). Note that the method for detecting the moving direction of the person 1 will be described in detail later (see FIG. 16).

[0090] The management device 50 newly registers the personal information 100 generated and set in steps S403 and S404 in the personal information table (S405), and then the management device 50 completes this process.

[0091] Next, a case where it is determined in step S402 that the personal information 100 has been detected from the personal information table (S402: YES) will be described.

[0092] The management device 50 updates the content of the person information 100 detected in step S401 based on the body detection information (S410). For example, the management device 50 updates the detected person information 100 as follows. The body coordinates 106 of the person information 100 are updated to the body coordinates 202 included in the body detection information. The image update time 103 and the body image update time 105 of the person information 100 are updated to the body image capture time included in the body detection information.

[0093] Then, the management device 50 completes this process.

[0094] <Post-movement processing in management device> FIG. 10 is a flowchart showing an example of the post-movement process (S210) shown in FIG.

[0095] The management device 50 generates person movement information based on the target person information (S501). In this embodiment, the target person information is person information 100 detected from the person information table or newly generated person information 100 in the face detection information processing shown in Fig. 8, or person information 100 detected from the person information table or newly generated person information 100 in the body detection information processing shown in Fig. 9. At this time, the management device 50 sets the following content in the person movement information. Tracking ID Zone Information -Facial image (or body image if no facial image is available) · Matching status Face Matching ID

[0096] Here, the zone information is information indicating whether the person 1 corresponding to the target person information is located in the face matching zone 301 or the passage determination zone 302 in the detection area 3. The face matching zone 301 and the passage determination zone 302 will be described in detail later (see FIG. 13).

[0097] The management device 50 determines whether the movement direction of the target person information is the first direction or the second direction (S502).

[0098] If the movement direction is the first direction (S502: first direction), the management device 50 transmits the person movement information generated in step S501 to the first edge device 41 (S503), and completes this process.

[0099] If the movement direction is the second direction (S502: second direction), the management device 50 transmits the person movement information generated in step S501 to the second edge device 42 (S504), and completes this process.

[0100] <Facial matching result information processing in control device> FIG. 11 is a flowchart showing an example of the face matching result information processing (S206) shown in FIG.

[0101] The management device 50 sets the content of the face matching result information received from the face matching device 60 in the personal information 100 corresponding to the face matching result information in the personal information table (S601). For example, the management device 50 sets the contents of the face matching result information in the person information 100 as follows. Set the face matching ID of the face matching result information to face matching ID 110 of person information 100 The matching result (matching success or failure) of the face matching result information is set to the matching status 111 of the person information 100

[0102] The management device 50 generates person verification information based on the person information 100 set in step S601 (S602). At this time, the management device 50 sets the following content in the person verification information. Tracking ID Zone Information Facial image · Matching status Face Matching ID

[0103] The management device 50 determines whether the movement direction 112 of the person information 100 set in step S601 is the first direction or the second direction (S603).

[0104] If the movement direction 112 is the first direction (S602: first direction), the management device 50 transmits the person verification information generated in step S602 to the first edge device 41 (S604), and completes this process.

[0105] If the movement direction 112 is the second direction (S602: second direction), the management device 50 transmits the person verification information generated in step S602 to the second edge device 42 (S605), and completes this process.

[0106] <Display processing in the first edge device> 12 is a flowchart showing an example of display processing performed by the first edge device 41 according to embodiment 1. Note that, for the display processing performed by the second edge device 42 according to embodiment 1, in the following description with reference to FIG. 12, "first edge device 41" should be read as "second edge device 42," "first display device 31" should be read as "second display device 32," and "first direction" should be read as "second direction."

[0107] The first edge device 41 determines whether or not it has received person movement information or person matching information from the management device 50 (S701). If the first edge device 41 has not received either person movement information or person matching information (S701: NO), it returns the process to step S701. If the first edge device 41 has received either person movement information or person matching information (S701: YES), it performs the following process.

[0108] The first edge device 41 determines whether a face image or a body image corresponding to the tracking ID included in the received person movement information or person matching information is displayed on the first display device 31 (S702).

[0109] If the face image or body image corresponding to the tracking ID is not displayed on the first display device 31 (S702: NO), the first edge device 41 newly displays the face image or body image and matching status included in the person movement information or person matching information on the first display device 31 (S703). Then, the first edge device 41 returns the process to step S701.

[0110] If a face image or body image corresponding to the tracking ID is displayed on the first display device 31 (S702: YES), the first edge device 41 updates the displayed face image or body image and matching status to the face image or body image and matching status included in the person movement information or person matching information and displays them (S704).Then, the first edge device 41 returns the process to step S701.

[0111] As a result, a matching result image showing the face image or body image of each person 1 moving in the first direction and the matching status (for example, matching success or failure, etc.) is displayed on the first display device 31, as shown in Fig. 2. Therefore, each person 1 moving in the first direction can visually confirm their own matching result from the first display device 31 installed at the destination in the direction of movement.

[0112] Similarly, as shown in Fig. 2, the second display device 32 displays a matching result image showing the face image or body image of each person 1 moving in the second direction and the matching status (for example, matching success or failure, etc.). Therefore, each person 1 moving in the second direction can visually confirm their own matching result from the second display device 32 installed at the destination in the direction of movement.

[0113] The above-mentioned display processing can be expressed as follows: That is, the management device 50 detects the movement direction of each person 1 moving between the first gate 11 and the second gate 12, and displays the matching result of the facial image 220 of the person 1 moving in the first direction on the first display device 31 but not on the second display device 32, and displays the matching result of the facial image 220 of the person 1 moving in the second direction on the second display device 32 but not on the first display device 31.

[0114] As a result, the matching result for person 1 is displayed on the display device installed in the direction of movement of person 1, and is not displayed on the display device installed opposite the direction of movement of person 1. Therefore, the management device 50 can appropriately show the matching result to each of person 1 moving in the first direction along the passage 2 and person 1 moving in the second direction.

[0115] <Display pattern> 13 is a diagram showing face matching zone 301 and passage determination zone 302 in detection area 3 according to embodiment 1. Note that, although the following description will be given taking first gate 11 and the first direction as an example, the same applies to second gate 12 and the second direction.

[0116] The image captured by the first photographing device 21 at the first gate 11 is used for face matching of the person 1 moving in the first direction. In the detection area 3, a predetermined area close to the second gate 12 is set as a face matching zone 301, and a predetermined area close to the first gate 11 is set as a passage determination zone 302. The first edge device 41 may change the display pattern on the first display device 31 depending on whether the person 1 is located in the face matching zone 301 or the passage determination zone 302. Next, the display pattern will be described.

[0117] 14 is a diagram for explaining an example of a display pattern in the first display device 31 according to embodiment 1. Note that, in the following description with reference to FIG. 14, for the display pattern in the second display device 32, the "first edge device 41" can be read as the "second edge device 42," and the "first display device 31" can be read as the "second display device 32."

[0118] If the person 1 is located outside the detection area 3 (outside the zone), the first edge device 41 does not display the matching result image of the person 1 on the first display device 31.

[0119] When person 1 is located in face matching zone 301, first edge device 41 controls the display of first display device 31 according to the following patterns (A0) to (A3). (A0) When a facial image 220 of person 1 exists and the matching status 111 is “matching in progress”, the first edge device 41 displays a matching result image including the facial image 220 of person 1 and the message “matching in progress” on the first display device 31. (A1) When the face image 220 of person 1 exists and the matching status 111 is “matching successful,” the first edge device 41 displays a matching result image including the face image 220 of person 1 and a message “matching successful” on the first display device 31. At this time, the first edge device 41 may output a matching success sound from a speaker provided in the first display device 31. (A2) When the face image 220 of person 1 exists and the matching status 111 is “matching failed”, the first edge device 41 displays a matching result image including the face image 220 of person 1 and a message “matching failed” on the first display device 31. At this time, the first edge device 41 may output a matching failure sound from a speaker provided in the first display device 31. (A3) When the face image 220 of the person 1 does not exist but the body image 200 exists, the first edge device 41 displays a matching result image including the body image 200 of the person 1 and a “Warning” message on the first display device 31. At this time, the first edge device 41 may output a warning sound from a speaker provided in the first display device 31.

[0120] When the person 1 is located in the passage determination zone 302, the first edge device 41 controls the display of the first display device 31 according to the following patterns (A4) to (A7). (A4) When a facial image 220 of person 1 exists and the matching status 111 is “matching in progress”, the first edge device 41 displays a matching result image including the facial image 220 of person 1 and the message “matching in progress” on the first display device 31. (A5) When the face image 220 of person 1 exists and the matching status 111 is “matching successful,” the first edge device 41 displays a matching result image including the face image 220 of person 1 and a message “passed” on the first display device 31. At this time, the first edge device 41 may output a passing sound from a speaker provided in the first display device 31. (A6) When the face image 220 of person 1 exists and the matching status 111 is “matching failed,” the first edge device 41 displays a matching result image including the face image 220 of person 1 and a message “unauthorized passage” on the first display device 31. At this time, the first edge device 41 may output an unauthorized passage sound from a speaker provided in the first display device 31. (A7) When the face image 220 of the person 1 does not exist but the body image 200 exists, the first edge device 41 displays a matching result image including the body image 200 of the person 1 and a message saying "unauthorized passage" on the first display device 31. At this time, the first edge device 41 may output an unauthorized passage sound from a speaker provided in the first display device 31.

[0121] The display priority may be set as follows. The passage determination zone 302 has a higher display priority than the face matching zone 301. The "passing" message has the highest display priority, followed by the "matching successful" message. If there is a face image 220, the display priority of the "matching" message is the lowest. The face image 220 has a higher display priority than the body image 200 alone.

[0122] For example, the display priorities of the above-mentioned patterns (A0) to (A7) may be set as follows in order from lowest to highest: (A3)<(A7)<(A0)<(A4)<(A2)<(A6)<(A1)<(A5)

[0123] <Method for detecting personal information of the same person> 15A and 15B are diagrams for explaining a method for detecting, from a person information table, person information 100 that is presumed to belong to the same person as person 1 indicated by the body detection information in Embodiment 1. Note that in FIG. 15, the description will be given assuming that person 1 is moving in a first direction. In the case where person 1 is moving in a second direction, in the following description referring to FIG. 15, the "first camera device 21" should be read as the "second camera device 22," and the "second camera device 22" should be read as the "first camera device 21."

[0124] A body image 200 of person 1 moving in a first direction captured by the first camera 21 shows the front view of the body of the person 1. Hereinafter, the X and Y coordinates of body coordinates 201A calculated from the front view of the body image 200 will be referred to as the X coordinate 201Ax and the Y coordinate 201Ay of the front of the body, respectively. Furthermore, a face image 220 of person 1 moving in the first direction captured by the first camera 21 shows the front view of the face of the person 1. Hereinafter, the X and Y coordinates of face coordinates 221A calculated from the front view of the face image 220 will be referred to as the X coordinate 221Ax and the Y coordinate 221Ay of the front of the face, respectively.

[0125] The body image 200 of the person 1 moving in the first direction captured by the second image capturing device 22 shows the back side of the body of the person 1. Hereinafter, the X coordinate and the Y coordinate of the body coordinate 201B calculated from the back side body image 200 will be referred to as the X coordinate 201Bx and the Y coordinate 201By of the back side of the body, respectively.

[0126] As shown in FIG. 15A , the Y coordinate 201Ay of the front face and the Y coordinate 201By of the back face are spaced apart by the thickness of the person 1's body in the front-to-back direction. Therefore, if the positional difference between the Y coordinate 201Ay of the front face and the Y coordinate 201By of the back face is smaller than a predetermined threshold W based on the body thickness (hereinafter referred to as the body thickness threshold), it can be assumed that the Y coordinate 201Ay of the front face and the Y coordinate 201By of the back face correspond to the Y coordinates of the same person. The body thickness threshold W may be a value slightly larger than the average body thickness. Similarly, if the positional difference between the Y coordinate 221Ay of the front face and the Y coordinate 201By of the back face is smaller than the body thickness threshold W, it can be assumed that the Y coordinate 221Ay of the front face and the Y coordinate 201By of the back face correspond to the Y coordinates of the same person.

[0127] 15B, if the amount of change in the X component between X-coordinate 201Ax of body coordinate 201A included in the received body detection information (i.e., the current X-coordinate) and X-coordinate 201Ax of body coordinate 201A included in personal information 100 (i.e., the past X-coordinate) is equal to or less than a predetermined threshold (hereinafter referred to as the body width threshold) W2, it can be estimated that X-coordinate 201Ax of the current body coordinate and X-coordinate 201Ax of the past body coordinate are the X-coordinates of the same person. Note that face coordinate 221A may be used instead of body coordinate 201A.

[0128] In other words, if the management device 50 can detect from the personal information table personal information 100 such that the positional difference between the X coordinate of the body coordinate 201 (or face coordinate 221) included in the received body detection information and the X coordinate of the body coordinate 201 (or face coordinate 221) included in the personal information 100 is less than the body width threshold W2, and the positional difference between the Y coordinate of the body coordinate 201 included in the received body detection information and the Y coordinate of the body coordinate 201 or the Y coordinate of the face coordinate 221 included in the personal information 100 is less than the body thickness threshold W, the management device 50 may estimate that the detected personal information 100 is the same person as the person 1 indicated by the body detection information.

[0129] Similarly, if the management device 50 can detect from the personal information table personal information 100 in which the positional difference between the X coordinate of the body coordinate 201 (or face coordinate 221) included in the received body detection information and the X coordinate of the body coordinate 201 (or face coordinate 221) included in the personal information 100 is smaller than the body width threshold W2, and the positional difference between the Y coordinate of the face coordinate 221 included in the received face detection information and the Y coordinate of the body coordinate 201 or the Y coordinate of the face coordinate 221 included in the personal information 100 is smaller than the body thickness threshold W, the management device 50 may estimate that the detected personal information 100 is the same person as the person 1 indicated by the face detection information.

[0130] The above-mentioned conditions regarding the X-coordinate positional difference and the body width threshold W2, and the conditions regarding the Y-coordinate positional difference and the body thickness threshold W are met when the time difference between the face photographing time included in the face detection information or the body photographing time included in the body detection information and the face image update time 108 or the body image update time 105 included in the person information 100 is sufficiently small (i.e., when the time difference is smaller than a predetermined time threshold T). Because person 1 is moving, if the time difference is large, person information 100 that satisfies the above-mentioned conditions regarding the X-coordinate positional difference and the body width threshold W2, and the conditions regarding the Y-coordinate positional difference and the body thickness threshold W is likely to belong to a different person who was in that position in the past.

[0131] 8, the following process may be performed: The management device 50 detects, from the personal information table, personal information 100 for which the time difference between the face capture time included in the face detection information and the face image update time 108 or the body image update time 105 included in the personal information 100 is less than a predetermined time threshold T, the positional difference between the X coordinate of the face coordinate 221 included in the face detection information and the X coordinate of the face coordinate 109 included in the personal information 100 is smaller than a body width threshold W2, and the positional difference between the Y coordinate of the face coordinate 221 included in the face detection information and the Y coordinate of the face coordinate 109 or the Y coordinate of the body coordinate 106 included in the personal information 100 is smaller than a body thickness threshold W.

[0132] 9, the following process may be performed. The management device 50 detects, from the personal information table, personal information 100 in which the time difference between the body photographing time included in the body detection information and the face image update time 108 or the body image update time 105 included in the personal information 100 is less than a predetermined time threshold T, the positional difference between the X coordinate of the body coordinate 201 included in the body detection information and the X coordinate of the body coordinate 106 included in the personal information 100 is smaller than a body width threshold W2, and the positional difference between the X coordinate of the body coordinate 201 included in the body detection information and the Y coordinate of the face coordinate 109 or the Y coordinate of the body coordinate 106 included in the personal information 100 is smaller than a body thickness threshold W.

[0133] The above content can be expressed as follows: That is, if the difference between the position (e.g., body coordinates 201 or face coordinates 221) of person 1 included in a first image (e.g., body image 200 or face image 220) captured by the first image capturing device 21 and the position (e.g., body coordinates 201 or face coordinates 221) of person 1 included in a second image (e.g., body image 200 or face image 220) captured by the second image capturing device 22 is smaller than a predetermined threshold W based on the thickness of the person's body, the management device 50 estimates that the position of person 1 included in the first image and the position of person 1 included in the second image represent the positions of the same person.

[0134] This allows the management device 50 to estimate whether, for example, a first image capturing the front of person 1 and a second image capturing the back of person 1 are of the same person.

[0135] <Method for detecting the direction of movement of a person> FIG. 16 is a diagram for explaining a method for detecting the moving direction of the person 1 according to the first embodiment.

[0136] The management device 50 detects the moving direction of the person 1 by, for example, any one of the following first, second, and third methods.

[0137] <<First Method>> The management device 50 detects the movement direction 112 of the person 1 based on the movement direction of the face coordinates 109 or the body coordinates 106 included in the person information 100 over time. For example, when the management device 50 detects that the Y coordinate of the face coordinates 109 or the Y coordinate of the body coordinates 106 included in the person information 100 is moving in the negative direction of the Y axis over time, the management device 50 detects the movement direction 112 of the person 1 as a first direction. When the management device 50 detects that the Y coordinate of the face coordinates 109 or the Y coordinate of the body coordinates 106 included in the person information 100 is moving in the positive direction of the Y axis over time, the management device 50 detects the movement direction 112 of the person 1 as a second direction.

[0138] <<Second Method>> The management device 50 determines whether the first detected image of the person 1 was captured by the first camera 21 or the second camera 22, based on the camera ID. If the first detected image of the person 1 was captured by the first camera 21, the management device 50 detects the movement direction 112 of the person 1 as the first direction. This is because the person 1 entering the detection area 3 from the second gate 12 is first captured by the first camera 21 provided at the first gate 11. If the first detected image of the person 1 was captured by the second camera 22, the management device 50 detects the movement direction 112 of the person 1 as the second direction. This is because the person 1 entering the detection area 3 from the first gate 11 is first captured by the second camera 22 provided at the second gate 12.

[0139] <<Third Method>> As shown in FIG. 16 , when the face coordinates 109 or body coordinates 106 of person 1 detected for the first time are included in a predetermined area 312 of the detection area 3 that is closer to the second gate 12, the management device 50 detects the movement direction 112 of the person 1 as a first direction. This is because person 1 entering the detection area 3 from the second gate 12 first enters the predetermined area 312 closer to the second gate 12. When the face coordinates 109 or body coordinates 106 of person 1 detected for the first time are included in a predetermined area 311 of the detection area 3 that is closer to the first gate 11, the management device 50 detects the movement direction 112 of person 1 as a second direction. This is because person 1 entering the detection area 3 from the first gate 11 first enters the predetermined area 311 closer to the first gate 11.

[0140] (Embodiment 2) Fig. 17 is a diagram illustrating an overview of a gate system 9 according to embodiment 2. Fig. 18 is a block diagram of the gate system 9 according to embodiment 2. Fig. 19 is a diagram illustrating an example of a display of an unauthorized passerby image 500 on a surveillance monitor 70 according to embodiment 2.

[0141] As shown in FIG. 17 , the gate system 9 according to the second embodiment includes a surveillance monitor 70 in addition to the devices shown in FIG. 3 . In the second embodiment, in addition to the processing described in the first embodiment, the management device 50 performs processing to display an image (hereinafter referred to as an unauthorized pedestrian image 500) of a person who does not have the right to pass through the gate device 10 (hereinafter referred to as an unauthorized pedestrian) on the surveillance monitor 70. For example, the surveillance monitor 70 is installed near the exit of the gate device 10, and an observer 80 monitors the surveillance monitor 70 and people passing through the gate device 10 near the exit of the gate device 10. In this way, when the unauthorized pedestrian image 500 is displayed on the surveillance monitor 70, the observer 80 can quickly find the unauthorized pedestrian by comparing the unauthorized pedestrian image 500 with the person passing through the gate device 10.

[0142] The second embodiment will be described in detail below. Note that the methods described in the first embodiment may be applied to the person's face matching method and tracking method. Furthermore, the second embodiment will describe a case where the first gate 11 in the first embodiment is the exit gate, the second gate 12 is the entrance gate, and a person moving in a first direction is judged as having unauthorized passage. However, the contents of the second embodiment also apply to a case where the first gate 11 in the first embodiment is the entrance gate, the second gate 12 is the exit gate, and a person moving in a second direction is judged as having unauthorized passage. Furthermore, in the following description, a person whose face matching is successful may be referred to as an authorized person 401, and a person whose face matching is unsuccessful may be referred to as an unauthorized person 402.

[0143] <Flowchart> FIG. 20 is a flowchart showing an example of processing performed by the management device 50 according to the second embodiment.

[0144] The management device 50 detects and tracks a person from an image captured by the first image capturing device 21, and stores the captured image in the memory 1002 or the storage 1003 (see FIG. 29) (S1001). The memory 1002 or the storage 1003 stores a plurality of captured images in chronological order.

[0145] The management device 50 performs face matching on the detected face image of the person (S1002). The face matching may be performed by the method described in the first embodiment.

[0146] The management device 50 sets a permitted person area 411 in an area of ​​the captured image that shows a person whose face has been successfully matched (permitted person), and sets a non-permitted person area 412 in an area of ​​the captured image that shows a person whose face has not been matched (non-permitted person) (S1003). At this time, the management device 50 may set the area so as to include as much of the person's entire body as possible. Furthermore, the permitted person area 411 and non-permitted person area 412 may be rectangular in shape.

[0147] The management device 50 determines whether or not the permitted person area 411 and the non-permitted person area 412 overlap each other (S1004).

[0148] If the permitted person area 411 and the non-permitted person area 412 do not overlap at all (S1004: NO), the management device 50 proceeds to step S1007.

[0149] If the permitted person area 411 and the non-permitted person area 412 partially overlap (S1004: YES), the management device 50 determines whether the permitted person area 411 overlaps at least a part of the upper part of the non-permitted person area 412 (S1005). For example, when viewed from the first image capturing device 21, if the permitted person is located behind the non-permitted person, the permitted person area 411 overlaps at least a part of the upper part of the non-permitted person area 412.

[0150] If it is determined in step S1005 that the permitted person area 411 overlaps at least a portion of the upper part of the non-permitted person area 412 (S1005: YES), the management device 50 proceeds to step S1007.

[0151] If it is determined in step S1005 that the permitted person area 411 does not overlap at least a portion of the upper part of the non-permitted person area 412 (S1005: NO), that is, if the permitted person area 411 overlaps at least a portion of the non-permitted person area 412 other than the upper part (for example, at least a portion of the lower part), the management device 50 performs the following process. That is, the management device 50 processes the image of the permitted person in the non-permitted person area 412 so that it cannot be seen, and generates the unauthorized passerby image 500 (S1006). Details of this processing process will be described later. Then, the process proceeds to step S1008.

[0152] In step S1007, the management device 50 generates an unauthorized passerby image 500 based on the image of the unauthorized person area 412 (S1007). For example, the management device 50 generates the unauthorized passerby image 500 using the image of the unauthorized person area 412 as is. Details of this process will be described later. Then, the process proceeds to step S1008.

[0153] In step S1008, the management device 50 displays the unauthorized passerby image 500 generated in step S1006 or step S1007 on the surveillance monitor 70, and the process returns to step S1001. When the management device 50 detects an unauthorized person 402 in the face matching zone 301 shown in FIG. 13 , the management device 50 may display the unauthorized passerby image 500 at the timing when the unauthorized person 402 enters the passage determination zone 302. However, the timing when the management device 50 displays the unauthorized passerby image 500 is not limited to this. For example, when the management device 50 detects an unauthorized person 402, the management device 50 may immediately display the unauthorized passerby image 500. Note that a method for displaying the unauthorized passerby image 500 on the surveillance monitor 70 will be described in detail later.

[0154] Through the above process, the unauthorized passerby image 500 corresponding to the unauthorized passerby is displayed on the surveillance monitor 70. Therefore, the surveillance staff 80 can quickly find the unauthorized passerby by looking at the surveillance monitor 70.

[0155] <When the permitted person area overlaps at least a portion of the unauthorized person area other than the upper portion (for example, at least a portion of the lower portion)> 21 is a diagram illustrating a case where an unauthorized passerby image is generated without processing when the permitted person area 411 overlaps at least a portion of the lower part of the prohibited person area 412. Note that "above the prohibited person area 412" may be a predetermined range including the upper side of the prohibited person area 412. "Below the prohibited person area 412" may be below the predetermined range including the upper side of the prohibited person area 412.

[0156] When the permitted person area 411 overlaps at least a portion of the lower part of the prohibited person area 412, if the image of the prohibited person area 412 is used as an unauthorized passerby image without any processing, the unauthorized passerby image will include not only the facial image of the prohibited person but also the facial image of the authorized person, as shown in image 601 in Figure 21. If such an unauthorized passerby image is displayed on the surveillance monitor 70, the surveillance officer 80 may have difficulty determining which of the facial images included in the unauthorized passerby image is the unauthorized passerby. Furthermore, it is undesirable to display the facial image of a person who is not an unauthorized passerby on the surveillance monitor 70 as an unauthorized passerby image.

[0157] 20, the management device 50 performs processing so that the facial image of the authorized person is not displayed in the unauthorized passerby image 500, thereby generating the unauthorized passerby image 500. For example, the management device 50 performs either the following first processing method or second processing method.

[0158] Fig. 22A is a diagram for explaining a first processing method for a non-permitted person region according to Embodiment 2. Hereinafter, the processing of step S1006 in Fig. 20 will be described in detail with reference to Fig. 22A.

[0159] If the permitted person area 411 overlaps at least a portion of the lower part of the non-permitted person area 412, that is, if a permitted person is present in front of the non-permitted person when viewed from the exit side, the management device 50 performs processing to cut out the portion of the non-permitted person area 412 that is overlapped by the permitted person area 411, as shown in image 602A, to generate unauthorized passerby image 500A. As a result, the facial image of the permitted person is not included in unauthorized passerby image 500A.

[0160] Alternatively, if the permitted person area 411 overlaps at least a portion of the lower part of the non-permitted person area 412, that is, if a permitted person is present in front of the non-permitted person as viewed from the exit side, the management device 50 performs processing to mask the portion of the image of the non-permitted person area 412 that is overlapped by the permitted person area 411, as shown in image 603A, to generate unauthorized passerby image 500B. As a result, the facial image of the permitted person cannot be seen in unauthorized passerby image 500B.

[0161] Through the above processing, the surveillance monitor 70 can display the unauthorized passerby image 500 (500A, 500B) in which the face image of the authorized person is not displayed.

[0162] Fig. 22B is a diagram for explaining a modification of the first processing method of the unauthorized person area according to Embodiment 2. The management device 50 may apply the processing method shown in Fig. 22B instead of the processing method shown in Fig. 22A.

[0163] When the permitted person area 411 overlaps at least a portion of the lower part of the disallowed person area 412, that is, when viewed from the exit side, when a permitted person is present in front of the disallowed person, the management device 50 may perform processing to cut out the portion of the facial image included in the permitted person area 411 from the image of the disallowed person area 412, as shown in image 602B, to generate unauthorized passerby image 500A. As a result, the facial image of the permitted person is not included in unauthorized passerby image 500A.

[0164] Alternatively, if the permitted person area 411 overlaps at least a portion of the lower part of the non-permitted person area 412, that is, if a permitted person is present in front of the non-permitted person when viewed from the exit side, the management device 50 performs processing to mask the portion of the facial image included in the permitted person area 411 of the image of the non-permitted person area 412, as shown in image 603B, to generate unauthorized passerby image 500B. As a result, the facial image of the permitted person cannot be seen in unauthorized passerby image 500B.

[0165] Through the above processing, the surveillance monitor 70 can display the unauthorized passerby image 500 (500A, 500B) in which the face image of the authorized person is not displayed.

[0166] FIG. 23 is a diagram for explaining a processing method when a plurality of permitted person regions 411 overlap at least partly below a non-permitted person region 412 according to the second embodiment.

[0167] When multiple permitted person regions 411 overlap at least a portion below the non-permitted person region 412, that is, when multiple permitted persons are present in front of the non-permitted person as viewed from the exit side, the management device 50 generates an unauthorized passerby image 500C by cutting out the overlapping portions of the permitted person regions 411 from the image of the non-permitted person region 412, as shown in image 604. Note that, as in FIG. 22B , the management device 50 may also generate an unauthorized passerby image 500C by cutting out the facial image portions of the permitted person regions 411 from the image of the non-permitted person region 412.

[0168] Alternatively, when multiple permitted person regions 411 overlap at least a portion below the non-permitted person region 412, that is, when multiple permitted persons are present in front of the non-permitted person as viewed from the exit side, the management device 50 performs processing to mask the portions of the image of the non-permitted person region 412 that are overlapped by the permitted person regions 411, as shown in image 605, to generate unauthorized passerby image 500D. Note that, as in FIG. 22B , the management device 50 may also perform processing to mask the portions of the facial images of the permitted person regions 411 from the image of the non-permitted person region 412 to generate unauthorized passerby image 500D.

[0169] 24 is a diagram for explaining a second processing method for the disallowed person area 412 according to Embodiment 2. Hereinafter, the processing of step S1006 in FIG. 20 will be described in detail with reference to FIG.

[0170] If the permitted person area 411 overlaps at least a portion of the lower part of the unauthorized person area 412, that is, if an authorized person is present in front of the unauthorized person when viewed from the exit side, the management device 50 performs processing to cut out the portion below the upper edge Z1 where the authorized person area 411 overlaps from the image of the unauthorized person area 412, as shown in image 606, to generate an unauthorized passerby image 500E. As a result, the unauthorized passerby image 500E does not include the facial image of the authorized person.

[0171] Alternatively, if the permitted person area 411 overlaps at least a portion of the lower part of the unauthorized person area 412, that is, if an authorized person is present in front of the unauthorized person when viewed from the exit side, the management device 50 generates unauthorized passerby image 500F by masking the portion of the image of the unauthorized person area 412 below the upper edge Z1 where the authorized person area 411 overlaps, as shown in image 607. This makes the facial image of the authorized person invisible in unauthorized passerby image 500F.

[0172] Through the above processing, the surveillance monitor 70 can display the unauthorized passerby image 500 (500E, 500F) in which the face image of the authorized person 401 is not displayed.

[0173] <When an authorized person area overlaps at least a portion above an unauthorized person area> 25 is a diagram for explaining a method for generating an unauthorized passerby image 500 in a case where the permitted person area 411 overlaps at least a part of the upper part of the non-permitted person area 412 according to Embodiment 2. Hereinafter, the processing of step S1007 in FIG. 20 will be described in detail with reference to FIG. 25.

[0174] When the permitted person area 411 overlaps at least a portion of the top of the prohibited person area 412, that is, when viewed from the exit side, when the permitted person 401 is located behind the prohibited person 402, the management device 50 may directly use the image of the prohibited person area 412 as the unauthorized passerby image 500G, as shown in image 608. This is because even if the image of the prohibited person area 412 is directly used as the unauthorized passerby image 500G, the facial image of the permitted person 401 will not be included in the unauthorized passerby image 500G.

[0175] Through the above processing, the surveillance monitor 70 can display the unauthorized passerby image 500 (500G) in which the face image of the authorized person 401 is not displayed.

[0176] <Details on displaying images of unauthorized passersby> The first photographing device 21 continuously photographs people passing through the passage 2, thereby generating multiple photographed images in chronological order. Accordingly, multiple unauthorized passerby images 500 of unauthorized passerby 402 are also generated in chronological order and stored in the memory 1002 or the storage 1003. The management device 50 may preferentially display, on the surveillance monitor 70, an unauthorized passerby image 500 that satisfies a predetermined condition A, among the multiple unauthorized passerby images 500 that have been generated. For example, the management device 50 may determine the unauthorized passerby image 500 to be preferentially displayed based on any one of the following conditions A1, A2, and A3.

[0177] <<Condition A1>> The management device 50 preferentially displays the unauthorized passerby image 500 generated using the most recently captured image among the multiple unauthorized passerby images 500 on the surveillance monitor 70. This allows the surveillance staff 80 to check the unauthorized passerby image 500 that includes the most recently captured facial image of the unauthorized person 402.

[0178] <<Condition A2>> The management device 50 preferentially displays the unauthorized passerby image 500 with the smallest proportion of the cut-out or masked area among the multiple unauthorized passerby images 500 on the surveillance monitor 70. This allows the surveillance staff 80 to check the unauthorized passerby image 500 in which the entire body of the unauthorized person 402 is more clearly visible.

[0179] <<Condition A3>> The management device 50 preferentially displays, among the multiple unauthorized passerby images 500, the unauthorized passerby image 500 having the largest ratio of maximum vertical length d2 to maximum horizontal length d1 (see FIGS. 22, 23, and 24) on the surveillance monitor 70. This allows the surveillance staff 80 to confirm the unauthorized passerby image 500 in which the entire body of the unauthorized person 402 is more clearly visible.

[0180] Furthermore, the plurality of unauthorized passerby images 500 under the above-mentioned condition A may be selected as unauthorized passerby images 500 that satisfy a predetermined condition B. For example, the management device 50 may select unauthorized passerby images 500 from the entire unauthorized passerby images 500 according to any one of the following conditions B1, B2, B3, and B4, and set them as the plurality of unauthorized passerby images 500 under the above-mentioned condition A.

[0181] <<Condition B1>> The management device 50 selects, from the entire unauthorized passerby image 500, an unauthorized passerby image 500 generated using an image captured in the face matching zone 301. In this way, an unauthorized passerby image 500 that is highly likely to show the entire body of the unauthorized person 402 is selected.

[0182] <<Condition B2>> The management device 50 selects, from all unauthorized passerby images 500, unauthorized passerby images 500 generated using images captured in the passage determination zone 302. In this way, unauthorized passerby images 500 that show the unauthorized person 402 after unauthorized passage has been confirmed are selected.

[0183] <<Condition B3>> The management device 50 selects unauthorized passerby images 500 that include the facial image of the unauthorized person 402 from all unauthorized passerby images 500. As a result, unauthorized passerby images 500 that do not include the facial image of the unauthorized person 402 are excluded.

[0184] <<Condition B4>> The management device 50 selects unauthorized passerby images in which the face of the unauthorized person 402 is facing forward within a predetermined range (for example, within a range from -α degrees to +α degrees) from all unauthorized passerby images 500. This allows the unauthorized passerby images 500 in which the face of the unauthorized person 402 is facing more forward.

[0185] The management device 50 may select the unauthorized passerby image 500 based on a combination of two or more of the above-mentioned conditions B1 to B4.

[0186] FIG. 26 is a diagram showing an example in which the display mode of the unauthorized passerby image 500 is changed depending on the position of the unauthorized person 402 on the passage 2 according to the second embodiment.

[0187] The management device 50 may change the display position of the unauthorized passerby image 500 on the surveillance monitor 70 in conjunction with the position of the unauthorized person 402 in the width direction of the passage 2.

[0188] For example, as shown in (a) of FIG. 26, when the unauthorized person 402 is located at the right end of the passage determination zone 302 as viewed from the exit side of passage 2, the management device 50 displays the unauthorized passerby image 500 on the right side of the surveillance monitor 70. Then, as shown in (b) and (c) of FIG. 26, when the unauthorized person 402 moves diagonally from the right end to the left end of the passage determination zone 302 as viewed from the exit side of passage 2, the management device 50 accordingly changes the display position of the unauthorized passerby image 500 from the right side to the left side of the surveillance monitor 70. This allows the monitor 80 to see the position of the unauthorized passerby image 500 displayed on the surveillance monitor 70 and grasp the position in passage 2 of the unauthorized passerby 402 corresponding to the unauthorized passerby image 500. This can improve the efficiency with which the monitor 80 captures the unauthorized person 402.

[0189] FIG. 27 is a diagram showing an example in which a plurality of unauthorized passerby images 500 according to the second embodiment are displayed.

[0190] When displaying one unauthorized passerby image 500, the management device 50 may display one unauthorized passerby image 500 using the entire screen of the surveillance monitor 70, as shown in FIG. 27(a).

[0191] When displaying a plurality of unauthorized passerby images 500 (500H, 500I), the management device 50 may increase the number of unauthorized passerby images 500 displayed by vertically dividing the screen of the surveillance monitor 70 as shown in (b) of FIG. 27. In this case, the management device 50 may display the unauthorized passerby image 500 with the most recent photographed time on the top row (or bottom row). The management device 50 may also increase the number of unauthorized passerby images 500 displayed by horizontally dividing the screen of the surveillance monitor 70. In this case, the management device 50 may display the unauthorized passerby image 500 with the most recent photographed time on the far right (or far left).

[0192] Furthermore, the management device 50 may prevent the display of multiple unauthorized passerby images 500 of the same unauthorized person 402. For example, the management device 50 may update a previous unauthorized passerby image 500 of the same unauthorized person 402 to a new unauthorized passerby image 500 captured at the time of capture, rather than displaying an additional unauthorized passerby image 500 of the same unauthorized person 402.

[0193] This allows the monitor 80 to view the surveillance monitor 70 and recognize the timing in chronological order when each of the multiple unauthorized persons 402 was detected. This allows the monitor 80 to, for example, prioritize capturing an older unauthorized person 402 in chronological order, or prioritize capturing a newer unauthorized person 402 in chronological order. In other words, the monitor 80 can view the surveillance monitor 70 and determine which unauthorized person 402 should be prioritized for capture.

[0194] Note that, when a match is successfully made with an unauthorized person 402 in the passage determination zone 302 (i.e., when the unauthorized person 402 is determined to be a permitted person 401), the management device 50 may automatically hide the display of the unauthorized passerby image 500 of the unauthorized person 402. This prevents the face image of a person who failed to be matched in the face matching zone 301 and was determined to be an unauthorized person 402, but was successfully matched in the passage determination zone 302 (permitted person 401), from being continuously displayed on the surveillance monitor 70.

[0195] FIG. 28 is a diagram showing an example of displaying a plurality of images 500 of an unauthorized passerby captured in time series like a moving image according to the second embodiment.

[0196] The management device 50 may display a plurality of unauthorized passerby images 500 captured in time series in chronological order on the surveillance monitor 70, and may also repeat the display in chronological order. For example, as shown in Fig. 28, the management device 50 may sequentially display an unauthorized passerby image 500 captured at time t = 1, an unauthorized passerby image 500 captured at time t = 2, an unauthorized passerby image 500 captured at time t = 3, and an unauthorized passerby image 500 captured at time t = 4, and then repeatedly return to displaying the unauthorized passerby image 500 captured at time t = 1.

[0197] Furthermore, even after the unauthorized person 402 has passed through the exit gate, the management device 50 may repeatedly display on the surveillance monitor 70, in chronological order as described above, multiple unauthorized passerby images 500 taken while the unauthorized person 402 was passing through passage 2.

[0198] As a result, the unauthorized passerby image 500 is displayed on the surveillance monitor 70 like a moving image, which makes it easier for the surveillance officer 80 to identify the unauthorized person 402 passing through the passage 2, compared to when the unauthorized passerby image 500 is displayed as a single still image. In addition, such a moving display attracts the attention of people who are attempting to pass illegally, which leads to a deterrent to unauthorized passage.

[0199] (Hardware configuration) FIG. 29 is a diagram illustrating an example of a hardware configuration of an information processing device according to the present disclosure.

[0200] The above-described first edge device 41, second edge device 42, management device 50, and face matching device 60 may be configured to include an information processing device 1000 shown in Fig. 29. Furthermore, the processes performed by the above-described first edge device 41, second edge device 42, management device 50, and face matching device 60 may be realized by the information processing device 1000 shown in Fig. 29 executing a computer program.

[0201] The information processing device 1000 includes a processor 1001, a memory 1002, a storage 1003, an input device 1004, an output device 1005, a communication device 1006, a graphics processing unit (GPU) 1007, a reading device 1008, and a bus 1009. The processor 1001, the memory 1002, the storage 1003, the input device 1004, the output device 1005, the communication device 1006, the GPU 1007, and the reading device 1008 are connected to the bus 1009 and can transmit and receive data bidirectionally via the bus 1009.

[0202] The processor 1001 executes a computer program stored in the memory 1002 to realize the processing performed by each of the above-mentioned devices. Therefore, the processing performed by each of the above-mentioned devices can be interpreted as processing mainly performed by the processor 1001. Examples of the processor 1001 include a central processing unit (CPU), a micro processing unit (MPU), a controller, a large scale integration (LSI), an application specific integrated circuit (ASIC), a programmable logic device (PLD), and a field-programmable gate array (FPGA).

[0203] The memory 1002 stores computer programs and data used by the information processing device 1000. The memory 1002 may include a read-only memory (ROM) and a random access memory (RAM). The memory 1002 may be configured as either a volatile storage medium or a non-volatile storage medium.

[0204] The storage 1003 is configured as a nonvolatile storage medium and stores computer programs and data handled by the information processing device 1000. Examples of the storage 1003 include a hard disk drive (HDD), a solid state drive (SSD), and a flash memory.

[0205] The input device 1004 accepts data to be input to the processor 1001. Examples of the input device 1004 include a keyboard, a mouse, a touchpad, and a microphone.

[0206] The output device 1005 outputs data generated by the processor 1001. For example, the output device 1005 outputs display data generated by the processor 1001 to the first display device 31 or the second display device 32. Alternatively, the output device 1005 may be a display or monitor that displays the display data itself. Alternatively, the output device 1005 outputs audio data generated by the processor 1001 to a speaker.

[0207] The communication device 1006 transmits and receives data to and from other devices via the communication network 4. The communication device 1006 may include a transmitter for transmitting data and a receiver for receiving data. The communication device 1006 may support either wired communication or wireless communication.

[0208] The GPU 1007 is a device that processes image rendering at high speed. The GPU 1007 may be used for AI processing (e.g., machine learning or deep learning).

[0209] The reading device 1008 is a device that reads data from an external recording medium such as a Digital Versatile Disk Read Only Memory (DVD-ROM) or a Universal Serial Bus (USB) memory.

[0210] (Summary of the second embodiment) The above description of the second embodiment discloses the following techniques.

[0211] <Technology 1> An information processing device (e.g., a management device 50) having a processor (1001) and a memory (1002), wherein the processor cooperates with the memory to set, for a photographed image of a plurality of persons passing through a specified passage (2), a permitted person area (411) including facial images of persons who are permitted to pass through the passage and a disallowed person area (412) including facial images of persons who are not permitted to pass through the passage, and when the permitted person area and the disallowed person area partially overlap, an unauthorized passerby image (500) is generated in which the facial images included in the permitted person area are not displayed and the facial images included in the disallowed person area are displayed, and the unauthorized passerby image is displayed on a specified monitor (e.g., a surveillance monitor 70). This allows the monitor (80) to recognize a person who is illegally passing through the passage by looking at the image of the illegal passerby displayed on the monitor, and also prevents an image of an illegal passerby that includes an image of a person who is permitted to pass from being displayed on the monitor.

[0212] <Technology 2> In the information processing device described in Technology 1, when an authorized person area overlaps at least a part of an unauthorized person area other than the upper part, the processor cuts out or masks the area in the image of the unauthorized person area that is partially overlapped by the authorized person area to generate an unauthorized passerby image. This makes it possible to prevent an image of an unauthorized passerby, including an image of a person who is permitted to pass, from being displayed on the monitor.

[0213] <Technology 3> In the information processing device described in Technology 1, when an authorized person area overlaps at least a part of the unauthorized person area other than the upper part, the processor cuts out or masks the facial image included in the authorized person area in the image of the unauthorized person area to generate an unauthorized passerby image. This makes it possible to prevent an image of an unauthorized passerby, including an image of a person who is permitted to pass, from being displayed on the monitor.

[0214] <Technology 4> In the information processing device according to any one of the first to third techniques, when the permitted person area overlaps at least a part of the upper part of the prohibited person area, the processor generates an unauthorized passerby image from the image of the prohibited person area. This makes it possible to generate an image of an unauthorized passerby that does not include facial images of people who are permitted to pass.

[0215] <Technology 5> In the information processing device according to any one of the first to fourth aspects, the processor preferentially displays on the monitor an unauthorized passerby image generated using a captured image with the latest capture time among a plurality of unauthorized passerby images. This allows the monitor to check the last (i.e., most recent) image of the unauthorized passerby on the monitor.

[0216] <Technology 6> In the information processing device according to any one of the first to fifth aspects, the processor preferentially displays on the monitor, among the plurality of unauthorized passerby images, an unauthorized passerby image having the smallest proportion of a cut-out or masked area. This allows the monitor to check an image of the unauthorized passerby on the monitor, which shows the entire body of the unauthorized passerby.

[0217] <Technology 7> In the information processing device according to any one of the first to sixth aspects, the processor preferentially displays on the monitor, among a plurality of unauthorized passerby images, an unauthorized passerby image having the largest ratio of maximum vertical length to maximum horizontal length. This allows the monitor to check an image of the unauthorized passerby on the monitor, which shows the entire body of the unauthorized passerby.

[0218] <Technology 8> In the information processing device according to any one of the first to seventh techniques, the captured image is captured by a capturing device installed so as to capture an image in the direction from the exit side to the entrance side of the passage. As a result, the photographing device captures an image including the face of the person walking through the passage toward the exit.

[0219] <Technology 9> In the information processing device according to either one of Techniques 1 or 8, the image of the unauthorized passerby is generated using a captured image including a face image that allows face matching of a person. This allows an image of the unauthorized passerby including the face of the unauthorized passerby to be generated.

[0220] <Technology 10> In the information processing device according to any one of the first to ninth techniques, the unauthorized passerby image is generated using a captured image including a face image in which the orientation of the face relative to the front of the person is within a predetermined range. This makes it possible to generate an image of the unauthorized passerby that includes a substantially frontal view of the face of the unauthorized passerby.

[0221] <Technology 11> In an information processing device described in any one of techniques 1 to 10, a processor changes the display mode on a monitor of an unauthorized passerby image corresponding to a person who is not allowed to pass, depending on the widthwise position of the passage where the person is detected. This allows the monitor to see the image of the unauthorized passerby displayed on the monitor and grasp the position of the unauthorized passerby in the passageway.

[0222] <Technology 12> In the information processing device according to either one of techniques 1 or 11, the processor changes the display mode of the unauthorized passerby images on the monitor depending on the number of unauthorized passerby images to be displayed on the monitor. This allows the monitor to display unauthorized passerby images corresponding to a plurality of unauthorized passersby even when a plurality of unauthorized passersby are detected.

[0223] <Technology 13> In the information processing device according to any one of the first to twelfth techniques, the processor displays a plurality of unauthorized passerby images consecutively in chronological order of their capture times. This allows the monitor to display a simple video-like (i.e., moving) image of an unauthorized passerby, which will attract the attention of security personnel and passersby and is expected to have a deterrent effect on those attempting to pass illegally.

[0224] <Technology 14> In the information processing device described in Technology 13, after a person who is not permitted to pass passes through a passageway, a processor repeatedly displays multiple images of unauthorized passersby, including images taken while the person was passing through the passageway, in chronological order of the time of the images being taken. This allows the monitor to display an image of the unauthorized passerby, like a simple video, even after the unauthorized passerby has passed through the passageway. This makes the monitor display eye-catching to passersby, and is expected to have a deterrent effect on those attempting to pass through the passageway illegally.

[0225] <Technology 15> In the information processing device described in any one of techniques 1 to 14, a face matching zone (301) for performing face matching is set on the entrance side of the passage, and a passage determination zone (302) for performing pass-through determination is set on the exit side of the passage, and the processor determines whether or not a person is permitted to pass through the passage based on the matching result of the face image included in the photographed image taken in the face matching zone, and when a person who has been determined not to be permitted to pass through the passage enters the passage determination zone, an unauthorized passerby image of the person is displayed. This allows the monitor to display an image of the unauthorized passerby at the time when the unauthorized passerby is confirmed.

[0226] <Technology 16> In the information processing device described in Technique 15, the unauthorized passerby image is generated using an image of a person who has been determined not to be permitted to pass through the passage, captured in a face matching zone. Since there is a high possibility that the whole body of a person is captured in an image captured in the face matching zone, the monitor can thereby display an image of an illegal passerby that is highly likely to include the whole body of the illegal passerby.

[0227] <Technology 17> In the information processing device described in Technique 15, the unauthorized passerby image is generated using an image of a person who has been determined not to be permitted to pass through the passage, taken in the passage determination zone. This allows the monitor to display an image of the unauthorized passerby captured after the unauthorized passerby has been confirmed.

[0228] <Technology 18> The person matching system includes a photographing device (e.g., a first photographing device 21) that generates a photographed image of a plurality of persons passing through a specified passage (2), a management device (50) that sets, in the photographed image, an allowed person area (411) including facial images of persons who are permitted to pass through the passage and a disallowed person area (412) including facial images of persons who are not permitted to pass through the passage, and generates an unauthorized passerby image (500) in which, when the permitted person area and the disallowed person area partially overlap, the facial image included in the allowed person area is not displayed and the facial image included in the disallowed person area is displayed, and a monitor (e.g., a surveillance monitor 70) that displays the unauthorized passerby image. This allows the monitor (80) to recognize a person who is illegally passing through the passage by looking at the image of the illegal passerby displayed on the monitor, and also prevents an image of an illegal passerby that includes an image of a person who is permitted to pass from being displayed on the monitor.

[0229] <Technology 19> The image display method sets, for a photographed image of a plurality of persons passing through a specified passage (2), an allowed person area (411) including facial images of persons who are permitted to pass through the passage and an unallowed person area (412) including facial images of persons who are not permitted to pass through the passage, and when the allowed person area and the unallowed person area partially overlap, generates an unauthorized passerby image (500) in which the facial images included in the allowed person area are not displayed but the facial images included in the unallowed person area are displayed, and displays the unauthorized passerby image on a specified monitor (e.g., surveillance monitor 70). This allows the monitor (80) to recognize a person who is illegally passing through the passage by looking at the image of the illegal passerby displayed on the monitor, and also prevents an image of an illegal passerby that includes an image of a person who is permitted to pass from being displayed on the monitor.

[0230] Although the embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications, alterations, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also fall within the technical scope of the present disclosure. Furthermore, the components in the above-described embodiments may be combined in any manner without departing from the spirit of the invention. [Industrial Applicability]

[0231] The techniques of this disclosure are useful for monitoring unauthorized persons. [Explanation of symbols]

[0232] 1 person 1A Face Registration Person 1B Person with unregistered face 1C Tailgating person 2 aisles 3 Detection Area 4. Communication Network 9 Gate System 10 Gate device 11 Gate 1 12 Gate 2 21 First Imaging Device 22 Second imaging device 31 1st display device 32 2nd display device 41 First Edge Device 42 Second Edge Device 50 Management device 60 Face matching device 70 Surveillance Monitor 100 Personal information 102 First image capture time 103 Image update time 104 Body Images 105 Body image update time 106 Body coordinates 107 Facial Images 108 Face image update time 109 Face Coordinates 110 Face Matching ID 111 Matching status 112 Direction of movement 200 Body Images 201, 201A, 201B body coordinates 202 Body coordinates 220 facial images 221, 221A face coordinates 301 Face Matching Zone 302 Passage Judgment Zone 303 Tailgating Judgment Zone 311 Designated area closer to Gate 1 312 Designated area closer to the second gate 401 Authorized Person 402 Unauthorized person 411 Permitted person area 412 Unauthorized person area 1000 computers 1001 processor 1002 memory 1003 Storage 1004 Input Device 1005 Output Device 1006 Communication equipment 1007 GPU 1008 Reading device 1009 Bus

Claims

1. An information processing device including a processor and a memory, wherein the processor cooperates with the memory to: For a photographed image of a plurality of persons passing through a predetermined passage, a permitted person area including images of persons who are permitted to pass through the passage and a non-permitted person area including images of persons who are not permitted to pass through the passage are set, When the permitted person area and the unauthorized person area partially overlap, an unauthorized passerby image is generated in which at least a part of the image included in the permitted person area is not displayed and the image included in the unauthorized person area is displayed; displaying the image of the unauthorized passerby on a predetermined monitor; Information processing device.

2. the image of the person included in the permitted person area includes a facial image of the person; The image of the person included in the unauthorized person area includes a facial image of the person. The information processing device according to claim 1 .

3. When the permitted person area overlaps at least a part of the prohibited person area other than the upper part, the processor cuts out or masks the area of ​​the image of the prohibited person area that is partially overlapped by the permitted person area to generate the unauthorized passerby image. The information processing device according to claim 2 .

4. When the permitted person area overlaps at least a part of the prohibited person area other than the upper part, the processor cuts out or masks a face image included in the permitted person area in the image of the prohibited person area to generate the unauthorized passerby image. The information processing device according to claim 2 .

5. the processor generates the unauthorized passerby image from the image of the unauthorized person area when the authorized person area overlaps at least a part of the upper part of the unauthorized person area; The information processing device according to claim 1 .

6. the processor preferentially displays on the monitor an unauthorized passerby image generated using the most recently captured image among the plurality of unauthorized passerby images. The information processing device according to claim 1 .

7. the processor preferentially displays on the monitor, among the plurality of unauthorized passerby images, an unauthorized passerby image having a smallest ratio of the area cut out or masked.

5. The information processing device according to claim 3 or 4.

8. the processor preferentially displays on the monitor, among the plurality of unauthorized passerby images, an unauthorized passerby image having the largest ratio of maximum vertical length to maximum horizontal length.

5. The information processing device according to claim 3 or 4.

9. The photographed image is photographed by a photographing device installed so as to photograph the direction from the exit side to the entrance side of the passage. The information processing device according to claim 1 .

10. The unauthorized passerby image is generated using the captured image including a face image that can be matched with a person's face. The information processing device according to claim 2 .

11. The unauthorized passerby image is generated using a photographed image including a facial image in which the orientation of the face relative to the front of the person is within a predetermined range. The information processing device according to claim 2 .

12. the processor changes the display mode of the unauthorized passerby image corresponding to the person who is not permitted to pass on the monitor according to the position in the width direction of the passage where the person is detected. The information processing device according to claim 1 .

13. the processor changes the display mode of the unauthorized passerby images on the monitor in accordance with the number of unauthorized passerby images to be displayed on the monitor. The information processing device according to claim 1 .

14. The processor displays the plurality of unauthorized passerby images in chronological order of their capture times. The information processing device according to claim 1 .

15. The processor: After the unauthorized person has passed through the passage, a plurality of unauthorized passerby images including images taken while the unauthorized passerby was passing through the passage are repeatedly displayed in chronological order of the photographing times. The information processing device according to claim 14.

16. A face matching zone for performing face matching is set on the entrance side of the passage, A passage determination zone for determining passage is set on the exit side of the passage, The processor: determining whether or not a person is permitted to pass through the passageway based on a comparison result of a face image included in a photographed image taken in the face comparison zone; When a person who has been determined not to be permitted to pass through the passage enters the passage determination zone, the unauthorized passerby image of the person is displayed. The information processing device according to claim 2 .

17. The unauthorized passerby image is generated using a photographed image of a person who has been determined not to be permitted to pass through the passage, photographed in the face matching zone. The information processing device according to claim 16.

18. The unauthorized passerby image is generated using a photographed image of a person who has been determined not to be permitted to pass through the passage, photographed in the passage determination zone. The information processing device according to claim 16.

19. an image capturing device that captures and generates captured images of a plurality of people passing through a predetermined passage; a management device that sets, for the captured image, a permitted person area including an image of a person who is permitted to pass through the passage and a disallowed person area including an image of a person who is not permitted to pass through the passage, and generates an unauthorized passerby image in which, when the permitted person area and the disallowed person area partially overlap, at least a portion of the image included in the permitted person area is not displayed and the image included in the disallowed person area is displayed; a monitor for displaying the image of the unauthorized passerby; Person matching system.

20. For a photographed image of a plurality of persons passing through a predetermined passage, a permitted person area including images of persons who are permitted to pass through the passage and a non-permitted person area including images of persons who are not permitted to pass through the passage are set, When the permitted person area and the unauthorized person area partially overlap, an unauthorized passerby image is generated in which at least a part of the image included in the permitted person area is not displayed and the image included in the unauthorized person area is displayed; displaying the image of the unauthorized passerby on a predetermined monitor; Image display method.

Images