Information processing device, person identification system, and image display method

Abstract

This information processing device: sets, with respect captured images obtained by capturing a plurality of persons passing through a passage, a permitted person region including an image of a person permitted to pass through the passage and a non-permitted person region including an image of a person not permitted to pass through the passage; generates, if the permitted person region and the non-permitted person region partially overlap, an unauthorized person image in which at least a part of the image included in the permitted person region is not displayed and in which an image included in the non-permitted person region is displayed; and displays the unauthorized person image on a monitor.

Description

Information processing device, person matching system, and image display method

[0001] The present disclosure relates to an information processing device, a person matching system, and an image display method.

[0002] There is known a technique for detecting multiple people from an image captured by a camera and tracking each of the detected people. Patent Literature 1 discloses a technique for, when a person in the background is partially hidden by a person in the foreground in an image captured by a camera, excluding the portion hidden by the person in the foreground when generating an image of the person in the background, and complementing the image of the excluded portion with the image of the remaining portion of the person in the background.

[0003] JP 2019-128824 A

[0004] In the technology disclosed in Patent Document 1, the interpolation cannot accurately reproduce the hidden parts of the person in the background, so the interpolated image of the person may give a different (wrong) impression from the actual person. On the other hand, if the person in the background is extracted and displayed from the captured image without interpolation, part of the person in the foreground will also be displayed.

[0005] An object of the present disclosure is to provide a technology for appropriately displaying an image of each person even when multiple people overlap in a captured image.

[0006] An information processing device according to one aspect of the present disclosure is an information processing device including a processor and a memory, wherein the processor works in cooperation with the memory to set, for a captured image of multiple people passing through a passageway, an allowed person area including images of people who are allowed to pass through the passageway and a disallowed person area including images of people who are not allowed to pass through the passageway, and when the allowed person area and the disallowed person area partially overlap, an unauthorized passerby image is generated in which at least a portion of the image included in the allowed person area is not displayed and the image included in the disallowed person area is displayed, and the unauthorized passerby image is displayed on a monitor.

[0007] A person matching system according to one aspect of the present disclosure includes a photographing device that generates a photographed image of multiple people passing through a passageway, a management device that sets, for the photographed image, an allowed person area including images of people who are permitted to pass through the passageway and an unallowed person area including images of people who are not permitted to pass through the passageway, and generates an unauthorized passerby image in which, if the allowed person area and the unallowed person area partially overlap, at least a portion of the image included in the allowed person area is not displayed and the image included in the unallowed person area is displayed, and a monitor that displays the unauthorized passerby image.

[0008] An image display method according to one aspect of the present disclosure sets, for a captured image of multiple people passing through a passageway, an allowed person area including images of people who are allowed to pass through the passageway and an unallowed person area including images of people who are not allowed to pass through the passageway, and if the allowed person area and the unallowed person area partially overlap, generates an unauthorized passerby image in which at least a portion of the image included in the allowed person area is not displayed and the image included in the unallowed person area is displayed, and displays the unauthorized passerby image on a monitor.

[0009] These comprehensive or specific aspects may be realized as a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium, or may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0010] According to the present disclosure, even when multiple people overlap in a captured image, the images of each person can be displayed appropriately.

[0011] FIG. 7 is a diagram for explaining an overview of a gate system according to the first embodiment; FIG. 8 is a diagram showing an example of display of a matching result on the first display device or the second display device according to the first embodiment; FIG. 9 is a block diagram showing an example of the configuration of a gate system according to the first embodiment; FIG. 10 is a diagram for explaining a face image and a body image according to the first embodiment; FIG. 1 is a diagram illustrating a face matching zone and a passage determination zone in a detection area where a person is detected; FIG. 2 is a diagram illustrating an example of a display pattern in a first display device according to embodiment 1; FIG. 3 is a diagram illustrating a method for detecting, from a person information table, personal information that is presumed to be the same person as the person indicated by body detection information according to embodiment 1; FIG. 4 is a diagram illustrating a method for detecting, from a person information table, personal information that is presumed to be the same person as the person indicated by body detection information according to embodiment 1;FIG. 1 is a diagram for explaining a case where an unauthorized passerby image is generated without processing; FIG. 2 is a diagram for explaining a first processing method of a unauthorized person area according to embodiment 2; FIG. 3 is a diagram for explaining a modified example of the first processing method of a unauthorized person area according to embodiment 2; FIG. 4 is a diagram for explaining a processing method when a plurality of authorized person areas overlap at least a part below a unauthorized person area according to embodiment 2;

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with appropriate reference to the drawings. However, more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter of the claims.

[0013] (First Embodiment) <Outline of Gate System> Fig. 1 is a diagram for explaining an outline of a gate system 9 according to the first embodiment. Fig. 2 is a diagram showing an example of display of a matching result on the first display device 31 or the second display device 32 according to the first embodiment.

[0014] The gate system 9 is a system that collates a facial image of a person 1 and determines whether the person 1 has the right to pass through the gate. The gate system 9 includes a first gate 11 and a second gate 12. The first gate 11 and the second gate 12 are installed a predetermined distance apart in a passage 2 through which the person 1 moves. In this embodiment, the direction from the second gate 12 toward the first gate 11 is defined as a first direction, and the direction from the first gate 11 toward the second gate 12 is defined as a second direction. In this embodiment, the area between the first gate 11 and the second gate 12 in the passage 2 is referred to as a person detection area 3. Note that, although the first embodiment describes a configuration in which the gate system 9 includes two gates, the gate system 9 may also be configured to include only one gate (e.g., the first gate 11) and to collate a person 1 heading in one direction (e.g., the first direction).

[0015] In this embodiment, for convenience of explanation, the axis along the first or second direction is referred to as the Y-axis, the axis perpendicular to the Y-axis and extending in the height direction is referred to as the Z-axis, and the axis perpendicular to the Y-axis and the Z-axis is referred to as the X-axis. The direction of the X-axis may also be referred to as the width direction of the gate.

[0016] The first gate 11 includes a first imaging device 21 capable of capturing an image of at least a portion of the detection area 3, and a first display device 31 having a screen facing the second gate 12. In FIG. 1 , the first gate 11 includes three first imaging devices 21, but the first gate 11 may include one or two first imaging devices 21, or four or more first imaging devices 21. Also, in FIG. 1 , the first gate 11 includes one first display device 31, but the first gate 11 may include two or more first display devices 31.

[0017] The second gate 12 is equipped with a second imaging device 22 capable of capturing an image of at least a portion of the detection area 3, and a second display device 32 having a screen facing the first gate 11. In Fig. 1, the second gate 12 is equipped with three second imaging devices 22, but the second gate 12 may be equipped with one or two second imaging devices 22, or may be equipped with four or more second imaging devices 22. Furthermore, in Fig. 1, the second gate 12 is equipped with one second display device 32, but the second gate 12 may be equipped with two or more second display devices 32.

[0018] In this embodiment, the first direction is the direction of entry into a facility, etc., for which the person has the right to enter or exit, and the second direction is the direction of exit from the facility, etc. However, the first direction may also be the direction of exit from the facility, etc., and the second direction may also be the direction of entry into the facility, etc.

[0019] The gate system 9 compares the facial image of the person 1 moving in the first direction captured by the first photographing device 21 at the first gate 11 with each pre-registered facial image for comparison, and displays the comparison result on the first display device 31, as illustrated in Figure 2.

[0020] The gate system 9 compares the facial image of the person 1 moving in the second direction captured by the second photographing device 22 at the second gate 12 with each pre-registered facial image for comparison, and displays the comparison result on the second display device 32, as illustrated in Figure 2.

[0021] The gate width of the first gate 11 and the second gate 12, or the width in the X direction of the detection area 3, may be wide enough for multiple people 1 to pass through side by side. As a result, the gate system 9 according to this embodiment allows more people 1 to enter and exit smoothly compared to a gate system with a gate width wide enough for only one person to pass through. For example, in a gate system with a gate width wide enough for one person, if the verification of a person 1 ahead fails, the following people 1 will be held up. However, in a gate system 9 according to this embodiment with a gate width wide enough for multiple people 1 to pass through side by side, even if the verification of a person 1 ahead fails, the following people 1 can simply move around the person 1 ahead who failed to be verified, and thus can pass through the gate smoothly without being held up. In other words, the gate system 9 according to this embodiment can prevent congestion when entering or exiting a facility, etc. The gate system 9 according to this embodiment will be described in detail below.

[0022] <Configuration of Gate System> Fig. 3 is a block diagram showing an example of the configuration of the gate system 9 according to embodiment 1. Fig. 4 is a diagram for explaining face images and body images according to embodiment 1.

[0023] The gate system 9 includes a first gate 11 and a second gate 12. The first gate 11 includes a first imaging device 21 and a first display device 31. The second gate 12 includes a second imaging device 22 and a second display device 32. The first gate 11, the first imaging device 21, the first display device 31, the second gate 12, the second imaging device 22, and the second display device 32 may be collectively referred to as a gate device 10.

[0024] The gate system 9 further includes a first edge device 41, a second edge device 42, a management device 50, and a face matching device 60. The first image capturing device 21 is connected to the first edge device 41 via an electrical cable (e.g., a USB cable or an Ethernet (registered trademark) cable). The first display device 31 is connected to the first edge device 41 via an electrical cable (e.g., an HDMI (registered trademark, the same applies hereinafter) cable). The second image capturing device 22 is connected to the second edge device 42 via an electrical cable (e.g., a USB cable or an Ethernet cable). The second display device 32 is connected to the second edge device 42 via an electrical cable (e.g., an HDMI cable). The first edge device 41 and the second edge device 42 are connected to the management device 50 via a communication network 4. The management device 50 is connected to the face matching device 60 via the communication network 4. Examples of the communication network 4 include a wired LAN, a wireless LAN, a mobile communication network (e.g., LTE, 4G, 5G), the Internet, and / or a VPN.

[0025] The first imaging device 21 captures an image of the detection area 3 to generate a captured image, and transmits the captured image to the first edge device 41. The second imaging device 22 captures an image of the detection area 3 to generate a captured image, and transmits the captured image to the second edge device 42. The captured image may be a still image or a moving image.

[0026] As shown in (a) and (b) of FIG. 4, the first edge device 41 detects the entire body of the person 1 from the photographed image received from the first image capturing device 21. In this embodiment, the image of the entire body of the person 1 detected from the photographed image is referred to as a body image 200. The first edge device 41 transmits the detected body image 200 to the management device 50. Furthermore, as shown in (a) of FIG. 4, the first edge device 41 detects the face portion of the person 1 from the photographed image received from the first image capturing device 21. In this embodiment, the image of the face portion of the person 1 detected from the photographed image is referred to as a face image 220. The first edge device 41 transmits the detected face image 220 to the management device 50.

[0027] As shown in (a) and (b) of Figures 4, the second edge device 42 detects the entire body of the person 1 from the captured image received from the second image capturing device 22. The second edge device 42 transmits a body image 200 detected from the captured image to the management device 50. As shown in (a) of Figure 4, the second edge device 42 detects the face portion of the person 1 from the captured image received from the second image capturing device 22. The second edge device 42 transmits a face image 220 detected from the captured image to the management device 50. Details of the processes performed by the first edge device 41 and the second edge device 42 will be described later.

[0028] The management device 50 generates person information 100 (see FIG. 6 ) corresponding to the person 1 located in the detection area 3 based on the body image 200 or face image 220 received from the first edge device 41, or the body image 200 or face image 220 received from the second edge device 42. The management device 50 may generate person information 100 for each person 1 detected in the detection area 3. Details of the person information 100 will be described later (see FIG. 6 ). The management device 50 transmits a face matching request including the face image 220 to the face matching device 60.

[0029] The face matching device 60 stores in advance a face image for matching of a person 1 who has the right to pass through a gate (for example, to enter or exit). For example, the face matching device 60 stores a face image for matching of a person 1 who has already registered to enter the facility. The face matching device 60 may manage the face image for matching by linking it to the name, age, sex, credit card information, etc. of the person 1.

[0030] Furthermore, the face matching device 60 matches the face image 220 received from the management device 50 with each of the face images for matching. If the face matching device 60 is successful in matching with any of the registered face images for matching, it generates face matching result information including a successful matching result and transmits it to the management device 50. At this time, the face matching device 60 may perform billing in real time using credit card information linked to the successfully matched face image for matching. If the face matching device 60 fails to match with any of the registered face images for matching, it generates face matching result information including a failed matching result and transmits it to the management device 50. Note that the matching between the face image 220 and the face images for matching may be achieved by a known face image matching technology. Also, "matching" may be read as "authentication."

[0031] The management device 50 receives face matching result information from the face matching device 60. The management device 50 transmits person matching information including face matching result information for the face image 220 of the person 1 moving in the first direction to the first edge device 41, and transmits person matching information including face matching result information for the face image 220 of the person 1 moving in the second direction to the second edge device 42.

[0032] When the first edge device 41 receives person matching information from the management device 50, it displays the face matching result information included in the person matching information on the first display device 31, as shown in Fig. 2. This allows a person 1 located in the detection area 3 and moving in the first direction to view the screen of the first display device 31 located in the direction of movement and know his / her own matching result. In other words, the first edge device 41 allows a person 1 moving in the first direction to visually recognize the matching result of the person 1.

[0033] When the second edge device 42 receives person matching information from the management device 50, it displays the face matching result information included in the person matching information on the second display device 32, as shown in FIG. 2. This allows person 1 located in the detection area 3 and moving in the second direction to view the screen of the second display device 32 located at the end of the moving direction and know his / her matching result. In other words, the second edge device 42 allows person 1 moving in the second direction to visually confirm the matching result for that person 1. Below, the processes performed by the first edge device 41, the second edge device 42, and the management device 50 will be described in detail.

[0034] <Detection Process in First Edge Device> Fig. 5 is a flowchart showing an example of the detection process performed by the first edge device 41 according to embodiment 1. Note that, for the detection process performed by the second edge device 42 according to embodiment 1, in the following description with reference to Fig. 5, the "first image capturing device 21" can be read as the "second image capturing device 22."

[0035] The first edge device 41 receives a captured image from the first image capturing device 21 (S101).

[0036] The first edge device 41 receives a distance image from the first image capturing device 21 (S102). In this way, the first image capturing device 21 may generate a distance image indicating the distance to the person 1 or the like present in the detection area 3. The same applies to the second image capturing device 22. The distance image may be generated using any known technology, such as a stereo camera or a distance measuring sensor (e.g., LiDAR (Light Detection And Ranging)). Note that, in this embodiment, the processing of step S102 is performed after the processing of step S101, but this is not limiting and the procedures may be reversed.

[0037] As shown in FIG. 4A, the first edge device 41 detects a facial image 220 of the person 1 from the captured image (S103). The detection of the facial image 220 of the person may be realized using any known technology, such as pattern matching or AI (Artificial Intelligence). The AI ​​may also be an image recognition technology using deep learning (e.g., CNN (Convolutional Neural Network)).

[0038] The first edge device 41 determines whether or not the face image 220 was detected from the captured image in step S103 (S104). If the face image 220 was not detected (S104: NO), the first edge device 41 proceeds to step S108.

[0039] If the face image 220 is detected (S104: YES), the first edge device 41 uses the distance image to calculate three-dimensional coordinates (hereinafter referred to as face coordinates 221 (see FIG. 4)) of the detected face image 220 (S105). For example, as shown in FIG. 4A, the first edge device 41 calculates the center point of the face image 220 as the face coordinates 221.

[0040] The first edge device 41 generates face detection information (S106), for example, including an ID (hereinafter referred to as a photographing device ID) for identifying the first photographing device 21 that is the sender of the photographed image, the face image 220, the size of the face image 220 (hereinafter referred to as a face image size), face coordinates 221, and the photographing time of the photographed image.

[0041] The first edge device 41 transmits the generated face detection information to the management device 50 (S107), and the process proceeds to the next step S108.

[0042] 4A or 4B, the first edge device 41 detects the body image 200 of the person 1 from the captured image (S108). The detection of the body image 200 of the person 1 may be realized using any known technology, such as pattern matching or AI. The AI ​​may also be an image recognition technology using deep learning (e.g., CNN).

[0043] The first edge device 41 determines whether or not the body image 200 was detected from the captured image in step S108 (S109). If the body image 200 was not detected (S109: NO), the first edge device 41 returns the process to step S101.

[0044] If the body image 200 is detected (S109: YES), the first edge device 41 uses the distance image to calculate three-dimensional coordinates (hereinafter referred to as body coordinates 201 (see FIG. 4)) of the detected body image 200 (S110). For example, the first edge device 41 calculates the center point of the body image 200 as the body coordinates 201, as shown in FIGS. 4A and 4B.

[0045] The first edge device 41 generates body detection information (S111). For example, the first edge device 41 generates body detection information including an image capturing device ID, a body ID for identifying the body image 200, the body image 200, body coordinates 201, and the image capturing time of the captured image.

[0046] The first edge device 41 transmits the generated body detection information to the management device 50 (S112), and the process returns to step S101. Note that in the present embodiment, the process of detecting a face image from the captured image (S103 to S107) is performed before the process of detecting a body image from the captured image (S108 to S112), but this is not limiting and the steps may be reversed.

[0047] In this way, by performing the process of detecting the facial image 220 and / or the body image 200 of the person 1 from the captured image and the process of calculating the facial coordinates 221 and / or the body coordinates 201 at the first edge device 41 or the second edge device 42, the processing load can be distributed compared to when these processes are performed at the management device 50. Note that if distribution of the processing load is not a consideration, the gate system 9 does not need to include the first edge device 41 and the second edge device 42. In this case, the first image capturing device 21, the first display device 31, the second image capturing device 22, and the second display device 32 may be directly connected to the management device 50. Then, the management device 50 may perform the process of detecting the facial image 220 and / or the body image 200 of the person 1 from the captured image and the process of calculating the facial coordinates 221 and / or the body coordinates 201.

[0048] <Personal Information> FIG. 6 shows an example of the configuration of personal information 100 according to the first embodiment.

[0049] The person information 100 is generated for each person 1 detected in the detection area 3. The person information 100 includes, as information or data, a tracking ID 101, a first image capture time 102, an image update time 103, a body image 104, a body image update time 105, body coordinates 106, a face image 107, a face image update time 108, face coordinates 109, a face matching ID 110, a matching status 111, and a movement direction 112.

[0050] The tracking ID 101 is set to an ID used to track the same person in the detection area 3. The tracking ID 101 can also be said to be an ID for identifying the person information 100.

[0051] The first image capturing time 102 is set to the time when the body image 104 or face image 107 included in the personal information 100 was first captured. The first image capturing time can also be said to be the time when the personal information 100 was first generated.

[0052] The image update time 103 is set to the time when the body image 104 or the face image 107 included in the personal information 100 was last updated. The image update time 103 can also be said to be the time when the personal information 100 was last updated.

[0053] The body image 104 is set to a body image 200 detected from a photographed image of the person 1 indicated by the personal information 100. The body image 104 may be updated as needed to the body image 200 detected from the latest photographed image.

[0054] The body image update time 105 is set to the time when the body image 104 included in the personal information 100 was last updated.

[0055] The body coordinates 106 are set to the body coordinates 201 calculated based on the body image 104. The body coordinates 106 may be expressed as three-dimensional coordinates (X coordinate, Y coordinate, Z coordinate) in the detection area 3.

[0056] The face image 107 is set to a face image 220 detected from a photographed image of the person 1 indicated by the personal information 100. The face image 107 may be updated as needed to the face image 220 detected from the latest photographed image.

[0057] The face image update time 108 is set to the time when the face image 107 included in the person information 100 was last updated.

[0058] The face coordinates 221 calculated based on the face image 107 are set as the face coordinates 109. The face coordinates 109 may be expressed as three-dimensional coordinates (X coordinate, Y coordinate, Z coordinate) in the detection area 3.

[0059] An ID assigned to a successfully matched face image for matching is set in face matching ID 110. In other words, face matching ID 110 can also be said to be an ID that certifies person 1 corresponding to face image 107. If no successfully matched face image for matching exists (i.e., if face matching fails), NULL may be set in face matching ID 110.

[0060] The matching status 111 is set to the matching status for the face image 107. For example, the matching status 111 is set to one of "before matching request", "matching in progress", "matching successful", and "matching failed".

[0061] The movement direction 112 is set to the movement direction of the person 1 indicated by the person information 100. For example, the movement direction is set to either the first direction or the second direction.

[0062] The management device 50 manages a plurality of pieces of personal information 100 as a personal information table (not shown). The personal information table may be stored in a memory 1002 or a storage 1003 (see FIG. 29 ) provided in the management device 50, or may be stored in a device different from the management device 50.

[0063] <Overall Processing in Management Device> FIG. 7 is a flowchart showing an example of overall processing performed by the management device 50 according to the first embodiment.

[0064] The management device 50 determines whether or not face detection information (see step S107 in FIG. 5) has been received from the first edge device 41 or the second edge device 42 (S201).

[0065] When the management device 50 receives face detection information (S201: YES), it executes face detection information processing (S202). Note that the face detection information processing will be described in detail later (see FIG. 8).

[0066] If the management device 50 has not received face detection information (S202: NO), it determines whether it has received body detection information (see step S112 in Figure 5) from the first edge device 41 or the second edge device 42 (S203).

[0067] When the management device 50 receives the body detection information (S203: YES), the management device 50 executes body detection information processing (S204). Details of the body detection information processing will be described later (see FIG. 9).

[0068] If the management device 50 has not received the body detection information (S203: NO), it determines whether or not it has received face matching result information (S205). The face matching result information is information returned from the face matching device 60 in response to a face matching request transmitted from the management device 50 to the face matching device 60 in Fig. 8, which will be described later, and includes the result of face matching (for example, matching success or failure).

[0069] When the management device 50 receives the face matching result information (S205: YES), it executes face matching result information processing (S206). Note that the face matching result information processing will be described in detail later (see FIG. 11).

[0070] If the management device 50 has not received the face matching result information (S205: NO), the process returns to step S201.

[0071] After executing the face detection information processing (S202) or the body detection information processing (S204), the management device 50 executes post-movement processing (S210) and returns the processing to step S201. Details of the post-movement processing will be described later (see FIG. 10 ). Note that in this embodiment, the processing of steps S201, S203, and S205 is performed in stages, but this is not limiting, and the processing of steps S201, S203, and S205 may be interchanged or may be performed simultaneously in parallel.

[0072] <Face Detection Information Processing in Management Apparatus> FIG. 8 is a flowchart showing an example of the face detection information processing (S202) shown in FIG.

[0073] The management device 50 detects, from the person information table, the person information 100 corresponding to the face detection information received from the first edge device 41 or the second edge device 42 (S301). In this embodiment, the person information 100 corresponding to the face detection information is the person information 100 that is presumed to be the same person as person 1 indicated by the face detection information. Note that the method for detecting, from the person information table, the person information 100 that is presumed to be the same person as person 1 indicated by the face detection information will be described in detail later (see FIGS. 15A and 15B ).

[0074] The management device 50 determines whether or not the personal information 100 has been detected from the personal information table in step S301 (S302).

[0075] Next, a case where it is determined in step S302 that the personal information 100 cannot be detected from the personal information table (S302: NO) will be described.

[0076] The management device 50 generates new personal information 100 based on the face detection information (S303). At this time, the management device 50 sets the following contents in the new personal information 100: - A new tracking ID is set as the tracking ID 101 of the personal information 100. - The face image 220 included in the face detection information is set as the face image 107 of the personal information 100. - The face coordinates 221 included in the face detection information are set as the face coordinates 109 of the personal information 100. - The first image capture time 102 of the personal information 100 is set to the capture time included in the earliest face detection information (i.e., the earliest acquired capture time). - The matching status 111 of the personal information 100 is set to "before matching request." The management device 50 detects the movement direction of the person 1 corresponding to the face detection information, and sets the detected movement direction as the movement direction 112 of the personal information 100 generated in step S303 (S304). Note that the method for detecting the movement direction of a person will be described in detail later (see FIG. 16 ).

[0077] The management device 50 newly registers the personal information 100 generated and set in steps S303 and S304 in the personal information table (S305).

[0078] The management device 50 transmits a face matching request including the face image 107 and the tracking ID 101 to the face matching device 60, and updates the matching status 111 of the person information 100 to "matching in progress" (S306).

[0079] Then, the management device 50 completes this process.

[0080] Next, a case where it is determined in step S302 that the personal information 100 has been detected from the personal information table (S302: YES) will be described.

[0081] The management device 50 updates the content of the person information 100 detected in step S301 based on the face detection information (S310). For example, the management device 50 updates the detected person information 100 as follows: Updates the face coordinates 109 of the person information 100 to the face coordinates 221 included in the face detection information Updates the image update time 103 and face image update time 108 of the person information 100 to the shooting time included in the face detection information The management device 50 determines whether the matching status 111 of the detected person information 100 is "before matching request" or "matching failed" (S311). If the matching status 111 is not "before matching request" or "matching failed" (S311: NO), the management device 50 completes this process.

[0082] If the matching status 111 is "before matching request" or "matching failed" (S311: YES), the management device 50 transmits a face matching request including the face image 107 and the tracking ID 101 to the face matching device 60, and updates the matching status 111 of the person information 100 to "matching in progress" (S312). Then, the management device 50 completes this process.

[0083] <Body Detection Information Processing in Control Device> FIG. 9 is a flowchart showing an example of the body detection information processing (S204) shown in FIG.

[0084] The management device 50 detects, from the person information table, personal information 100 corresponding to the body detection information received from the first edge device 41 or the second edge device 42 (S401). In this embodiment, the personal information 100 corresponding to the body detection information is personal information 100 that is presumed to be the same person as person 1 indicated by the body detection information. Details of a method for detecting, from the person information table, personal information 100 that is presumed to be the same person as person 1 indicated by the body detection information will be described later (see FIGS. 15A and 15B ).

[0085] The management device 50 determines whether or not the personal information 100 has been detected from the personal information table in step S401 (S402).

[0086] Next, a case where it is determined in step S402 that the personal information 100 cannot be detected from the personal information table (S402: NO) will be described.

[0087] The management device 50 generates new personal information 100 based on the received body detection information (S403). At this time, the management device 50 sets the following contents in the new personal information 100: - A new tracking ID is set as the tracking ID 101 of the personal information 100; - The body image 200 included in the body detection information is set as the body image 104 of the personal information 100; - The body coordinates 202 included in the body detection information are set as the body coordinates 106 of the personal information 100; - The first image capture time 102 of the personal information 100 is set to the capture time included in the earliest body detection information (i.e., the earliest acquired capture time). The management device 50 detects the movement direction of the person 1 corresponding to the body detection information and sets the detected movement direction as the movement direction 112 of the personal information 100 generated in step S403 (S404). Note that the method for detecting the movement direction of the person 1 will be described in detail later (see FIG. 16 ).

[0088] The management device 50 newly registers the personal information 100 generated and set in steps S403 and S404 in the personal information table (S405), and the management device 50 then completes this process.

[0089] Next, a case where it is determined in step S402 that the personal information 100 has been detected from the personal information table (S402: YES) will be described.

[0090] The management device 50 updates the content of the person information 100 detected in step S401 based on the body detection information (S410). For example, the management device 50 updates the detected person information 100 as follows: Updates the body coordinates 106 of the person information 100 to the body coordinates 202 included in the body detection information Updates the image update time 103 and body image update time 105 of the person information 100 to the body image capture time included in the body detection information Then, the management device 50 completes this process.

[0091] <Post-Movement Processing in Management Apparatus> FIG. 10 is a flowchart showing an example of the post-move processing (S210) shown in FIG.

[0092] The management device 50 generates person movement information based on the target person information (S501). In this embodiment, the target person information is the person information 100 detected from the person information table or the newly generated person information 100 in the face detection information processing shown in FIG. 8, or the person information 100 detected from the person information table or the newly generated person information 100 in the body detection information processing shown in FIG. 9. At this time, the management device 50 sets the following information in the person movement information: Tracking ID Zone information Face image (or, if there is no face image, a body image) Matching status Face matching ID Here, the zone information is information indicating whether the person 1 corresponding to the target person information is located in the face matching zone 301 or the passage determination zone 302 in the detection area 3. The face matching zone 301 and the passage determination zone 302 will be described in detail later (see FIG. 13).

[0093] The management device 50 determines whether the movement direction of the target person information is the first direction or the second direction (S502).

[0094] If the movement direction is the first direction (S502: first direction), the management device 50 transmits the person movement information generated in step S501 to the first edge device 41 (S503), and completes this process.

[0095] If the movement direction is the second direction (S502: second direction), the management device 50 transmits the person movement information generated in step S501 to the second edge device 42 (S504), and completes this process.

[0096] <Face Matching Result Information Processing in Control Device> FIG. 11 is a flowchart showing an example of the face matching result information processing (S206) shown in FIG.

[0097] The management device 50 sets the content of the face matching result information received from the face matching device 60 to the person information 100 in the person information table that corresponds to the face matching result information (S601). For example, the management device 50 sets the content of the face matching result information to the person information 100 as follows: - Sets the face matching ID of the face matching result information to the face matching ID 110 of the person information 100 - Sets the matching result (matching success or matching failure) of the face matching result information to the matching status 111 of the person information 100 The management device 50 generates person matching information based on the person information 100 set in step S601 (S602). At this time, the management device 50 sets the following content to the person matching information: - Tracking ID - Zone information - Face image - Matching status - Face matching ID The management device 50 determines whether the movement direction 112 of the person information 100 set in step S601 is the first direction or the second direction (S603).

[0098] If the movement direction 112 is the first direction (S603: first direction), the management device 50 transmits the person matching information generated in step S602 to the first edge device 41 (S604), and completes this process.

[0099] If the movement direction 112 is the second direction (S603: second direction), the management device 50 transmits the person matching information generated in step S602 to the second edge device 42 (S605), and completes this process.

[0100] <Display Processing in First Edge Device> Fig. 12 is a flowchart showing an example of display processing performed by the first edge device 41 according to embodiment 1. Note that, for the display processing performed by the second edge device 42 according to embodiment 1, in the following description with reference to Fig. 12, "first edge device 41" may be read as "second edge device 42," "first display device 31" may be read as "second display device 32," and "first direction" may be read as "second direction."

[0101] The first edge device 41 determines whether or not it has received person movement information or person matching information from the management device 50 (S701). If the first edge device 41 has not received either person movement information or person matching information (S701: NO), it returns the process to step S701. If the first edge device 41 has received either person movement information or person matching information (S701: YES), it performs the following process.

[0102] The first edge device 41 determines whether a face image or a body image corresponding to the tracking ID included in the received person movement information or person matching information is displayed on the first display device 31 (S702).

[0103] If the face image or body image corresponding to the tracking ID is not displayed on the first display device 31 (S702: NO), the first edge device 41 newly displays the face image or body image and the matching status included in the person movement information or person matching information on the first display device 31 (S703). Then, the first edge device 41 returns the process to step S701.

[0104] If a face image or body image corresponding to the tracking ID is displayed on the first display device 31 (S702: YES), the first edge device 41 updates the displayed face image or body image and matching status to the face image or body image and matching status included in the person movement information or person matching information and displays them (S704).Then, the first edge device 41 returns the process to step S701.

[0105] 2, a matching result image showing the face image or body image of each person 1 moving in the first direction and the matching status (e.g., matching success or failure, etc.) is displayed on the first display device 31. Therefore, each person 1 moving in the first direction can visually confirm their own matching result from the first display device 31 installed at the destination in the direction of movement.

[0106] 2, the second display device 32 displays a matching result image showing the face image or body image of each person 1 moving in the second direction and the matching status (e.g., matching success or failure, etc.) Therefore, each person 1 moving in the second direction can visually confirm their own matching result from the second display device 32 installed at the destination in the direction of movement.

[0107] The details of the above-mentioned display process can be expressed as follows: That is, the management device 50 detects the movement direction of each person 1 moving between the first gate 11 and the second gate 12, and displays the matching result of the facial image 220 of the person 1 moving in the first direction on the first display device 31 but not on the second display device 32, and displays the matching result of the facial image 220 of the person 1 moving in the second direction on the second display device 32 but not on the first display device 31.

[0108] As a result, the matching result for person 1 is displayed on the display device installed in the direction of movement of person 1, and is not displayed on the display device installed opposite the direction of movement of person 1. Therefore, the management device 50 can appropriately show the matching result to each of person 1 moving in the first direction along the passage 2 and person 1 moving in the second direction.

[0109] 13 is a diagram showing face matching zone 301 and passage determination zone 302 in detection area 3 according to embodiment 1. Note that, although the following description will be given using first gate 11 and first direction as an example, the same applies to second gate 12 and second direction.

[0110] The image captured by the first photographing device 21 at the first gate 11 is used for face matching of the person 1 moving in the first direction. In the detection area 3, a predetermined area close to the second gate 12 is set as a face matching zone 301, and a predetermined area close to the first gate 11 is set as a passage determination zone 302. The first edge device 41 may change the display pattern on the first display device 31 depending on whether the person 1 is located in the face matching zone 301 or the passage determination zone 302. Next, the display pattern will be described.

[0111] 14 is a diagram for explaining an example of a display pattern in the first display device 31 according to embodiment 1. Note that, with regard to the display pattern in the second display device 32, in the following description with reference to FIG. 14, the "first edge device 41" may be read as the "second edge device 42," and the "first display device 31" may be read as the "second display device 32."

[0112] If the person 1 is located outside the detection area 3 (outside the zone), the first edge device 41 does not display the matching result image of the person 1 on the first display device 31 .

[0113] When person 1 is located in face matching zone 301, first edge device 41 controls the display of first display device 31 according to the following patterns (A0) to (A3).

[0114] (A0) When a facial image 220 of person 1 exists and the matching status 111 is “matching in progress”, the first edge device 41 displays a matching result image including the facial image 220 of person 1 and the message “matching in progress” on the first display device 31.

[0115] (A1) When the facial image 220 of person 1 exists and the matching status 111 is “matching successful,” the first edge device 41 displays a matching result image including the facial image 220 of person 1 and a message “matching successful” on the first display device 31. At this time, the first edge device 41 may output a matching success sound from a speaker provided in the first display device 31.

[0116] (A2) When the facial image 220 of person 1 exists and the matching status 111 is “matching failed,” the first edge device 41 displays a matching result image including the facial image 220 of person 1 and a message stating “matching failed” on the first display device 31. At this time, the first edge device 41 may output a matching failure sound from a speaker provided in the first display device 31.

[0117] (A3) When the face image 220 of person 1 does not exist but the body image 200 exists, the first edge device 41 displays a matching result image including the body image 200 of person 1 and a “warning” message on the first display device 31. At this time, the first edge device 41 may output a warning sound from a speaker provided in the first display device 31.

[0118] When person 1 is located in passage determination zone 302, first edge device 41 controls the display of first display device 31 according to the following patterns (A4) to (A7).

[0119] (A4) When a facial image 220 of person 1 exists and the matching status 111 is “matching in progress,” the first edge device 41 displays a matching result image including the facial image 220 of person 1 and the message “matching in progress” on the first display device 31.

[0120] (A5) When the facial image 220 of person 1 exists and the matching status 111 is “matching successful,” the first edge device 41 displays a matching result image including the facial image 220 of person 1 and a message “passed” on the first display device 31. At this time, the first edge device 41 may output a passing sound from a speaker provided in the first display device 31.

[0121] (A6) When the face image 220 of person 1 exists and the matching status 111 is “matching failed,” the first edge device 41 displays a matching result image including the face image 220 of person 1 and a message saying “unauthorized passage” on the first display device 31. At this time, the first edge device 41 may output an unauthorized passage sound from a speaker provided in the first display device 31.

[0122] (A7) When the face image 220 of person 1 does not exist but the body image 200 exists, the first edge device 41 displays a matching result image including the body image 200 of person 1 and a message saying “illegal passage” on the first display device 31. At this time, the first edge device 41 may output an illegal passage sound from a speaker provided in the first display device 31.

[0123] The display priority may be set as follows: The passage determination zone 302 has a higher display priority than the face matching zone 301. The "passing" message has the highest display priority, followed by the "matching successful" message. If a face image 220 is present, the "matching in progress" message has the lowest display priority. The presence of a face image 220 has a higher display priority than the presence of only a body image 200.

[0124] For example, the display priorities of the above-described patterns (A0) to (A7) may be set as follows, from lowest to highest: (A3) < (A7) < (A0) < (A4) < (A2) < (A6) < (A1) < (A5) <Method for Detecting Personal Information of the Same Person> FIGS. 15A and 15B are diagrams for explaining a method for detecting, from the personal information table, personal information 100 that is estimated to belong to the same person as person 1 indicated by the body detection information in embodiment 1. Note that in FIGS. 15A and 15B , the description will be given assuming that person 1 is moving in a first direction. In the case where person 1 is moving in a second direction, in the following description referring to FIGS. 15A and 15B , the "first camera device 21" should be read as the "second camera device 22," and the "second camera device 22" should be read as the "first camera device 21."

[0125] A body image 200 of person 1 moving in a first direction captured by the first camera 21 shows the front view of the body of person 1. Hereinafter, the X and Y coordinates of body coordinates 201A calculated from the front view of body image 200 will be referred to as the X coordinate 201Ax and Y coordinate 201Ay of the front of the body, respectively. Furthermore, a face image 220 of person 1 moving in the first direction captured by the first camera 21 shows the front view of the face of person 1. Hereinafter, the X and Y coordinates of face coordinates 221A calculated from the front view of face image 220 will be referred to as the X coordinate 221Ax and Y coordinate 221Ay of the front of the face, respectively.

[0126] The body image 200 of the person 1 moving in the first direction captured by the second image capturing device 22 shows the back of the body of the person 1. Hereinafter, the X coordinate and the Y coordinate of the body coordinate 201B calculated from the back body image 200 will be referred to as the X coordinate 201Bx and the Y coordinate 201By of the back of the body, respectively.

[0127] As shown in FIG. 15A , the Y coordinate 201Ay of the front of the body and the Y coordinate 201By of the back of the body are separated by the thickness of the person 1 in the front-to-back direction. Therefore, if the positional difference between the Y coordinate 201Ay of the front of the body and the Y coordinate 201By of the back of the body is smaller than a predetermined threshold W based on the body thickness (hereinafter referred to as the body thickness threshold), it can be assumed that the Y coordinate 201Ay of the front of the body and the Y coordinate 201By of the back of the body correspond to the Y coordinates of the same person. The body thickness threshold W may be a value slightly larger than the average body thickness. Similarly, if the positional difference between the Y coordinate 221Ay of the front of the face and the Y coordinate 201By of the back of the body is smaller than the body thickness threshold W, it can be assumed that the Y coordinate 221Ay of the front of the face and the Y coordinate 201By of the back of the body correspond to the Y coordinates of the same person.

[0128] 15B , if the amount of change in the X component between X-coordinate 201Ax of body coordinate 201A included in the received body detection information (i.e., the current X-coordinate) and X-coordinate 201Ax of body coordinate 201A included in personal information 100 (i.e., the past X-coordinate) is equal to or less than a predetermined threshold value (hereinafter referred to as the body width threshold value) W2, it can be estimated that X-coordinate 201Ax of the current body coordinate and X-coordinate 201Ax of the past body coordinate are the X-coordinates of the same person. Note that face coordinate 221A may be used instead of body coordinate 201A.

[0129] In other words, if the management device 50 can detect from the personal information table personal information 100 such that the positional difference between the X coordinate of the body coordinate 201 (or face coordinate 221) included in the received body detection information and the X coordinate of the body coordinate 201 (or face coordinate 221) included in the personal information 100 is less than the body width threshold W2, and the positional difference between the Y coordinate of the body coordinate 201 included in the received body detection information and the Y coordinate of the body coordinate 201 or the Y coordinate of the face coordinate 221 included in the personal information 100 is less than the body thickness threshold W, the management device 50 may estimate that the detected personal information 100 is the same person as the person 1 indicated by the body detection information.

[0130] Similarly, if the management device 50 can detect from the personal information table personal information 100 in which the positional difference between the X coordinate of the body coordinate 201 (or face coordinate 221) included in the received body detection information and the X coordinate of the body coordinate 201 (or face coordinate 221) included in the personal information 100 is smaller than the body width threshold W2, and the positional difference between the Y coordinate of the face coordinate 221 included in the received face detection information and the Y coordinate of the body coordinate 201 or the Y coordinate of the face coordinate 221 included in the personal information 100 is smaller than the body thickness threshold W, the management device 50 may estimate that the detected personal information 100 is the same person as the person 1 indicated by the face detection information.

[0131] The above-described conditions regarding the X-coordinate positional difference and the body width threshold W2, and the Y-coordinate positional difference and the body thickness threshold W are met when the time difference between the face photographing time included in the face detection information or the body photographing time included in the body detection information and the face image update time 108 or the body image update time 105 included in the person information 100 is sufficiently small (i.e., when the time difference is smaller than a predetermined time threshold T). Because person 1 is moving, if the time difference is large, person information 100 that satisfies the above-described conditions regarding the X-coordinate positional difference and the body width threshold W2, and the Y-coordinate positional difference and the body thickness threshold W is likely to belong to a different person who was in that position in the past.

[0132] 8 may perform the following process: The management device 50 detects, from the personal information table, personal information 100 for which the time difference between the face capture time included in the face detection information and the face image update time 108 or the body image update time 105 included in the personal information 100 is less than a predetermined time threshold T, the positional difference between the X coordinate of the face coordinate 221 included in the face detection information and the X coordinate of the face coordinate 109 included in the personal information 100 is smaller than a body width threshold W2, and the positional difference between the Y coordinate of the face coordinate 221 included in the face detection information and the Y coordinate of the face coordinate 109 or the Y coordinate of the body coordinate 106 included in the personal information 100 is smaller than a body thickness threshold W.

[0133] 9 may perform the following process: The management device 50 detects, from the personal information table, personal information 100 in which the time difference between the body image capture time included in the body detection information and the face image update time 108 or the body image update time 105 included in the personal information 100 is less than a predetermined time threshold T, the positional difference between the X coordinate of the body coordinate 201 included in the body detection information and the X coordinate of the body coordinate 106 included in the personal information 100 is smaller than a body width threshold W2, and the positional difference between the X coordinate of the body coordinate 201 included in the body detection information and the Y coordinate of the face coordinate 109 or the Y coordinate of the body coordinate 106 included in the personal information 100 is smaller than a body thickness threshold W.

[0134] The above content can be expressed as follows: That is, if the difference between the position (e.g., body coordinates 201 or face coordinates 221) of person 1 included in a first image (e.g., body image 200 or face image 220) captured by first image capture device 21 and the position (e.g., body coordinates 201 or face coordinates 221) of person 1 included in a second image (e.g., body image 200 or face image 220) captured by second image capture device 22 is smaller than a predetermined threshold W based on the thickness of the person's body, management device 50 estimates that the position of person 1 included in the first image and the position of person 1 included in the second image represent the positions of the same person.

[0135] This allows the management device 50 to estimate, for example, whether a first image capturing the front of person 1 and a second image capturing the back of person 1 are of the same person.

[0136] <Method for Detecting Person's Moving Direction> FIG. 16 is a diagram for explaining a method for detecting the moving direction of person 1 according to the first embodiment.

[0137] The management device 50 detects the moving direction of the person 1 by, for example, the following first, second, or third method.

[0138] <<First Method>> The management device 50 detects the movement direction 112 of the person 1 based on the movement direction over time of the face coordinates 109 or the body coordinates 106 included in the person information 100. For example, when the management device 50 detects that the Y coordinate of the face coordinates 109 or the Y coordinate of the body coordinates 106 included in the person information 100 is moving in the negative direction of the Y axis over time, the management device 50 detects the movement direction 112 of the person 1 as a first direction. When the management device 50 detects that the Y coordinate of the face coordinates 109 or the Y coordinate of the body coordinates 106 included in the person information 100 is moving in the positive direction of the Y axis over time, the management device 50 detects the movement direction 112 of the person 1 as a second direction.

[0139] <<Second Method>> The management device 50 determines whether the image of the person 1 detected for the first time was captured by the first camera 21 or the second camera 22, based on the camera device ID. If the image of the person 1 detected for the first time was captured by the first camera 21, the management device 50 detects the movement direction 112 of the person 1 as the first direction. This is because the person 1 entering the detection area 3 from the second gate 12 is first photographed by the first camera 21 provided at the first gate 11. If the image of the person 1 detected for the first time was captured by the second camera 22, the management device 50 detects the movement direction 112 of the person 1 as the second direction. This is because the person 1 entering the detection area 3 from the first gate 11 is first photographed by the second camera 22 provided at the second gate 12.

[0140] 16 , when the face coordinates 109 or body coordinates 106 of a person 1 detected for the first time are included in a predetermined area 312 of the detection area 3 that is closer to the second gate 12, the management device 50 detects the movement direction 112 of the person 1 as a first direction. This is because the person 1 entering the detection area 3 from the second gate 12 first enters the predetermined area 312 closer to the second gate 12. When the face coordinates 109 or body coordinates 106 of a person 1 detected for the first time are included in a predetermined area 311 of the detection area 3 that is closer to the first gate 11, the management device 50 detects the movement direction 112 of the person 1 as a second direction. This is because the person 1 entering the detection area 3 from the first gate 11 first enters the predetermined area 311 closer to the first gate 11.

[0141] (Embodiment 2) Fig. 17 is a diagram for explaining an overview of a gate system 9 according to embodiment 2. Fig. 18 is a block diagram of the gate system 9 according to embodiment 2. Fig. 19 is a diagram showing an example of a display of an unauthorized passerby image 500 on a surveillance monitor 70 according to embodiment 2.

[0142] As shown in Fig. 17 , the gate system 9 according to the second embodiment includes a surveillance monitor 70 in addition to the devices shown in Fig. 3 . In the second embodiment, in addition to the processing described in the first embodiment, the management device 50 performs processing to display an image (hereinafter referred to as an unauthorized pedestrian image 500) of a person who does not have the right to pass through the gate device 10 (hereinafter referred to as an unauthorized pedestrian) on the surveillance monitor 70. For example, the surveillance monitor 70 is installed near the exit of the gate device 10, and an observer 80 monitors the surveillance monitor 70 and people passing through the gate device 10 near the exit of the gate device 10. In this way, when the unauthorized pedestrian image 500 is displayed on the surveillance monitor 70, the observer 80 can quickly find the unauthorized pedestrian by comparing the unauthorized pedestrian image 500 with the person passing through the gate device 10.

[0143] The second embodiment will be described in detail below. The methods for matching and tracking a person's face may be the same as those described in the first embodiment. The second embodiment will describe a case in which the first gate 11 in the first embodiment is an exit gate, the second gate 12 is an entrance gate, and a person moving in a first direction is judged as having illegal passage. However, the contents of the second embodiment also apply to a case in which the first gate 11 in the first embodiment is an entrance gate, the second gate 12 is an exit gate, and a person moving in a second direction is judged as having illegal passage. In the following description, a person whose face has been successfully matched may be referred to as an authorized person 401, and a person whose face has not been matched may be referred to as an unauthorized person 402.

[0144] <Flowchart> FIG. 20 is a flowchart showing an example of processing performed by the management device 50 according to the second embodiment.

[0145] The management device 50 detects and tracks a person from an image captured by the first image capturing device 21, and stores the image in the memory 1002 or the storage 1003 (see FIG. 29 ) (S1001). The memory 1002 or the storage 1003 stores a plurality of captured images in chronological order.

[0146] The management device 50 performs face matching on the detected face image of the person (S1002). The face matching may be performed by the method described in the first embodiment.

[0147] The management device 50 sets a permitted person area 411 for an area in the captured image that shows a person whose face has been successfully matched (a permitted person), and sets a non-permitted person area 412 for an area in the captured image that shows a person whose face has not been matched (a non-permitted person) (S1003). At this time, the management device 50 may set the area so as to include as much of the person's entire body as possible. Furthermore, the shapes of the permitted person area 411 and the non-permitted person area 412 may be rectangular.

[0148] The management device 50 determines whether or not the permitted person area 411 and the non-permitted person area 412 overlap each other (S1004).

[0149] If the permitted person area 411 and the non-permitted person area 412 do not overlap at all (S1004: NO), the management device 50 proceeds to step S1007.

[0150] If the permitted person area 411 and the non-permitted person area 412 partially overlap (S1004: YES), the management device 50 determines whether the permitted person area 411 overlaps at least a part of the upper part of the non-permitted person area 412 (S1005). For example, if the permitted person is located behind the non-permitted person as viewed from the first image capture device 21, the permitted person area 411 overlaps at least a part of the upper part of the non-permitted person area 412.

[0151] If it is determined in step S1005 that the permitted person area 411 overlaps at least a portion of the upper part of the non-permitted person area 412 (S1005: YES), the management device 50 proceeds to step S1007.

[0152] If it is determined in step S1005 that the permitted person area 411 does not overlap at least a portion of the upper part of the prohibited person area 412 (S1005: NO), that is, if the permitted person area 411 overlaps at least a portion of the prohibited person area 412 other than the upper part (for example, at least a portion of the lower part), the management device 50 performs the following process. That is, the management device 50 processes the image of the permitted person in the prohibited person area 412 so that it is not visible, and generates the unauthorized passerby image 500 (S1006). Details of this processing will be described later. The process then proceeds to step S1008.

[0153] In step S1007, the management device 50 generates an unauthorized passerby image 500 based on the image of the unauthorized person area 412 (S1007). For example, the management device 50 generates the unauthorized passerby image 500 by directly using the image of the unauthorized person area 412. Details of this process will be described later. Then, the process proceeds to step S1008.

[0154] In step S1008, the management device 50 displays the unauthorized passerby image 500 generated in step S1006 or step S1007 on the surveillance monitor 70, and returns the process to step S1001. When the management device 50 detects an unauthorized person 402 in the face matching zone 301 shown in FIG. 13 , the management device 50 may display the unauthorized passerby image 500 at the timing when the unauthorized person 402 enters the passage determination zone 302. However, the timing when the management device 50 displays the unauthorized passerby image 500 is not limited to this. For example, when the management device 50 detects an unauthorized person 402, the management device 50 may immediately display the unauthorized passerby image 500. Note that a method for displaying the unauthorized passerby image 500 on the surveillance monitor 70 will be described in detail later.

[0155] Through the above process, the unauthorized passerby image 500 corresponding to the unauthorized passerby is displayed on the surveillance monitor 70. Therefore, the surveillance staff 80 can quickly find the unauthorized passerby by looking at the surveillance monitor 70.

[0156] <When an authorized person area overlaps at least a portion of a prohibited person area other than the upper portion (for example, at least a portion of the lower portion)> Figure 21 is a diagram illustrating a case where an unauthorized passerby image is generated without processing when an authorized person area 411 overlaps at least a portion of the lower portion of a prohibited person area 412. Note that "above" the prohibited person area 412 may be a predetermined range including the upper edge of the prohibited person area 412. "Below" the prohibited person area 412 may be below the predetermined range including the upper edge of the prohibited person area 412.

[0157] When the permitted person area 411 overlaps at least a portion of the lower part of the prohibited person area 412, if the image of the prohibited person area 412 is used as an unauthorized passerby image without any processing, the unauthorized passerby image will include not only the facial image of the prohibited person but also the facial image of the authorized person, as shown in image 601 in Figure 21. If such an unauthorized passerby image is displayed on the surveillance monitor 70, the surveillance officer 80 may be confused about which of the facial images included in the unauthorized passerby image is the unauthorized passerby. Furthermore, it is undesirable to display the facial image of a person who is not an unauthorized passerby as an unauthorized passerby image on the surveillance monitor 70.

[0158] 20, the management device 50 performs processing so that the facial image of the authorized person is not displayed in the unauthorized passerby image 500, thereby generating the unauthorized passerby image 500. For example, the management device 50 performs either the following first processing method or second processing method.

[0159] Fig. 22A is a diagram for explaining a first processing method for a non-permitted person region according to Embodiment 2. Hereinafter, the processing of step S1006 in Fig. 20 will be described in detail with reference to Fig. 22A.

[0160] If the permitted person area 411 overlaps at least a portion of the lower part of the prohibited person area 412, that is, if a permitted person is present in front of the prohibited person as viewed from the exit side, the management device 50 performs processing to cut out the portion of the image of the prohibited person area 412 that is overlapped by the permitted person area 411, as shown in image 602A, to generate unauthorized passerby image 500A. As a result, the facial image of the permitted person is not included in unauthorized passerby image 500A.

[0161] Alternatively, if the permitted person area 411 overlaps at least a portion of the lower part of the unauthorized person area 412, that is, if an authorized person is present in front of the unauthorized person as viewed from the exit side, the management device 50 performs processing to mask the portion of the image of the unauthorized person area 412 that overlaps with the permitted person area 411, as shown in image 603A, to generate unauthorized passerby image 500B. As a result, the facial image of the authorized person becomes invisible in unauthorized passerby image 500B.

[0162] Through the above process, the surveillance monitor 70 can display the unauthorized passerby image 500 (500A, 500B) without displaying the face image of the authorized person.

[0163] 22B is a diagram for explaining a modification of the first processing method of the unauthorized person area according to Embodiment 2. The management device 50 may apply the processing method shown in FIG. 22B instead of the processing method shown in FIG. 22A.

[0164] When the permitted person area 411 overlaps at least a portion of the lower part of the unauthorized person area 412, that is, when viewed from the exit side, when a permitted person is present in front of the unauthorized person, the management device 50 may perform processing to cut out the portion of the facial image included in the permitted person area 411 from the image of the unauthorized person area 412, as shown in image 602B, to generate unauthorized passerby image 500A. As a result, the facial image of the permitted person is not included in unauthorized passerby image 500A.

[0165] Alternatively, if the permitted person area 411 overlaps at least a portion of the lower part of the unauthorized person area 412, that is, if an authorized person is present in front of the unauthorized person when viewed from the exit side, the management device 50 performs processing to mask the portion of the facial image included in the permitted person area 411 of the image of the unauthorized person area 412, as shown in image 603B, to generate unauthorized passerby image 500B. As a result, the facial image of the authorized person cannot be seen in unauthorized passerby image 500B.

[0166] Through the above process, the surveillance monitor 70 can display the unauthorized passerby image 500 (500A, 500B) without displaying the face image of the authorized person.

[0167] FIG. 23 is a diagram for explaining a processing method in the case where a plurality of permitted person regions 411 overlap at least partly below a non-permitted person region 412 according to the second embodiment.

[0168] When multiple permitted person regions 411 overlap at least a portion below the prohibited person region 412, that is, when multiple permitted persons are present in front of the prohibited person as viewed from the exit side, the management device 50 generates an unauthorized passerby image 500C by cutting out the overlapping portions of the permitted person regions 411 from the image of the prohibited person region 412, as shown in image 604. Note that, as in FIG. 22B , the management device 50 may also generate an unauthorized passerby image 500C by cutting out the facial image portions of the permitted person regions 411 from the image of the prohibited person region 412.

[0169] Alternatively, when multiple permitted person regions 411 overlap at least a portion below the prohibited person region 412, that is, when multiple permitted persons are present in front of the prohibited person as viewed from the exit side, the management device 50 performs processing to mask the portions of the image of the prohibited person region 412 that are overlapped by the permitted person regions 411, as shown in image 605, to generate an unauthorized passerby image 500D. Note that, as in FIG. 22B , the management device 50 may also perform processing to mask the facial image portions of the permitted person regions 411 from the image of the prohibited person region 412 to generate an unauthorized passerby image 500D.

[0170] 24 is a diagram for explaining a second processing method for the unauthorized person area 412 according to Embodiment 2. The process of step S1006 in FIG. 20 will be described in detail below with reference to FIG.

[0171] If the permitted person area 411 overlaps at least a portion of the lower part of the prohibited person area 412, that is, if a permitted person is present in front of the prohibited person when viewed from the exit side, the management device 50 generates an unauthorized passerby image 500E by cutting out the portion below the upper edge Z1 where the permitted person area 411 overlaps from the image of the prohibited person area 412, as shown in image 606. As a result, the unauthorized passerby image 500E does not include the facial image of the permitted person.

[0172] Alternatively, if the permitted person area 411 overlaps at least a portion of the lower part of the unauthorized person area 412, that is, if an authorized person is present in front of the unauthorized person when viewed from the exit side, the management device 50 generates an unauthorized passerby image 500F by masking the portion of the image of the unauthorized person area 412 below the upper edge Z1 where the permitted person area 411 overlaps, as shown in image 607. This makes the facial image of the authorized person invisible in the unauthorized passerby image 500F.

[0173] Through the above processing, the surveillance monitor 70 can display the unauthorized passerby image 500 (500E, 500F) in which the face image of the authorized person 401 is not displayed.

[0174] <When an authorized person area overlaps at least a part of the upper portion of a prohibited person area> Fig. 25 is a diagram for explaining a method for generating an unauthorized passerby image 500 when an authorized person area 411 overlaps at least a part of the upper portion of a prohibited person area 412 according to Embodiment 2. Hereinafter, the processing of step S1007 in Fig. 20 will be described in detail with reference to Fig. 25 .

[0175] When the permitted person area 411 overlaps at least a portion of the upper part of the prohibited person area 412, that is, when viewed from the exit side, when the permitted person 401 is located behind the prohibited person 402, the management device 50 may use the image of the prohibited person area 412 as the unauthorized passerby image 500G as is, as shown in image 608. This is because even if the image of the prohibited person area 412 is used as the unauthorized passerby image 500G as is, the facial image of the permitted person 401 will not be included in the unauthorized passerby image 500G.

[0176] Through the above processing, the surveillance monitor 70 can display the unauthorized passerby image 500 (500G) in which the face image of the authorized person 401 is not displayed.

[0177] <Details of Display of Unauthorized Passerby Images> The first image capturing device 21 continuously captures images of people passing through the passage 2, thereby generating multiple captured images in chronological order. Accordingly, multiple unauthorized passerby images 500 of unauthorized passerby 402 are also generated in chronological order and stored in the memory 1002 or the storage 1003. The management device 50 may preferentially display on the surveillance monitor 70, among the multiple generated unauthorized passerby images 500, those that satisfy a predetermined condition A. For example, the management device 50 may determine which unauthorized passerby image 500 to display preferentially based on one of the following conditions A1, A2, or A3.

[0178] <<Condition A1>> The management device 50 preferentially displays, on the surveillance monitor 70, the unauthorized passerby image 500 generated using the most recently captured image among the multiple unauthorized passerby images 500. This allows the surveillance staff 80 to check the unauthorized passerby image 500 that includes the most recently captured facial image of the unauthorized person 402.

[0179] <<Condition A2>> The management device 50 preferentially displays the unauthorized passerby image 500 with the smallest proportion of the cut-out or masked area among the multiple unauthorized passerby images 500 on the surveillance monitor 70. This allows the surveillance staff 80 to confirm the unauthorized passerby image 500 in which the entire body of the unauthorized person 402 is more clearly visible.

[0180] <<Condition A3>> The management device 50 preferentially displays, among the multiple unauthorized passerby images 500, the unauthorized passerby image 500 having the largest ratio of the maximum vertical length d2 to the maximum horizontal length d1 (see FIGS. 22, 23, and 24) on the surveillance monitor 70. This allows the surveillance staff 80 to confirm the unauthorized passerby image 500 in which the entire body of the unauthorized person 402 is more clearly visible.

[0181] Furthermore, the plurality of unauthorized passerby images 500 under the above-mentioned condition A may be selected as unauthorized passerby images 500 that satisfy a predetermined condition B. For example, the management device 50 may select unauthorized passerby images 500 from the entire unauthorized passerby images 500 based on any one of the following conditions B1, B2, B3, and B4, and set them as the plurality of unauthorized passerby images 500 under the above-mentioned condition A.

[0182] <<Condition B1>> The management device 50 selects unauthorized passerby images 500 generated using images captured in the face matching zone 301 from the entire unauthorized passerby images 500. In this way, unauthorized passerby images 500 that are highly likely to show the entire body of the unauthorized person 402 are selected.

[0183] <<Condition B2>> The management device 50 selects, from all unauthorized passerby images 500, unauthorized passerby images 500 generated using images captured in the passage determination zone 302. In this way, unauthorized passerby images 500 that show the unauthorized person 402 after unauthorized passage has been confirmed are selected.

[0184] <<Condition B3>> The management device 50 selects unauthorized passerby images 500 that include the facial image of the unauthorized person 402 from all unauthorized passerby images 500. As a result, unauthorized passerby images 500 that do not include the facial image of the unauthorized person 402 are excluded.

[0185] <<Condition B4>> The management device 50 selects unauthorized passerby images in which the face of the unauthorized person 402 is facing forward within a predetermined range (for example, within a range from −α degrees to +α degrees) from the entire unauthorized passerby image 500. This selects unauthorized passerby images 500 in which the face of the unauthorized person 402 is facing more forward.

[0186] The management device 50 may select the unauthorized passerby image 500 based on a combination of two or more of the above-mentioned conditions B1 to B4.

[0187] FIG. 26 is a diagram showing an example in which the display mode of the unauthorized passerby image 500 is changed depending on the position of the unauthorized person 402 on the passage 2 according to the second embodiment.

[0188] The management device 50 may change the display position of the unauthorized passerby image 500 on the surveillance monitor 70 in conjunction with the position of the unauthorized person 402 in the width direction of the passage 2 .

[0189] For example, as shown in (a) of Figure 26, when the unauthorized person 402 is located at the right end of the passage determination zone 302 as viewed from the exit side of passage 2, the management device 50 displays the unauthorized passerby image 500 on the right side of the surveillance monitor 70. Then, as shown in (b) and (c) of Figure 26, when the unauthorized person 402 moves diagonally from the right end of the passage determination zone 302 to the left end as viewed from the exit side of passage 2, the management device 50 accordingly changes the display position of the unauthorized passerby image 500 from the right side to the left side of the surveillance monitor 70. This allows the monitor 80 to see the position of the unauthorized passerby image 500 displayed on the surveillance monitor 70 and grasp the position in passage 2 of the unauthorized passerby 402 corresponding to the unauthorized passerby image 500. This can improve the efficiency with which the monitor 80 captures the unauthorized person 402.

[0190] FIG. 27 is a diagram showing an example in which a plurality of unauthorized passerby images 500 according to the second embodiment are displayed.

[0191] When displaying one unauthorized passerby image 500, the management device 50 may display one unauthorized passerby image 500 using the entire screen of the surveillance monitor 70, as shown in FIG. 27(a).

[0192] When displaying a plurality of unauthorized passerby images 500 (500H, 500I), the management device 50 may increase the number of unauthorized passerby images 500 displayed by vertically dividing the screen of the surveillance monitor 70 as shown in FIG. 27(b). In this case, the management device 50 may display the unauthorized passerby image 500 with the most recent photographed time on the top row (or bottom row). The management device 50 may also increase the number of unauthorized passerby images 500 displayed by horizontally dividing the screen of the surveillance monitor 70. In this case, the management device 50 may display the unauthorized passerby image 500 with the most recent photographed time on the far right (or far left).

[0193] Furthermore, the management device 50 may prevent the display of multiple unauthorized passerby images 500 of the same unauthorized person 402. For example, the management device 50 may update a previous unauthorized passerby image 500 of the same unauthorized person 402 to a new unauthorized passerby image 500 captured at the time of capture, rather than displaying an additional unauthorized passerby image 500 of the same unauthorized person 402.

[0194] This allows the monitor 80 to view the surveillance monitor 70 and recognize the timing in chronological order when each of the multiple unauthorized persons 402 was detected. This allows the monitor 80 to, for example, prioritize capturing an older unauthorized person 402 in chronological order, or prioritize capturing a newer unauthorized person 402 in chronological order. In other words, the monitor 80 can view the surveillance monitor 70 and determine which unauthorized person 402 should be prioritized for capture.

[0195] When the matching of an unauthorized person 402 in the passage judgment zone 302 is successful (i.e., when the unauthorized person 402 is determined to be a permitted person 401), the management device 50 may automatically hide the display of the unauthorized passerby image 500 of the unauthorized person 402. This prevents the face image of a person who failed to match in the face matching zone 301 and was determined to be an unauthorized person 402 but was successfully matched in the passage judgment zone 302 (permitted person 401) from being continuously displayed on the surveillance monitor 70.

[0196] FIG. 28 is a diagram showing an example of displaying a plurality of images 500 of unauthorized passersby taken in time series like a moving image according to the second embodiment.

[0197] The management device 50 may display a plurality of unauthorized passerby images 500 captured in chronological order on the surveillance monitor 70, and may repeat the display in chronological order. For example, as shown in Fig. 28 , the management device 50 may sequentially display an unauthorized passerby image 500 captured at time t = 1, an unauthorized passerby image 500 captured at time t = 2, an unauthorized passerby image 500 captured at time t = 3, and an unauthorized passerby image 500 captured at time t = 4, and then repeatedly return to displaying the unauthorized passerby image 500 captured at time t = 1.

[0198] Furthermore, even after the unauthorized person 402 has passed through the exit gate, the management device 50 may repeatedly display on the surveillance monitor 70 in chronological order as described above, multiple unauthorized passerby images 500 taken while the unauthorized person 402 was passing through passage 2.

[0199] As a result, the unauthorized passerby image 500 is displayed on the surveillance monitor 70 like a moving image, which makes it easier for the surveillance officer 80 to identify the unauthorized person 402 passing through the passage 2, compared to when the unauthorized passerby image 500 is displayed as a single still image. In addition, such a moving display attracts the attention of people who may be attempting to pass unauthorizedly, which helps to deter unauthorized passage.

[0200] (Hardware Configuration) FIG. 29 is a diagram illustrating an example of a hardware configuration of an information processing device according to the present disclosure.

[0201] The above-described first edge device 41, second edge device 42, management device 50, and face matching device 60 may be configured to include an information processing device 1000 shown in Fig. 29. Furthermore, the processes performed by the above-described first edge device 41, second edge device 42, management device 50, and face matching device 60 may be realized by the information processing device 1000 shown in Fig. 29 executing a computer program.

[0202] The information processing device 1000 includes a processor 1001, a memory 1002, a storage 1003, an input device 1004, an output device 1005, a communication device 1006, a graphics processing unit (GPU) 1007, a reading device 1008, and a bus 1009. The processor 1001, the memory 1002, the storage 1003, the input device 1004, the output device 1005, the communication device 1006, the GPU 1007, and the reading device 1008 are connected to the bus 1009 and can transmit and receive data bidirectionally via the bus 1009.

[0203] The processor 1001 executes computer programs stored in the memory 1002 to realize the processing performed by each of the above-mentioned devices. Therefore, the processing performed by each of the above-mentioned devices can be interpreted as processing mainly performed by the processor 1001. Examples of the processor 1001 include a central processing unit (CPU), a micro processing unit (MPU), a controller, a large scale integration (LSI), an application specific integrated circuit (ASIC), a programmable logic device (PLD), and a field-programmable gate array (FPGA).

[0204] The memory 1002 stores computer programs and data used by the information processing device 1000. The memory 1002 may include a read-only memory (ROM) and a random access memory (RAM). The memory 1002 may be configured as either a volatile storage medium or a non-volatile storage medium.

[0205] The storage 1003 is configured as a nonvolatile storage medium and stores computer programs and data handled by the information processing apparatus 1000. Examples of the storage 1003 include a hard disk drive (HDD), a solid state drive (SSD), and a flash memory.

[0206] The input device 1004 accepts data to be input to the processor 1001. Examples of the input device 1004 include a keyboard, a mouse, a touchpad, and a microphone.

[0207] The output device 1005 outputs data generated by the processor 1001. For example, the output device 1005 outputs display data generated by the processor 1001 to the first display device 31 or the second display device 32. Alternatively, the output device 1005 may be a display or monitor that displays the display data itself. Alternatively, the output device 1005 outputs audio data generated by the processor 1001 to a speaker.

[0208] The communication device 1006 transmits and receives data to and from other devices via the communication network 4. The communication device 1006 may include a transmitter that transmits data and a receiver that receives data. The communication device 1006 may support either wired communication or wireless communication.

[0209] The GPU 1007 is a device that processes image rendering at high speed. The GPU 1007 may be used for AI processing (e.g., machine learning or deep learning).

[0210] The reading device 1008 is a device that reads data from an external recording medium such as a Digital Versatile Disk Read Only Memory (DVD-ROM) or a Universal Serial Bus (USB) memory.

[0211] (Summary of Second Embodiment) The above description of the second embodiment discloses the following techniques.

[0212] <Technology 1> An information processing device (e.g., management device 50) comprising a processor (1001) and a memory (1002), wherein the processor cooperates with the memory to set, for a captured image of a plurality of persons passing through a predetermined passage (2), a permitted person area (411) including facial images of persons who are permitted to pass through the passage and a disallowed person area (412) including facial images of persons who are not permitted to pass through the passage, and when the permitted person area and the disallowed person area partially overlap, an unauthorized passerby image (500) is generated in which the facial images included in the permitted person area are not displayed but the facial images included in the disallowed person area are displayed, and the unauthorized passerby image is displayed on a predetermined monitor (e.g., surveillance monitor 70).

[0213] This allows the monitor (80) to see the image of the unauthorized passerby displayed on the monitor and recognize the person who is passing through the passage illegally, and also prevents the image of the unauthorized passerby including the image of a person who is permitted to pass from being displayed on the monitor.

[0214] <Technology 2> In the information processing device described in Technology 1, when the permitted person area overlaps at least a part of the prohibited person area except for the upper part, the processor cuts out or masks the area in the image of the prohibited person area that is partially overlapped by the permitted person area to generate an image of the unauthorized passerby.

[0215] This makes it possible to prevent an image of an unauthorized passerby, including an image of a person who is permitted to pass, from being displayed on the monitor.

[0216] <Technology 3> In the information processing device described in Technology 1, when an authorized person area overlaps at least a part of the unauthorized person area other than the upper part, the processor cuts out or masks the facial image included in the authorized person area in the image of the unauthorized person area to generate an image of an unauthorized passerby.

[0217] This makes it possible to prevent an image of an unauthorized passerby, including an image of a person who is permitted to pass, from being displayed on the monitor.

[0218] <Technology 4> In the information processing device according to any one of technologies 1 to 3, when the permitted person area overlaps at least a part of the upper part of the prohibited person area, the processor generates an unauthorized passerby image from the image of the prohibited person area.

[0219] This makes it possible to generate an image of an unauthorized passerby that does not include facial images of people who are permitted to pass.

[0220] <Technology 5> In the information processing device according to any one of technologies 1 to 4, the processor preferentially displays on the monitor an unauthorized passerby image generated using the most recently captured image among a plurality of unauthorized passerby images.

[0221] This allows the monitor to check the last captured (i.e., most recent) image of the unauthorized passerby on the monitor.

[0222] <Technology 6> In the information processing device according to any one of technologies 1 to 5, the processor preferentially displays on the monitor, among a plurality of unauthorized passerby images, an unauthorized passerby image having the smallest proportion of a cut-out or masked area.

[0223] This allows the monitor to check an image of the unauthorized passerby on the monitor, which shows the entire body of the unauthorized passerby.

[0224] <Technology 7> In the information processing device according to any one of technologies 1 to 6, the processor preferentially displays on the monitor, among a plurality of unauthorized passerby images, the unauthorized passerby image having the largest ratio of maximum vertical length to maximum horizontal length.

[0225] This allows the monitor to check an image of the unauthorized passerby on the monitor, which shows the entire body of the unauthorized passerby.

[0226] <Technology 8> In the information processing device according to any one of technologies 1 to 7, the captured image is captured by a camera installed to capture an image in the direction from the exit side to the entrance side of the passage.

[0227] As a result, the photographing device captures an image including the face of the person walking through the passage toward the exit.

[0228] <Technology 9> In the information processing device according to either Technology 1 or Technology 8, the unauthorized passerby image is generated using a captured image including a face image that can be used for face matching of a person.

[0229] This allows an image of the unauthorized passerby including the face of the unauthorized passerby to be generated.

[0230] <Technology 10> In the information processing device according to any one of technologies 1 to 9, the unauthorized passerby image is generated using a captured image including a face image in which the orientation of the face relative to the front of the person is within a predetermined range.

[0231] This makes it possible to generate an image of the unauthorized passerby that includes a substantially frontal view of the face of the unauthorized passerby.

[0232] <Technology 11> In the information processing device described in any one of technologies 1 to 10, the processor changes the display mode on the monitor of the unauthorized passerby image corresponding to the person who is not allowed to pass, depending on the position in the width direction of the passage where the person is detected.

[0233] This allows the monitor to see the image of the unauthorized passerby displayed on the monitor and grasp the position of the unauthorized passerby in the passageway.

[0234] <Technology 12> In the information processing device according to any one of technologies 1 to 11, the processor changes the display mode of the unauthorized passerby images on the monitor in accordance with the number of unauthorized passerby images to be displayed on the monitor.

[0235] This allows the monitor to display unauthorized passerby images corresponding to a plurality of unauthorized passersby even when a plurality of unauthorized passersby are detected.

[0236] <Technology 13> In the information processing device according to any one of techniques 1 to 12, the processor displays a plurality of unauthorized passerby images consecutively in chronological order of their capture times.

[0237] This allows the monitor to display a simple moving image of an unauthorized passerby, which will attract the attention of security personnel and passersby, and is expected to have a deterrent effect on those attempting to pass illegally.

[0238] <Technology 14> In the information processing device described in Technology 13, after a person who is not permitted to pass passes through a passageway, the processor repeatedly displays a plurality of unauthorized passerby images, including images taken of the person while the person is passing through the passageway, in chronological order of the time of the images being taken.

[0239] This allows the monitor to display an image of the unauthorized passerby, like a simple video, even after the unauthorized passerby has passed through the passageway. This makes the monitor's display eye-catching, and is expected to have a deterrent effect on those attempting to pass through the passageway illegally.

[0240] <Technology 15> In an information processing device according to any one of technologies 1 to 14, a face matching zone (301) for performing face matching is set on the entrance side of a passage, and a passage determination zone (302) for performing pass-through determination is set on the exit side of the passage, and the processor determines whether or not a person is permitted to pass through the passage based on the result of matching of a face image included in an image taken in the face matching zone, and when a person who has been determined not to be permitted to pass through the passage enters the passage determination zone, an unauthorized passerby image of the person is displayed.

[0241] This allows the monitor to display an image of the unauthorized passerby at the time when the unauthorized passerby is confirmed.

[0242] <Technology 16> In the information processing device according to Technology 15, the unauthorized passerby image is generated using an image captured in a face matching zone of a person who has been determined not to be permitted to pass through the passage.

[0243] Since there is a high possibility that the whole body of a person is captured in an image captured in the face matching zone, the monitor can thereby display an image of an illegal passerby that is highly likely to include the whole body of the illegal passerby.

[0244] <Technology 17> In the information processing device according to Technology 15, the unauthorized passerby image is generated using an image of a person who has been determined not to be permitted to pass through the passage, photographed in a passage determination zone.

[0245] This allows the monitor to display an image of the unauthorized passerby captured after the unauthorized passerby has been confirmed.

[0246] <Technology 18> The person matching system includes an imaging device (e.g., a first imaging device 21) that generates an image of a plurality of persons passing through a predetermined passage (2), a management device (50) that sets, in the captured image, a permitted person area (411) including facial images of persons who are permitted to pass through the passage and a non-permitted person area (412) including facial images of persons who are not permitted to pass through the passage, and, when the permitted person area and the non-permitted person area partially overlap, generates an unauthorized passerby image (500) in which the facial image included in the permitted person area is not displayed and the facial image included in the non-permitted person area is displayed, and a monitor (e.g., a surveillance monitor 70) that displays the unauthorized passerby image.

[0247] This allows the monitor (80) to see the image of the unauthorized passerby displayed on the monitor and recognize the person who is passing through the passage illegally, and also prevents the image of the unauthorized passerby including the image of a person who is permitted to pass from being displayed on the monitor.

[0248] <Technology 19> The image display method sets, for a photographed image of a plurality of persons passing through a predetermined passageway (2), a permitted person area (411) including facial images of persons who are permitted to pass through the passageway and a non-permitted person area (412) including facial images of persons who are not permitted to pass through the passageway, and when the permitted person area and the non-permitted person area partially overlap, generates an unauthorized passerby image (500) in which the facial images included in the permitted person area are not displayed and the facial images included in the non-permitted person area are displayed, and displays the unauthorized passerby image on a predetermined monitor (e.g., surveillance monitor 70).

[0249] This allows the monitor (80) to see the image of the unauthorized passerby displayed on the monitor and recognize the person who is passing through the passage illegally, and also prevents the image of the unauthorized passerby including the image of a person who is permitted to pass from being displayed on the monitor.

[0250] Although the embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications, alterations, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also fall within the technical scope of the present disclosure. Furthermore, the components in the above-described embodiments may be combined in any manner without departing from the spirit of the invention.

[0251] The techniques of this disclosure are useful for monitoring unauthorized persons.

[0252] 1 Person 2 Passage 3 Detection area 4 Communication network 9 Gate system 10 Gate device 11 First gate 12 Second gate 21 First image capturing device 22 Second image capturing device 31 First display device 32 Second display device 41 First edge device 42 Second edge device 50 Management device 60 Face matching device 70 Surveillance monitor 100 Person information 102 First image capturing time 103 Image update time 104 Body image 105 Body image update time 106 Body coordinates 107 Face image 108 Face image update time 109 Face coordinates 110 Face matching ID 111 Matching status 112 Movement direction 200 Body image 201, 201A, 201B Body coordinates 202 Body coordinates 220 Face image 221, 221A Face coordinates 301 Face matching zone 302 Passage determination zone 311 Predetermined area closer to first gate 312 Predetermined area closer to second gate 401 Permitted person 402 Unpermitted person 411 Permitted person area 412 Unpermitted person area 500, 500A, 500B, 500C, 500D, 500E, 500F, 500G, 500H, 500I Images of unauthorized passersby 601, 602A, 602B, 603A, 603B, 604, 605, 606, 607, 608 Images 1000 Information processing device 1001 Processor 1002 Memory 1003 Storage 1004 Input device 1005 Output device 1006 Communication device 1007 GPU 1008 Reading device 1009 Bus

Claims

1. An information processing device comprising a processor and a memory, wherein the processor cooperates with the memory to: set, in a captured image of a plurality of persons passing through a passageway, an allowed person area including images of persons who are permitted to pass through the passageway, and a disallowed person area including images of persons who are not permitted to pass through the passageway; when the allowed person area and the disallowed person area partially overlap, generate at least one unauthorized passerby image in which at least a portion of the image included in the allowed person area is not displayed and the image included in the disallowed person area is displayed; and display the at least one unauthorized passerby image on a monitor.

2. The information processing device according to claim 1, wherein the image of the person included in the permitted person area includes a facial image of the person, and the image of the person included in the non-permitted person area includes a facial image of the person.

3. The information processing device described in claim 2, wherein, when the permitted person area overlaps at least a portion of the prohibited person area other than the upper part, the processor cuts out or masks the area in the image of the prohibited person area that is partially overlapped by the permitted person area to generate the at least one unauthorized passerby image.

4. The information processing device described in claim 2, wherein, when the permitted person area overlaps at least a part of the prohibited person area other than the upper part, the processor cuts out or masks a facial image included in the permitted person area in the image of the prohibited person area to generate the at least one unauthorized passerby image.

5. An information processing device according to any one of claims 1 to 4, wherein the processor generates the at least one unauthorized passerby image from an image of the unauthorized person area when the authorized person area overlaps at least a portion of the upper part of the unauthorized person area.

6. An information processing device according to any one of claims 1 to 4, wherein the at least one unauthorized passerby image is a plurality of unauthorized passerby images, and the processor preferentially displays on the monitor an unauthorized passerby image generated using the most recently captured image among the plurality of unauthorized passerby images.

7. The information processing device according to claim 3 or 4, wherein the at least one unauthorized passerby image is a plurality of unauthorized passerby images, and the processor preferentially displays on the monitor, among the plurality of unauthorized passerby images, an unauthorized passerby image having the smallest proportion of the area that is cut out or masked.

8. The information processing device according to claim 3 or 4, wherein the at least one unauthorized passerby image is a plurality of unauthorized passerby images, and the processor preferentially displays on the monitor, among the plurality of unauthorized passerby images, the unauthorized passerby image having the largest ratio of maximum vertical length to maximum horizontal length.

9. An information processing device according to any one of claims 1 to 4, wherein the captured image is captured by a camera installed to capture an image in the direction from the exit side to the entrance side of the passage.

10. The information processing device according to any one of claims 2 to 4, wherein the at least one unauthorized passerby image is generated using the captured image that includes a facial image that can be matched with a person's face.

11. An information processing device according to any one of claims 2 to 4, wherein the at least one unauthorized passerby image is generated using a captured image including a facial image in which the orientation of the face relative to the front of the person is within a predetermined range.

12. An information processing device according to any one of claims 1 to 4, wherein the processor changes the display mode on the monitor of the at least one unauthorized passerby image corresponding to the person who is not permitted to pass, depending on the position in the width direction of the passage where the person is detected.

13. An information processing device according to any one of claims 1 to 4, wherein the processor changes the display mode of the at least one unauthorized passerby image on the monitor depending on the number of the at least one unauthorized passerby image to be displayed on the monitor.

14. An information processing device according to any one of claims 1 to 4, wherein the at least one unauthorized passerby image is a plurality of unauthorized passerby images, and the processor displays the plurality of unauthorized passerby images consecutively in chronological order of their photographing times.

15. The information processing device according to claim 14, wherein the at least one unauthorized passerby image is a plurality of unauthorized passerby images, and the processor repeatedly displays the plurality of unauthorized passerby images, including images taken of the unauthorized passerby while the unauthorized passerby is passing through the passageway, in chronological order of the time of the image capture after the unauthorized passerby has passed through the passageway.

16. An information processing device according to any one of claims 2 to 4, wherein a face matching zone for performing face matching is set on the entrance side of the passage, and a passage determination zone for performing passage determination is set on the exit side of the passage, and the processor determines whether or not a person is permitted to pass through the passage based on the matching results of a face image contained in an image taken in the face matching zone, and when a person who has been determined not to be permitted to pass through the passage enters the passage determination zone, the processor displays at least one unauthorized passerby image for the person.

17. The information processing device according to claim 16, wherein the at least one unauthorized pedestrian image is generated using an image of a person who has been determined not to be permitted to pass through the passage, photographed in the face matching zone.

18. The information processing device according to claim 16, wherein the at least one unauthorized passerby image is generated using an image of a person who has been determined not to be permitted to pass through the passage, taken in the passage determination zone.

19. A person verification system comprising: a photographing device that generates a photographed image of a plurality of people passing through a passage; a management device that sets, in the photographed image, an allowed person area including images of people who are permitted to pass through the passage and an unallowed person area including images of people who are not permitted to pass through the passage, and generates an unauthorized passerby image in which, when the allowed person area and the unallowed person area partially overlap, at least a portion of the image included in the allowed person area is not displayed and the image included in the unallowed person area is displayed; and a monitor that displays the unauthorized passerby image.

20. An image display method comprising: setting, for a captured image of multiple people passing through a passageway, a permitted person area including images of people who are permitted to pass through the passageway and a disallowed person area including images of people who are not permitted to pass through the passageway; if the permitted person area and the disallowed person area partially overlap, generating an unauthorized passerby image in which at least a portion of the image included in the permitted person area is not displayed and the image included in the disallowed person area is displayed; and displaying the unauthorized passerby image on a monitor.

Images