Monitoring system and monitoring method

Abstract

To make it possible to inhibit an erroneous report from being generated, by enabling a user to quickly and properly perform determination for maintenance of registration information on a monitoring target person by referring to an execution situation of past alarming with respect to respective monitoring target persons.SOLUTION: In response to operation of a user requesting browsing, a monitoring system generates alarming history outline information on the basis of alarming history information for each monitoring target person and displays a target person list screen 201 on which the alarming history information for each monitoring target person is added to registration information for each monitoring target person. Especially, in an alarming lapse period display column 241, an alarming lapse period on the monitoring target person, which is a lapse period from a last alarming time to a current time (the latest non-alarming period), is displayed as the alarming history outline information. Furthermore, in an alarming count display column 242, the number of times of executing alarming with respect to the monitoring target persons, which is the number of times of executing alarming in the past, is displayed as the alarming history outline information.SELECTED DRAWING: Figure 13

Description

Technical Field

[0001] The present invention relates to a monitoring system and a monitoring method for executing, by a processor, a process of performing notification regarding detection of a person to be monitored as a notification event based on an image of a visitor to a facility being photographed.

Background Art

[0002] In stores such as supermarkets and shopping malls, for crime prevention, a person who has engaged in problematic behavior (such as theft, violence, drinking, etc.) in the store in the past is registered as a person to be monitored, and the arrival of the person to be monitored is detected as a notification event, and a monitoring system that notifies store managers, store employees, security guards, etc. is known.

[0003] In such a monitoring system, it is possible to detect the arrival of a person to be monitored by photographing a visitor with a camera and performing face verification. On the other hand, unlike face verification for determining whether a user is a legitimate user, since the face of a person cannot be photographed in an optimal state at the time of registration, for example, a face image of a person to be monitored is acquired from a monitoring video of a camera and registered.

[0004] As a technique related to the registration of face images in such a monitoring system, conventionally, when a store employee discovers a person who has engaged in problematic behavior in the store and reports it to the head office, a photographed image in which the target person appears is searched, and a face image of the target person is generated from the photographed image. The face image of the target person is presented to the user as a candidate image for registration, and the face image of the target person is registered according to the user's selection operation (see Patent Document 1).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] In a surveillance system that detects a person to be monitored using face recognition technology, unlike an authentication system that also uses face recognition technology, there is no procedure for registering and deleting users. Therefore, once a person is registered as a person to be monitored, a face image of a person to be monitored with a low probability of coming to the store is left registered, and the number of face images to be subject to face recognition increases uselessly.

[0007] Furthermore, as described above, in a surveillance system for crime prevention, unlike face recognition for determining whether a user is a legitimate user, it is not possible to photograph a person's face in an optimal state at the time of registration. Therefore, for example, a face image of a person to be monitored is acquired from a surveillance video of a camera and registered. However, in this method, it may be difficult to acquire an appropriate face image of the person to be monitored. For this reason, the possibility of false alarms that detect another person as a person to be monitored increases.

[0008] For this reason, it is desirable to perform maintenance of registration information regarding a person to be monitored at an appropriate timing. For example, if there is a person to be monitored who has not come to the store recently, it may be possible to exclude the person from the monitoring target or temporarily exclude the person from the reporting target without excluding the person from the monitoring target. However, the conventional technology has no consideration from such a viewpoint and has a problem that it is difficult to suppress the occurrence of false alarms.

[0009] Therefore, the main object of the present invention is to provide a surveillance system and a surveillance method that can suppress the occurrence of false alarms by enabling a user to quickly and appropriately make a judgment for maintaining registration information regarding a person to be monitored with reference to the past reporting implementation status regarding each person to be monitored.

Means for Solving the Problems

[0010] The monitoring system of the present invention is a monitoring system that executes, by a processor, a process of issuing a report regarding detection of a person to be monitored as a reporting event based on an image of a visitor to a facility, including a camera that acquires a captured image within the facility, a registered image storage unit that stores registered face images for each person to be monitored, a reporting history storage unit that stores reporting history information regarding reports made in the past for each person to be monitored, a terminal device on which an operation of a user who manages information for each person to be monitored is performed, and a notification device that performs a predetermined reporting operation in response to an instruction from the processor. The processor executes a face verification process of comparing a face image of a visitor with the registered face image of the person to be monitored to determine whether they are the same person, issues a report regarding detection of the person to be monitored based on the result, and generates summary reporting history information based on the reporting history information for each person to be monitored in response to an operation of a user who requests browsing, and causes the terminal device to display a screen in which the summary reporting history information for each person to be monitored is added to the registration information for each person to be monitored.

[0011] Further, the monitoring method of the present invention is a monitoring method that executes, by a processor, a process of issuing a report regarding detection of a person to be monitored as a reporting event based on an image of a visitor to a facility, stores reporting history information regarding reports made in the past for each person to be monitored in a reporting history storage unit, executes a face verification process of comparing a face image of a visitor with the registered face image of the person to be monitored to determine whether they are the same person, issues a report regarding detection of the person to be monitored based on the result, and generates summary reporting history information based on the reporting history information for each person to be monitored in response to an operation of a user who requests browsing, and causes the terminal device to display a screen in which the summary reporting history information for each person to be monitored is added to the registration information for each person to be monitored.

Advantages of the Invention

[0012] According to the present invention, a user can quickly and appropriately make a judgment for maintenance of registration information of a person to be monitored, for example, a judgment on the necessity of exclusion from a monitoring target or a reporting target for each person to be monitored, with reference to summary reporting history information representing the implementation status of past reports regarding each person to be monitored, so that the occurrence of false reports can be suppressed.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Mode for Carrying Out the Invention

[0014] A first invention made to solve the above problems is a monitoring system that executes, by a processor, a process of issuing a report regarding detection of a person to be monitored as a reporting event based on an image of a visitor to a facility, including a camera that acquires a captured image within the facility, a registered image storage unit that stores registered face images for each person to be monitored, a reporting history storage unit that stores reporting history information regarding reports made in the past for each person to be monitored, a terminal device on which an operation of a user who manages information for each person to be monitored is performed, and a notification device that performs a predetermined reporting operation in response to an instruction from the processor. The processor executes a face verification process of comparing a face image of a visitor with a registered face image of the person to be monitored to determine whether they are the same person, issues a report regarding detection of the person to be monitored based on the result, and generates summary reporting history information based on the reporting history information for each person to be monitored in response to an operation of a user who requests browsing, and causes the terminal device to display a screen in which the summary reporting history information for each person to be monitored is added to the registration information for each person to be monitored.

[0015] According to this, since a user can quickly and appropriately make a judgment for maintenance of the registration information of a person to be monitored, for example, a judgment on the necessity of exclusion from the monitoring target or the reporting target, with reference to the summary reporting history information indicating the summary of the implementation status of past reports for each person to be monitored, the occurrence of false reports can be suppressed.

[0016] Further, a second invention is configured such that the summary reporting history information is the elapsed time from when the last report was made until now.

[0017] According to this, a user can appropriately judge the necessity of exclusion from the monitoring target or the reporting target based on the elapsed time (the most recent non-reporting period) from when the last alarm report was made until now.

[0018] Further, a third invention is configured such that the summary reporting history information is the number of times reports were made in the past.

[0019] According to this, the user can appropriately determine the necessity of exclusion from the monitoring target or the notification target based on the number of times (notification times) the alarm has been issued in the past. Note that it may be the number of times within a limited period.

[0020] Further, in the fourth invention, the processor is configured to cause the terminal device to display a screen including information regarding the number of face images registered for each monitoring target person.

[0021] According to this, the user can determine the necessity of additional registration of face images based on the number of registered face images.

[0022] Further, in the fifth invention, the processor causes the terminal device to display a screen including an operation unit for changing the order of information for each monitoring target person, and in response to a user operation on the operation unit, causes the terminal device to display a screen in which the order of information for each target person displayed in a list has been rearranged.

[0023] According to this, the user can rearrange and view the information for each target person displayed in a list by operating an operation unit (for example, a sort button) for changing the order of information for each monitoring target person.

[0024] Further, in the sixth invention, the processor is configured to cause the terminal device to display a screen including the number of correct and incorrect notifications for each registered face image.

[0025] According to this, the user can easily confirm the correctness / incorrectness status of the notification based on the result of the face verification process performed for each registered face image.

[0026] Further, in the seventh invention, the processor is configured to set a category for each monitoring target person in response to a user operation of selecting the category to which the monitoring target person belongs from among a plurality of preset categories.

[0027] According to this, by allowing the user to specify a category (for example, most important, important, attention) for the person under surveillance according to the characteristics of the person under surveillance, the person under surveillance can be easily identified.

[0028] Also, in the eighth invention, the processor sets the reporting condition for each person under surveillance according to the operation of the user who designates the reporting condition used when determining whether to report based on the result of the face matching process, and determines whether to report based on the result of the face matching process and the reporting condition.

[0029] According to this, the user can specify the reporting condition for each person under surveillance and cause the reporting to be performed according to the characteristics of the person under surveillance.

[0030] Also, in the ninth invention, the processor sets, as the reporting condition, a threshold value for reporting determination regarding the matching score representing the similarity between the face image of the visitor and the face image of the person under surveillance obtained in the face matching process.

[0031] According to this, the user can specify the threshold value for reporting determination (alarm sensitivity) as a value different from the threshold value for face matching used in the face matching process and cause the reporting to be performed according to the characteristics of the person under surveillance.

[0032] Also, in the tenth invention, the processor sets the expiration date for each person under surveillance according to the operation of the user who designates the expiration date when regular surveillance is to be carried out.

[0033] According to this, the user can specify the expiration date when regular surveillance is to be carried out.

[0034] Also, in the eleventh invention, the processor sets the date and time of the expiration date and the content of the action when the expiration date has passed according to the operation of the user who designates the date and time of the expiration date and the content of the action when the expiration date has passed.

[0035] According to this, the user can specify the expiration date and time and the content of the action when the expiration date has passed. In this case, as the action when the expiration date has passed, for example, either exclusion from the reporting target or exclusion from the monitoring target may be selected by the user.

[0036] Further, in the twelfth invention, when the summary information of the reporting history of any monitored person corresponds to a predetermined attention - calling level, the processor causes the terminal device to display a screen on which the summary information of the reporting history of the monitored person is highlighted.

[0037] According to this, the user can easily grasp the monitored persons to be noted. In this case, for example, when the reporting elapsed period is equal to or longer than a predetermined threshold defined by the operation rule, the characters representing the reporting elapsed period and its background may be highlighted.

[0038] Further, the thirteenth invention is a monitoring method in which a processor executes a process of performing a report regarding detection of a monitored person as a reporting event based on an image of a visitor to a facility, accumulates reporting history information regarding reports made in the past for each monitored person in a reporting history storage unit, executes a face verification process of comparing the face image of the visitor with the registered face image of the monitored person to determine whether they are the same person, performs a report regarding detection of the monitored person based on the result, and generates summary information of the reporting history based on the reporting history information for each monitored person in response to an operation of a user requesting browsing, and causes the terminal device to display a screen in which the summary information of the reporting history for each monitored person is added to the registration information for each monitored person.

[0039] According to this, similar to the first invention, the user can quickly and appropriately make a judgment for maintaining the registration information of the monitored person, for example, a judgment on the necessity of exclusion from the monitoring target or the reporting target for each monitored person, with reference to the summary information of the reporting history representing the summary of the past reporting implementation status for each monitored person, so that the occurrence of false reports can be suppressed.

[0040] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0041] (First Embodiment) FIG. 1 is an overall configuration diagram of a monitoring system according to the first embodiment.

[0042] This system detects the arrival of a person to be monitored (a person registered as a monitoring target) as a reporting event for crime prevention in stores such as supermarkets and shopping malls, and reports it to store managers, store clerks, security guards, etc. The person to be monitored is a person who has taken problematic actions (such as theft, violence, drinking, shoplifting, customer harassment, etc.) in the store in the past, and is registered as a monitoring target when discovered. Note that the facilities to which this system is applied are not limited to stores.

[0043] This system includes a face recognition camera 1, an in-store monitoring camera 2, a control terminal 3, a monitoring terminal 4 (terminal device, notification device), an alarm lamp 5, and a staff terminal 6. The face recognition camera 1, the in-store monitoring camera 2, the control terminal 3, the monitoring terminal 4, the alarm lamp 5, and the staff terminal 6 are installed in the store. The face recognition camera 1, the in-store monitoring camera 2, the control terminal 3, the monitoring terminal 4, the alarm lamp 5, and the staff terminal 6 are interconnected via a LAN (local network in the store) and are also connected to a wide area network such as the Internet via a router.

[0044] Also, this system includes a recording server 11, a monitoring server 12, a face authentication server 13, a user management server 14, and a personal information storage server 15. The recording server 11, the monitoring server 12, the face authentication server 13, the user management server 14, and the personal information storage server 15 are configured in the cloud. The recording server 11, the monitoring server 12, the face authentication server 13, the user management server 14, and the personal information storage server 15 are connected to a wide area network such as the Internet.

[0045] The face recognition camera 1 is installed at the entrance of the store and captures images of customers (visitors to the facility). Note that the face recognition camera 1 may be installed at an appropriate location inside the store to capture images of people staying inside the store.

[0046] The in-store surveillance camera 2 is installed at an appropriate location inside the store and captures images of people staying inside the store.

[0047] The control terminal 3 controls the face recognition camera 1, the in-store surveillance camera 2, and the warning light 5. Also, the control terminal 3 extracts the face image and full-body image of the customer from the video captured by the face recognition camera 1. Here, the control terminal 3 is described as controlling the face recognition camera 1, the in-store surveillance camera 2, and the warning light 5 as the devices to be controlled, but the devices to be controlled are not limited to these. For example, as the device to be controlled, only the face recognition camera 1 may be controlled, and the in-store surveillance camera 2 and the warning light 5 may be controlled by other control devices (not shown). Also, as the device to be controlled, the face recognition camera 1 and the in-store surveillance camera 2 may be controlled, and the warning light 5 may be controlled by other control devices (not shown).

[0048] The monitoring terminal 4 is composed of a PC. A monitoring application is installed on the monitoring terminal 4. A monitor or store manager operates the monitoring terminal 4 to view the screen displayed on the monitoring terminal 4. The monitoring terminal 4 displays the videos captured by the face recognition camera 1 and the in-store surveillance camera 2. Note that on the monitoring terminal 4, in addition to the real-time video being captured by the face recognition camera 1 and the in-store surveillance camera 2, the past videos stored in the recording server 11 are also displayed. Also, the monitoring terminal 4 displays the alarm screen 401 based on the alarm information transmitted from the monitoring server 12. Thereby, the occurrence of the alarm event, that is, the visit of the person to be monitored, can be notified to the monitor or store manager. Note that when displaying the real-time video on the monitoring terminal 4, it is possible to directly acquire and display the videos captured by the face recognition camera 1 and the in-store surveillance camera 2, but the same videos can also be displayed via the recording server 11.

[0049] In addition, the monitoring terminal 4 has the function as a management terminal. That is, in the monitoring terminal 4, the user (monitor or store manager) can perform an operation of registering a face image and other information regarding the person to be monitored. Also, in the monitoring terminal 4, based on the past reporting execution status, the user can perform a setting operation of excluding from the monitoring target or temporarily excluding from the reporting target without excluding from the monitoring target.

[0050] The warning lamp 5 performs the lighting of the lamp and the output of the warning sound as a reporting operation in response to an instruction of the reporting operation from the monitoring server 12. Thereby, the occurrence of the reporting event, that is, the arrival of the person to be monitored at the store, can be notified to the monitor or the store manager.

[0051] The staff terminal 6 is composed of a mobile terminal such as a smartphone or a tablet terminal. The staff such as the store manager and the store clerk hold the staff terminal 6 and operate the staff terminal 6 to view the screen displayed on the staff terminal 6. The staff terminal 6 receives an e-mail transmitted from the monitoring server 12 and displays the reporting information included in the e-mail. Thereby, the occurrence of the reporting event, that is, the arrival of the person to be monitored at the store, can be notified to the staff such as the store manager and the store clerk. Also, the staff terminal 6 acquires and displays detailed reporting information including the face image of the person to be monitored who is the party of the reporting event from the monitoring server 12 based on the link information included in the e-mail.

[0052] The recording server 11 stores the videos captured by the face verification camera 1 and the in-store monitoring camera 2.

[0053] The monitoring server 12 acquires the face image of the customer from the control terminal 3, requests the face verification server 13 to perform a face verification to determine whether the customer is a person to be monitored (a person registered as a monitoring target), and when detecting the arrival of the person to be monitored based on the face verification result obtained from the face verification server 13, performs a reporting to notify the store manager and the store clerk to that effect.

[0054] In addition, the monitoring server 12 stores information other than the face images among the information of the persons to be monitored. Specifically, information such as the person ID, person name (common name), gender, and category for each person to be monitored is stored. These pieces of information are presented to the user as a target person list together with the face images. Also, the monitoring server 12 stores information associated with the face images of the visitors. Specifically, information such as the person ID, camera ID, store visit date and time, and registration date and time for each visitor is stored. Further, the monitoring server 12 stores information (reporting history information) regarding the reports made in the past for each person to be monitored.

[0055] In response to a face comparison request from the monitoring server 12, the face authentication server 13 performs a face comparison to determine whether a visitor is a person to be monitored by comparing the face image of the visitor with the face image of the person to be monitored. Also, the face authentication server 13 (registered image storage unit) stores the face images of the persons to be monitored. Here, the face authentication server 13 usually sets a comparison group using the face images of authorized target persons and executes face authentication processing, such as entry / exit management by face authentication and settlement by face authentication. However, by newly setting a comparison group for the face images of the persons to be monitored and executing face authentication processing, the reporting function for the persons to be monitored can be realized.

[0056] The user management server 14 holds user information. The user management server 14 cooperates with the monitoring server 12 to manage the login of users (monitors) on the monitoring terminal 4. Specifically, when a user logs in on the monitoring terminal 4, when the monitoring terminal 4 accesses the user management server 14, user authentication is performed on the user management server 14, and if the user authentication is successful, the login is permitted.

[0057] The personal information storage server 15 stores the personal information of the visitors. Specifically, the personal information storage server 15 (visitor image storage unit) acquires and stores the face images and full-body images of the visitors from the control terminal 3 via the monitoring server 12. Also, the personal information storage server 15 acquires and stores the face images and full-body images regarding the persons to be monitored for whom reports have been made from the monitoring server 12.

[0058] Incidentally, in this system, when a person to be monitored, that is, a person who has taken problematic actions in the store, is discovered by a person engaged in the work (such as a store clerk or a security guard), the face image of that person is registered in the database of the face authentication server 13, and information other than the face image regarding that person is registered in the database of the monitoring terminal 4 (new registration). In this case, the user can register the face image of that person by finding a captured image in which that person appears from the recorded videos of the face camera 1 for face verification or the in-store monitoring camera 2, and cutting out and creating a face image from the captured image using an image editing application or the like. Also, the face image of that person can be registered by searching for the face image of that person among the face images of the customers stored in the personal information storage server 15. At this time, the user can specify search conditions (recording date and time and recording location) based on the date and time and location when that person was discovered, and search for face images corresponding to the search conditions among the face images of the customers stored in the personal information storage server 15, and the face images presented to the user are narrowed down.

[0059] Also, for a person already registered as a monitoring target, the user can additionally register a face image when there are few registered face images or when no appropriate face image is registered (additional registration). In particular, in the case of a person whose degree and frequency of problematic actions make it desirable not to miss them as much as possible, by additionally registering a face image and increasing the variations of the face images, that person can be reliably detected. In this case, the user can specify an appropriate face image among the already registered face images of that person as a search source, and face images similar to the search source face image are searched among the face images of the customers stored in the personal information storage server 15, and the corresponding face images are presented to the user.

[0060] Next, the schematic configurations of the monitoring server 12 and the monitoring terminal 4 will be described. FIG. 2 is a block diagram showing the schematic configuration of the monitoring server 12. FIG. 3 is a block diagram showing the schematic configuration of the monitoring terminal 4. FIG. 4 is a block diagram showing the outline of the process related to customer monitoring. FIG. 5 is a block diagram showing the outline of the process related to face image registration. FIG. 6 is a block diagram showing the outline of the process related to target person management. FIG. 7 is a block diagram showing the outline of the process related to confirmation of notification history.

[0061] As shown in FIG. 2, the monitoring server 12 includes a communication unit 31, a storage unit 32, and a processor 33.

[0062] The communication unit 31 communicates with the control terminal 3, the monitoring terminal 4, the recording server 11, the face authentication server 13, the user management server 14, and the personal information storage server 15.

[0063] The storage unit 32 stores programs executed by the processor 33 and the like. Further, the storage unit 32 stores information other than the face image among the information of the monitoring target persons, specifically, information such as person ID, person name (common name), gender, and category for each of the monitoring target persons. In addition, the storage unit 32 stores information associated with the face images of the customers, specifically, information such as person ID, camera ID, store visit date and time, and registration date and time for each of the customers. Further, the storage unit 32 stores information related to notifications made in the past for each monitoring target person (notification history information).

[0064] The processor 33 performs various processes by executing the programs stored in the storage unit 32. In the present embodiment, the processor 33 performs processes such as customer monitoring process, face search process, notification history management process, and target person management process.

[0065] The customer monitoring process includes a notification event detection process, a notification determination process, and a notification control process.

[0066] As shown in FIG. 4, in the reporting event detection process, the processor 33 detects the arrival of the monitored person, which is the target reporting event. Specifically, the face image of the arriving person acquired from the control terminal 3 is provided to the face authentication server 13, and the face authentication server 13 is made to perform a face matching process of comparing the face image of the arriving person with the face image of the monitored person. As the face matching result, the success or failure of the face matching, and when the face matching is successful, the ID of the monitored person determined to be the same person as the arriving person and the matching score are acquired. The matching score represents the similarity between the face image of the arriving person and the face image of the monitored person.

[0067] In the reporting determination process, when the processor 33 detects a reporting event (the arrival of the monitored person) by the reporting event detection process, the processor 33 determines whether or not to issue a report according to whether or not the reporting event satisfies a predetermined reporting condition. Specifically, it is determined whether or not the matching score acquired as the face matching result is equal to or higher than the alarm sensitivity (threshold value for reporting determination). At this time, the processor 33 acquires the alarm sensitivity associated with the ID of the monitored person as the face matching result acquired from the face authentication server 13 and makes a determination.

[0068] In the reporting control process, when the detected reporting event satisfies the reporting condition, that is, when the matching score is equal to or higher than the alarm sensitivity, the processor 33 performs predetermined reporting control on the monitoring terminal 4, the warning lamp 5, and the staff terminal 6. Specifically, the display information of the reporting screen 401 (see FIG. 15) is transmitted to the monitoring terminal 4. Also, an instruction to turn on the lamp and output a warning sound as a reporting operation is transmitted to the warning lamp 5. Also, an e-mail including the reporting information is transmitted to the staff terminal 6.

[0069] As shown in FIG. 2, the face search process includes an image search process, a face state detection process, and a search result presentation process. In the face search process, a face search screen 101 (see FIG. 11) is displayed on the display 42 of the monitoring terminal 4.

[0070] As shown in FIG. 5, in the image search process, the processor 33 acquires the search conditions (shooting date and time, and shooting location) specified by the user on the face search screen 101 (see FIG. 11), and requests the personal information storage server 15 to provide a face image corresponding to the search conditions (shooting date and time, and shooting location) from among the face images of past visitors stored in the personal information storage server 15.

[0071] Also, in the image search process, when the user designates a source face image on the face search screen 101, the processor 33 causes the face authentication server 13 to execute a face matching process, and selects a face image similar to the source face image from among the face images of past visitors corresponding to the search conditions (shooting date and time, and shooting location). In the face matching process of the face authentication server 13, the face image of a past visitor corresponding to the search conditions is compared with the source face image, and a matching score representing the degree of similarity between the two is obtained. In the image search process, a face image whose matching score is equal to or higher than a predetermined threshold for face search is selected as the search result, that is, a face image similar to the source face image. Note that the threshold for face search may be set lower than the threshold for face verification.

[0072] In the face state detection process, the processor 33 detects a characteristic state of the face as face reference information that clarifies the differences from the source face image for each face image of the search result. Specifically, characteristic states regarding accessories (such as glasses, hats) and the orientation of the face (such as sideways) are detected. Note that information regarding the differences (information regarding the relationship) between the face image of the search result and the source face image may be detected as face reference information by comparing the two.

[0073] In the search result presentation process, the processor 33 displays the face image selected as the search result in the image search process on the face search screen 101 (see FIG. 11). Also, in the search result presentation process, the processor 33 displays the face reference information representing the characteristic state of the face detected in the face state detection process on the face search screen 101. Further, in the search result presentation process, the processor 33 displays the matching score obtained in the face matching process performed by the face authentication server 13 on the face search screen 101 as the degree of similarity between the source face image and the image of the search result.

[0074] As shown in FIG. 6, in the transmission history management process, each time a transmission corresponding to a transmission event (the arrival of a monitored person at the store) occurs in the transmission control process, the processor 33 accumulates information regarding the transmission event in the storage unit 32 as transmission history information.

[0075] Also, as shown in FIG. 7, in the transmission history management process, the processor 33 acquires accumulated transmission information (identification information of a transmission event in the accumulated state) from the monitoring terminal 4, and based on the accumulated transmission information, acquires transmission history information regarding the transmission event in the accumulated state from the storage unit 32, and provides the transmission history information to the monitoring terminal 4.

[0076] As shown in FIG. 6, in the target person management process, the processor 33 acquires registration information regarding each monitored person based on the content operated by the user on the target person list screen 201 (see FIG. 13) and the target person registration / editing screen 301 (see FIG. 14) displayed on the monitoring terminal 4, and stores the information in the storage unit 32. Also, in the target person management process, the processor 33 provides the registration information regarding each monitored person stored in the storage unit 32 to the monitoring terminal 4, and causes the monitoring terminal 4 to display the target person list screen 201 (see FIG. 13) and the target person registration / editing screen 301 (see FIG. 14).

[0077] As shown in FIG. 3, the monitoring terminal 4 includes a communication unit 41, a display 42, an input device 43, a storage unit 44, and a processor 45.

[0078] The communication unit 41 communicates with the recording server 11, the monitoring server 12, and the user management server 14.

[0079] The display 42 displays various screens. The input device 43 is for the user to perform operations on the screens displayed on the display 42.

[0080] The storage unit 44 stores programs executed by the processor 45 and the like.

[0081] The processor 45 performs various processes by executing the programs stored in the storage unit 44. In the present embodiment, the processor 45 performs screen control processing, acquisition processing of transmitted report accumulation information, and the like.

[0082] In the screen control processing, the processor 45 generates display information regarding various screens based on the information provided from the monitoring server 12, and displays the screen on the display 42 based on the display information. Also, in the screen control processing, the processor 45 acquires operation input information in response to an operation of the input device 43 by the user, and executes a predetermined process based on the operation input information.

[0083] Specifically, as shown in FIG. 4, when monitoring visitors, the transmitted report screen 401 (see FIG. 15) is displayed on the display 42. Also, as shown in FIG. 5, when registering a face image, the face search screen 101 (see FIGS. 11 and 12) is displayed on the display 42. Also, as shown in FIG. 6, when managing target persons, the target person list screen 201 (see FIG. 13) and the target person registration / editing screen 301 (see FIG. 14) are displayed on the display 42. Also, as shown in FIG. 7, when confirming the transmitted report history, the transmitted report accumulation details screen 601 (see FIG. 16) and the transmitted report history list screen 701 (see FIG. 17) are displayed on the display 42 after the transmitted report screen 401 (see FIG. 15).

[0084] As shown in FIG. 7, in the acquisition processing of transmitted report accumulation information, the processor 45 acquires information (transmitted report accumulation information) regarding transmitted report events in an accumulation state that the user has not confirmed. Specifically, the total number of transmitted report events in an accumulation state that the user has not confirmed (total transmitted report accumulation number) and the number of monitored target persons related to the transmitted report events (number of target persons) are obtained. Also, the number of transmitted report events for each monitored target person in an accumulation state that the user has not confirmed (individual transmitted report accumulation number) is obtained.

[0085] Note that in the present embodiment, the acquisition processing of transmitted report accumulation information is performed by the monitoring terminal 4, but the acquisition processing of transmitted report accumulation information may be performed by the monitoring server 12.

[0086] Next, an overview of the processing during customer monitoring performed by this system will be described. FIG. 8 is a sequence diagram showing an overview of the processing at the time of notification.

[0087] The face recognition camera 1 captures a person (customer) entering the store from the store entrance and transmits the video to the control terminal 3. The control terminal 3 extracts the face image and full body image of the customer from the video received from the face recognition camera 1, and transmits the face image and full body image of the customer to the monitoring server 12.

[0088] The monitoring server 12 transmits the face image and full body image of the customer received from the control terminal 3 to the personal information storage server 15. The personal information storage server 15 accumulates the face image and full body image of the customer received from the monitoring server 12, and transmits a response of image accumulation to the monitoring server 12. The monitoring server 12 stores the accompanying information (such as the shooting date and time) of the face image in its own database.

[0089] The monitoring server 12 transmits a face verification request including the face image of the customer to the face authentication server 13. The face authentication server 13 performs a face verification to compare the face image of the customer with the face image of the person to be monitored in response to the face verification request, and determines whether the customer is the person to be monitored, and transmits a face verification response including the face verification result to the monitoring server 12.

[0090] The monitoring server 12 determines whether to issue a notification regarding the arrival of the person to be monitored based on the face verification result included in the face verification response received from the face authentication server 13. Here, if it is determined to issue a notification, the monitoring server 12 performs predetermined notification control on the monitoring terminal 4, the warning light 5, and the staff terminal 6. Specifically, the monitoring server 12 transmits the display information of the notification screen 401 to the monitoring terminal 4. Also, the monitoring server 12 transmits an instruction to turn on the lamp and output a warning sound as a notification operation to the warning light 5. Also, the monitoring server 12 transmits an email including the notification information to the staff terminal 6.

[0091] Next, the processing during visitor monitoring performed by the monitoring server 12 will be described. FIG. 9 is a flowchart showing the procedure of the processing during visitor monitoring performed by the monitoring server 12.

[0092] In the monitoring server 12, first, a face image of a visitor transmitted from the control terminal 3 is received (ST101). At this time, in the control terminal 3, processing for extracting the face image of the visitor from the captured image output from the face photographing camera 1 is performed.

[0093] Next, in the monitoring server 12, based on the control of the processor 33, the communication unit 31 transmits a face verification request to the face authentication server 13 (ST102). The face verification request includes the face image of the visitor.

[0094] At this time, in response to the face verification request from the monitoring server 12, the face authentication server 13 compares the face image of the visitor with the face image of the person to be monitored, obtains a verification score representing the similarity between the two, compares the verification score with a predetermined threshold for face verification, and determines whether the face verification is successful (face verification process). Specifically, when the verification score is equal to or higher than the threshold, it is determined that the face verification is successful. Next, the face authentication server 13 transmits a response to the face verification to the monitoring server 12. The response to the face verification includes whether the face verification is successful as the face verification result. Also, when the face verification is successful, the response to the face verification includes the ID of the person to be monitored determined to be the same person as the visitor and the verification score.

[0095] Next, in the monitoring server 12, the communication unit 31 receives the response to the face verification from the face authentication server 13 (ST103). Next, in the monitoring server 12, the processor 33 determines whether the face verification is successful based on the face verification result included in the response to the face verification (ST104).

[0096] Here, when face verification is successful (Yes in ST104), next, the processor 33 determines whether the reporting condition is satisfied, specifically, whether the verification score is greater than or equal to the alarm sensitivity (threshold value for reporting determination) (reporting determination process) (ST105). At this time, the processor 33 obtains the alarm sensitivity associated with the ID of the person under surveillance included in the response of the face verification and makes the determination.

[0097] Here, when the reporting condition is satisfied, that is, when the verification score is greater than or equal to the alarm sensitivity (Yes in ST105), next, the processor 33 performs a predetermined reporting control process on the monitoring terminal 4, the warning lamp 5, and the staff terminal 6 (ST106). At this time, for example, on the monitoring terminal 4, a reporting screen 401 (see FIG. 15) is displayed as a reporting operation. Also, on the reporting screen 401, the verification score included in the response of the face verification is displayed as the similarity between the face image of the person under surveillance determined to be the same person in the face verification and the face image of the visitor. Note that when reported to the staff terminal 6, the reporting screen 401 can be displayed by an operation of instructing the link displayed in the received email.

[0098] On the other hand, when face verification fails, that is, when the verification score is less than the threshold value for face verification (No in ST104), or when the reporting condition is not satisfied, that is, when the verification score is less than the alarm sensitivity (threshold value for reporting determination) (No in ST105), the processor 33 performs a process of discarding the face verification result (ST107).

[0099] Incidentally, the threshold value for face verification is set to a uniform value and the face verification process is performed. On the other hand, the alarm sensitivity (threshold value for reporting determination) is set individually for each person under surveillance according to the user's designation. For example, the alarm sensitivity is set low for a person who should not be missed as much as possible. Thereby, although the possibility of false alarms is high, reporting is surely carried out. Also, the alarm sensitivity is set high for a person with significant false alarms. Thereby, false alarms can be suppressed.

[0100] In this embodiment, categories (e.g., most important, important, attention) are individually assigned to the persons to be monitored. Therefore, the alarm sensitivity for each person to be monitored may be set according to the category to which the person to be monitored belongs, regardless of the user's designation. For example, the alarm sensitivity may be set low for a person whose category is most important, and may be set high for a person whose category is attention.

[0101] Next, the face search process performed by the monitoring server 12 will be described. FIG. 10 is a flowchart showing the procedure of the face search process.

[0102] In the monitoring server 12, first, in response to an operation in which the user designates a source face image and other search conditions (shooting date and time, and shooting location) on the monitoring terminal 4, the processor 33 acquires the source face image and other search conditions (ST201). Specifically, a face search screen 101 (see FIG. 11) is displayed on the monitoring terminal 4, and the user performs an operation of designating a source face image and other search conditions on this face search screen 101.

[0103] Next, in the monitoring server 12, based on other search conditions (shooting date and time, and shooting location), the processor 33 acquires, from the personal information storage server 15, face images of past visitors stored in the personal information storage server 15 that match the search conditions (shooting date and time, and shooting location) (ST202). At this time, based on the control of the processor 33, the communication unit 31 transmits a request for a face image to the personal information storage server 15, and receives, from the personal information storage server 15, a face image that matches the search conditions as a response to the face image request.

[0104] Next, in the monitoring server 12, based on the control of the processor 33, a similarity determination loop is started (ST203), and the communication unit 31 transmits a face verification request to the face authentication server 13 (ST204). The face verification request includes the source face image designated by the user and face images of past visitors that match the search conditions (shooting date and time, and shooting location).

[0105] At this time, in response to the face matching request from the monitoring server 12, the face authentication server 13 compares the source face image with the face images of past visitors to obtain a matching score representing the similarity between the two (face matching process). Next, the face authentication server 13 sends a response of the face matching to the monitoring server 12. The response of the face matching includes the similarity (matching score) regarding each face image that is the matching result.

[0106] Next, in the monitoring server 12, the communication unit 31 receives the response of the face matching sent from the face authentication server 13 (ST205).

[0107] Next, in the monitoring server 12, the processor 33 determines whether the similarity (matching score) regarding each face image included in the response of the face matching is equal to or greater than a predetermined threshold for face search (ST206).

[0108] Here, if the similarity of the face image to be determined is less than the threshold for face search (No in ST206), the processor 33 performs a process of discarding the face matching result of the face image (ST208).

[0109] On the other hand, if the similarity of the face image to be determined is equal to or greater than the threshold for face search (Yes in ST206), the processor 33 sets the face image to be determined as a face image (registered candidate face image) to be presented as a search result (ST207).

[0110] When the process ends for all the face images included in the response of the face matching, the similarity determination loop ends (ST209). Next, the processor 33 presents the face images (registered candidate face images) of the search results to the user (ST210). At this time, the matching score of each face image included in the response of the face matching is presented to the user as the similarity with the source face image specified by the user. Specifically, on the face search screen 101 (see FIG. 11) displayed on the monitoring terminal 4, the face images of the search results are displayed together with the similarity.

[0111] Also, in this embodiment, the number of face images (registered candidate face images) presented to the user as search results is limited to the upper limit value specified by the user. When the number of face images with a similarity equal to or higher than the threshold exceeds the upper limit value, face images are selected in order from the highest similarity until the number of face images reaches the upper limit value.

[0112] Note that in the example shown in FIG. 10, a similar image search is performed with a specified source face image, but the face image search process may be performed based only on other search conditions (shooting date and shooting location) without specifying the source face image.

[0113] Next, the face search screen 101 displayed on the monitoring terminal 4 will be described. FIGS. 11 and 12 are explanatory diagrams showing the face search screen 101.

[0114] When the user newly registers a face image of a person to be monitored or adds a face image different from the registered face image, the user can find an appropriate face image by inputting search conditions on the face search screen 101 to execute a face image search process and viewing the search results.

[0115] The face search screen 101 is provided with a tab 102 for "Alarm", a tab 103 for "Face Search", a tab 104 for "Target List", a tab 105 for "Device Monitoring", and a tab 106 for "Settings". The face search screen 101 is displayed when the user operates the tab 103 for "Face Search". When the user operates the tab 102 for "Alarm", the user is transferred to the alarm history list screen 701 (see FIG. 17). When the user operates the tab 104 for "Target List", the user is transferred to the target list screen 201 (see FIG. 13). When the user operates the tab 105 for "Device Monitoring", the user is transferred to a device monitoring screen (not shown). When the user operates the tab 106 for "Settings", the user is transferred to a screen (not shown) related to system settings, operation settings, and user settings.

[0116] In addition, on the face search screen 101, a display box 107 for "store name" and a display box 108 for "user name" are provided. In the display box 107 for "store name", the name of the target store is displayed. Also, in the display box 107 for "store name", the user can switch the target store by operating the pull-down menu. In the display box 108 for "user name", the user name of the logged-in user is displayed. Also, in the display box 108 for "user name", the user can switch the logged-in user by instructing the execution of logout by operating the pull-down menu.

[0117] In addition, on the face search screen 101, a search condition setting section 111, a "search" button 112, and a search result display section 113 are provided. In the search condition setting section 111, the user can specify search conditions. When the user specifies search conditions in the search condition setting section 111 and operates the "search" button 112, the search results are displayed in the search result display section 113. In this embodiment, face images (registered candidate face images) that meet the search conditions among the face images of visitors stored in the personal information storage server 15 are displayed as search results. Also, when the user designates a source face image (search face image), a similar image search is performed, and face images similar to the source face image are displayed as search results.

[0118] The search condition setting section 111 is provided with a search target period specifying section 121, a source face image specifying section 122, a search target camera specifying section 123, and a search result display number specifying section 124.

[0119] In the search target period specifying section 121, the user can specify the search target period (start date and time, end date and time). The search target period is set based on the date and time when the visit of the person of interest is predicted. In the face image search process, face images whose shooting date and time are included in the search target period are selected as search results.

[0120] In the search target period specifying section 121, the user can input the start date and time into the input box 132 by turning on (checking) the "start" checkbox 131. Also, the user can input the end date and time into the input box 134 by turning on the "end" checkbox 133. If both the "start" checkbox 131 and the "end" checkbox 133 are off (unchecked), the current date and time is set as the end date and time, and one hour before the current date and time is set as the start date and time. Also, if the "start" checkbox 131 is off and the "end" checkbox 133 is on, one hour before the input end date and time is set as the start date and time. If the "start" checkbox 131 is on and the "end" checkbox 133 is off, one hour after the input start date and time is set as the end date and time.

[0121] In the search source face image specifying section 122, the user can specify the search source face image when performing a similar image search. Also, in the search source face image specifying section 122, the user can specify a threshold value regarding similarity for narrowing down the search results. The search source face image specifying section 122 is provided with two face image display frames 135, a similarity input box 136, and a "clear" button 137.

[0122] The search source face image specified by the user is displayed in the face image display frame 135. When the user operates the face image display frame 135, an operation menu (not shown) is displayed. When the user selects the "file selection" item in the operation menu, a file selection dialog (not shown) is displayed. In this file selection dialog, the user can select a face image created and saved in advance as the search source face image. Also, when the user selects the "select from target person list" item in the operation menu, the user is redirected to the target person list screen 201 (see FIG. 13). When the user selects a person under surveillance here, the representative face image related to the selected person under surveillance is set as the search source face image. Also, when the user selects the "clear" item in the operation menu, the set face image is deleted.

[0123] In addition, when newly registering a face image of a person to be monitored, a face image acquired by the user from a video or the like at the date and time when the person to be monitored was discovered may be set as the reference face image for search. Also, when adding and registering a face image of a person to be monitored, a registered face image, particularly a representative face image, may be set as the reference face image for search.

[0124] In the similarity input box 136, the user can specify a similarity threshold value used when selecting a face image from the search results. In the similar image search, a face image with a similarity to the reference face image equal to or higher than the threshold value is selected as the search result. Note that an initial value of the similarity threshold is set in advance, and when the user does not input, the initial value is displayed in the input box 346.

[0125] When the user operates the "Clear" button 137, the set reference face image is deleted and the similarity is initialized.

[0126] Note that the image search process may be performed without specifying a reference face image in the reference face image specifying unit 122. In this case, the similar image search is not performed, and a face image corresponding to the search conditions regarding the date and time and the location, specifically, the search conditions specified by the search target period specifying unit 121 and the search target camera specifying unit 123 is searched.

[0127] The search target camera specifying unit 123 is provided with a check box 138 for each face detection camera 1. The user can specify whether or not to target each face detection camera 1 for search by operating the check box 138. By specifying the face detection camera 1, the user can limit the face images displayed as the search results by the shooting location. The user specifies the face detection camera 1 based on the location where the appearance of the person of interest is predicted.

[0128] The search result display number specifying section 124 is provided with an input box 139 for the "search result display number". The user can input the upper limit value of the number of face images to be displayed as search results in the input box 139 in the search result display section 113. When the upper limit value of the number of face images is set, face images with high similarity within the range of the upper limit value are selected and displayed in the search result display section 113.

[0129] The search result display section 113 is provided with a tab 141 for "time / location" and a tab 142 for "list". The example shown in FIG. 11 is the case of the first display mode displayed when the user operates the tab 141 for "time / location". The example shown in FIG. 12 is the case of the second display mode displayed when the user operates the tab 142 for "list".

[0130] As shown in FIG. 11, in the case of the first display mode, the face images of the search results are arranged side by side according to the shooting time and shooting location. Specifically, the face images of the search results are separated and arranged for each face recognition camera 1 corresponding to the shooting location. Also, the face images taken by the same face recognition camera 1 are arranged side by side in the horizontal direction in chronological order according to the shooting time.

[0131] On the other hand, as shown in FIG. 12, in the case of the second display mode, the face images of the search results are arranged in descending order of similarity. Also, when the face image of the search source is not set as a search condition, that is, when similar image search is not performed, the face images of the search results are arranged in chronological order according to the shooting time.

[0132] Note that among the face images presented to the user as search results, there are those in which a visitor is detected as the same person as the monitored person by face recognition during store visitor monitoring and a report is issued, and there are also those in which the visitor is not detected as the same person as the monitored person.

[0133] The search result display unit 113 is provided with a check box 143 for "displaying a full-body image". The example shown in FIG. 11 is the case where the check box 143 for "displaying a full-body image" is off (unchecked). The example shown in FIG. 12 is the case where the check box 143 for "displaying a full-body image" is on (checked).

[0134] As shown in FIG. 11, when the check box 143 for "displaying a full-body image" is off, only the face image of the search result is displayed on the search result display unit 113. Also, as shown in FIG. 12, when the check box 143 for "displaying a full-body image" is on, the face image of the search result and the full-body image of the same person as that face image are displayed on the search result display unit 113. Note that the full-body image may be an image that includes the entire body of a person, or may be an image that includes a body part above the upper body. Since the user can confirm that it is the person being monitored from the appearance such as the clothes shown in the full-body image, the user can efficiently select the face image of the person being monitored with reference to the full-body image.

[0135] The search result display unit 113 is provided with an image display size adjustment unit 144. In the image display size adjustment unit 144, when the user operates the "small" button, the search result image (face image, full-body image) is displayed enlarged by one step. When the user operates the "large" button, the search result image is displayed reduced by one step. When the user operates the slider of the slide bar, the search result image is enlarged or reduced step by step and displayed.

[0136] The search result display unit 113 is provided with a menu button 151 for each detected face image. When the user operates the menu button 151, an operation menu (not shown) is displayed.

[0137] When the user selects the item "Set for face search" in the operation menu, the face image corresponding to the operated menu button 151 is set as the search source face image in a state where it overwrites the set search source face image displayed in the face image display frame 135. This operation can be performed individually for each of the set search source face images displayed in each of the two face image display frames 135.

[0138] Also, when the user selects the item "New registration" in "Register to target person list" in the operation menu, the target person registration editing screen 301 (see FIG. 14) is displayed. On the target person registration editing screen 301, the face image corresponding to the operated menu button 151 is displayed first in the registered face image list display section 323 and is also displayed as the representative face image. Also, when the user selects the item "Additional registration" in "Register to target person list" in the operation menu, the target person list screen 201 (see FIG. 13) is displayed. Then, when the user selects a monitoring target person on the target person list screen 201, the target person registration editing screen 301 is displayed. On the target person registration editing screen 301, the face image corresponding to the operated menu button 151 is displayed in a state where it is added at the end of the registered face image list display section 323. Note that on the target person registration editing screen 301, when the user operates the "Confirm" button 302, the face image corresponding to the operated menu button 151 is registered in association with the monitoring target person.

[0139] Also, when the user selects the item "Playback from shooting time" in the operation menu, the video monitoring screen 501 (see FIG. 15) is displayed. On the video monitoring screen 501, the video of the face camera 1 including the shooting image that is the source of the face image corresponding to the operated menu button 151 is played back from the shooting date and time. Also, when the user selects the item "Display in original size" in the operation menu, the face image corresponding to the operated menu button 151 is pop-up displayed in the original size.

[0140] In addition, in the search result display section 113, for each face image in the search results, a shooting date and time display column 152 and a similarity display column 153 are provided. In the shooting date and time display column 152, the shooting date and time of the face image in the search results are displayed. In the similarity display column 153, the similarity between the face image in the search results and the reference face image is displayed.

[0141] In addition, in the search result display section 113, a face reference information display column 154 is provided for each face image in the search results. In the face reference information display column 154, information regarding characteristic states of the face, as face reference information for clarifying differences between the face image in the search results and the reference face image, is displayed. Specifically, characters representing characteristic states regarding face accessories (such as glasses, hats, etc.) and the orientation of the face (such as sideways) are displayed. These reference information may be displayed not only as characters but also as fixed-form images such as icons.

[0142] As described above, in this embodiment, as information for reference when the user selects a face image to register from among the face images of the detection results (registration candidate face images), the similarity between the reference face image and the face images in the search results, and face reference information regarding characteristic states of the face in the face images of the search results are displayed. Thereby, the user can quickly discriminate the face image of the monitored person of interest and can quickly find a face image in a characteristic state different from the registered face images.

[0143] Next, the subject list screen 201 displayed on the monitoring terminal 4 will be described. FIG. 13 is an explanatory diagram showing the subject list screen 201.

[0144] On the subject list screen 201, a "New Registration" button 202 and an "Import" button 203 are provided. When the user operates the "New Registration" button 202, the user is transferred to the subject registration editing screen 301 (see FIG. 14). When the user operates the "Import" button 203, a subject list import screen (not shown) is displayed. On the subject list import screen, the user can instruct the reading of a subject list file stored in a folder specified by the user.

[0145] In addition, a registered user number display section 204 is provided on the subject list screen 201. The total number of registered monitored subjects is displayed on the registered user number display section 204.

[0146] In addition, a subject-specific display section 211 for displaying information for each monitored subject is arranged on the subject list screen 201.

[0147] The subject-specific display section 211 is provided with a category display column 221, a face image display column 222, a registration date and time display column 223, a person name display column 224, an age display column 225, a gender display column 226, and a remarks display column 227. A category mark and category name representing the category to which the monitored subject belongs are displayed in the category display column 221. A representative face image of the monitored subject is displayed in the face image display column 222. The date and time when the monitored subject was registered is displayed in the registration date and time display column 223. The name (common name) given to the monitored subject is displayed in the person name display column 224. The age of the monitored subject is displayed in the age display column 225. The gender (male, female, unspecified) of the monitored subject is displayed in the gender display column 226. Reference information regarding the monitored subject is displayed as remarks in the remarks display column 227.

[0148] In addition, a check box 231 for "alarm target" is provided in the subject-specific display section 211. By operating the check box 231 for "alarm target", the user can switch whether or not to target the corresponding monitored subject for notification. Specifically, when the check box 231 for "alarm target" is checked, the corresponding monitored subject is set as the notification target. In this case, when face verification is successful and the visitor is determined to be a monitored subject and the notification conditions are met, notification is carried out. On the other hand, when the check box 231 for "alarm target" is unchecked and the corresponding monitored subject is not set as the notification target, even if face verification is successful and the visitor is determined to be a monitored subject and the notification conditions are met, notification is not carried out.

[0149] In addition, the per - subject display section 211 is provided with a check box 232 for "Delete from the list". By operating the check box 232 for "Delete from the list", the user can switch whether to exclude the corresponding monitored person from the monitoring target. Specifically, when the check box 232 for "Delete from the list" is checked, the corresponding monitored person is excluded from the monitoring target.

[0150] In addition, the per - subject display section 211 is provided with a button 233 for "Go to detailed settings". When the user operates the button 233 for "Go to detailed settings", the screen transitions to the subject registration and editing screen 301 (see FIG. 14) for the corresponding monitored person.

[0151] In addition, the per - subject display section 211 is provided with a reporting elapsed period display column 241 and a reporting frequency display column 242. In the reporting elapsed period display column 241, as summary information of the reporting history, the reporting elapsed period for the monitored person, that is, the elapsed period from the last report to the present (the most recent non - reporting period) is displayed. In the reporting frequency display column 242, as summary information of the reporting history, the number of reports for the monitored person, that is, the number of times reports have been made in the past is displayed. Based on the reporting elapsed period and the reporting frequency, the user can determine whether to exclude the person registered as a monitored person from the monitoring target or whether to exclude the person registered as a monitored person from the reporting target. For example, when the reporting elapsed period is equal to or longer than a predetermined period (e.g., 1 year) defined by the operation rules, the user can determine to exclude the person from the reporting target.

[0152] In addition, in the reporting elapsed time display column 241, when the reporting elapsed time (reporting history summary information) of the monitored person corresponds to a predetermined attention - calling level, the reporting elapsed time of the monitored person is highlighted. Specifically, when the reporting elapsed time is equal to or greater than a predetermined threshold value defined by the operation rules, the reporting elapsed time is highlighted. Also, in the highlighting of the reporting elapsed time, for example, the characters representing the reporting elapsed time are displayed in different colors or thicknesses. Also, the background of the reporting elapsed time display column 241 is displayed in different colors or patterns. As a result, the user can easily identify the monitored person who should be paid attention to. Note that in the reporting count display column 242, when the reporting count (reporting history summary information) corresponds to a predetermined attention - calling level, the characters representing the reporting count or the reporting count display column 242 may be highlighted.

[0153] In addition, a registered face image count display column 243 is provided in the person - specific display section 211. In the registered face image count display column 243, the number of registered face images, that is, the number of face images registered for the monitored person, is displayed. The user can determine the necessity of additional registration of face images based on the number of registered face images.

[0154] Also, on the person - list screen 201, sort buttons 251 (operation section) for descending and ascending order are provided for each item of category, registration date and time, person name, age, gender, alarm target, reporting elapsed time, reporting count, and number of registered face images. When the user operates either the descending - order or ascending - order sort button 251, the arrangement of the person - specific display section 211 is sorted in descending or ascending order for each item.

[0155] Also, on the person - list screen 201, a button 205 for "confirm" and a button 207 for "cancel" are provided. When the user operates the button 205 for "confirm", the content input this time on the person - list screen 201 is set. When the user operates the button 207 for "cancel", the content input this time on the person - list screen 201 is discarded.

[0156] Next, the subject registration and editing screen 301 displayed on the monitoring terminal 4 will be described. FIG. 14 is an explanatory diagram showing the subject registration and editing screen 301.

[0157] The subject registration and editing screen 301 is provided with a registration method guidance section 311 and a face image registration section 312. In the registration method guidance section 311, characters guiding the first face image registration method are displayed. In the first face image registration method, a face image can be registered from an operation menu displayed by operating the menu button 151 of the face image on the face search screen 101 (see FIG. 11). Also, a face image can be registered from an operation menu displayed by operating the menu button 652 of the notification accumulation details screen 601 (see FIG. 16). In the face image registration section 312, the user can select a face image file in order to register a face image according to the second face image registration method. When the user operates the "Reference" button 313, a file selection dialog (not shown) is displayed, and a face image file can be selected in this file selection dialog.

[0158] In addition, the subject registration and editing screen 301 is provided with a representative face image display section 321 and a check box 322 for "always display on the alarm list screen". In the representative face image display section 321, a representative face image of the monitored person is displayed. The representative face image is selected by the user from the registered face images displayed in the registered face image list display section 323. When there is only one registered face image, that face image is set as the representative face image. Also, the user can select whether to display the representative face image of the monitored person as the face image of an important person on the alarm history list screen 701 (see FIG. 17) by operating the check box 322 for "always display on the alarm list screen".

[0159] In addition, the subject registration and editing screen 301 is provided with a registered face image list display section 323. In the registered face image list display section 323, the registered face images related to the monitored person are arranged and displayed.

[0160] On the registered face image list display section 323, when the user performs an operation to select a face image (for example, mouse over), a menu button 324 appears for each face image. When the user operates the menu button 324, an operation menu (not shown) is displayed.

[0161] When the user selects the "representative face image" item in the operation menu, the face image corresponding to the operated menu button 324 is set as the representative face image. Also, when the user selects the "clear" item in the operation menu, the face image corresponding to the operated menu button 324 is deleted. Further, when the user selects the "display in original size" item in the operation menu, the face image and the full body image corresponding to the operated menu button 324 are popped up and displayed in the original size.

[0162] In addition, on the registered face image list display section 323, a reporting count display column 325 is provided for each registered face image. In the reporting count display column 325, the number of correct reports and incorrect reports is displayed. Thereby, the user can determine whether the registered face image is appropriate or not. For example, when there is no reporting, or when the number of incorrect reports is significantly larger compared to the number of correct reports, the user can determine that the registered face image is inappropriate. When the user determines that the registered face image is inappropriate, the user may perform an operation to delete the face image. Also, if there is a shortage of appropriate face images, the user may perform an operation to additionally register a face image.

[0163] In addition, on the subject registration and editing screen 301, a reporting elapsed period display column 331 and a reporting count display column 332 are provided. The display contents of the reporting elapsed period display column 331 and the reporting count display column 332 are the same as those of the reporting elapsed period display column 241 and the reporting count display column 242 on the subject list screen 201.

[0164] In addition, on the subject registration and editing screen 301, a registration information input section 335 is provided.

[0165] The registration information input section 335 is provided with an input box 341 for "person's name", an input box 342 for "age", and a checkbox 343 for "gender". In the input box 341 for "person's name", the user can input the person's name (common name) of the person to be monitored. In the input box 342 for "age", the user can input the age of the person to be monitored. In the checkbox 343 for "gender", the user can select the gender (male, female, unspecified) of the person to be monitored.

[0166] The registration information input section 335 is provided with an input box 344 for "remarks". The user can input reference information regarding the person to be monitored as remarks in the input box 344. Specifically, information regarding the importance of monitoring the person to be monitored (for example, habitual, most important, attention required, monitor immediately upon discovery, etc.) and information regarding the specific actions of the person to be monitored (for example, theft four times in the past, violence in the store, with alcohol consumption, etc.) are input.

[0167] The registration information input section 335 is provided with an input box 345 for "category". The user can select the category (for example, most important, important, attention) of the person to be monitored in the input box 345. When the category of the person to be monitored is selected, the category mark corresponding to that category is displayed. Note that the candidates for the category can be input by the user on the operation setting screen (not shown).

[0168] The registration information input section 335 is provided with an input box 346 for "alarm sensitivity". The user can input the alarm sensitivity (threshold value for alarm reporting determination) of the person to be monitored in the input box 346. Also, an initial value of the alarm sensitivity is set in advance, and the initial value of the alarm sensitivity is displayed in the input box 346 when the user has not input anything.

[0169] In addition, the registration information input unit 335 is provided with an expiration date setting unit 347. The expiration date setting unit 347 is provided with a check box 351 for "expiration date" and an input box 352 for date and time (year, month, day, hour, minute, and second). By operating the check box 351 for "expiration date", the user can switch between enabling and disabling the expiration date. Specifically, when the check box 351 for "expiration date" is checked, the expiration date becomes valid and input operations regarding the expiration date become possible. The expiration date is the expiration date of the period during which regular monitoring is carried out. When the expiration date elapses, the regular monitoring ends.

[0170] In addition, the expiration date setting unit 347 is provided with a radio button 353 for "alarm stop" and a radio button 354 for "registration cancellation" regarding the action when the expiration date has passed. When the user turns on the radio button 353 for "alarm stop", when the expiration date has passed, the monitored person is excluded from the reporting target, and the check box 231 for "alarm target" on the target person list screen 201 (see FIG. 13) becomes unchecked. When the user turns on the radio button 354 for "registration cancellation", when the expiration date has passed, the monitored person is excluded from the monitoring target, and the monitored person is deleted from the target person list screen 201.

[0171] In addition, the registration information input unit 335 is provided with a button 361 for "reset registration information". When the user operates the button 361 for "reset registration information", the input contents of each item in the registration information input unit 335 are returned to the initial values.

[0172] In addition, on the subject registration editing screen 301, a "Confirm" button 302 and a "Cancel" button 303 are provided. When the user operates the "Confirm" button 302, the content entered this time on the subject registration editing screen 301 is registered. In this case, when returning to the subject list screen 201 (see FIG. 13), the display content is updated according to the content entered this time on the subject registration editing screen 301. Also, when the user operates the "Cancel" button 303, the content entered this time on the subject registration editing screen 301 is discarded and the previous setting content is restored.

[0173] Next, the reporting screen 401 displayed on the monitoring terminal 4 will be described. FIG. 15 is an explanatory diagram showing the reporting screen 401.

[0174] On the monitoring terminal 4, when reporting is performed on the monitoring server 12, the reporting screen 401 is pop-up displayed on the video monitoring screen 501. Note that when the face search screen 101 (see FIG. 11) or the like is in the display state on the monitoring terminal 4, the reporting screen 401 is pop-up displayed on the face search screen 101.

[0175] Note that on the video monitoring screen 501, a "Video" tab 502, a "Live" tab 503, and a video display unit 504 are provided. The user can play back the video stored in the recording server 11 on the video display unit 504 by selecting the "Video" tab 502. Also, the user can display the real-time video output from the face recognition camera 1 and the in-store monitoring camera 2 on the video display unit 504 by selecting the "Live" tab 503. In addition, a camera selection unit 511 is provided on the video monitoring screen 501. In the camera selection unit 511, a checkbox 521 related to the face recognition camera 1 and a checkbox 522 related to the in-store monitoring camera 2 are provided. The user can select the face recognition camera 1 and the in-store monitoring camera 2 for displaying the video on the video display unit 504 by operating the checkboxes 521 and 522.

[0176] On the notification screen 401, a category display section 411, a person name display section 412, and a remarks display section 413 are provided. In the category display section 411, a category mark and a category name are displayed as information regarding the category to which the monitored person, who is the party to the notified event (the arrival of the monitored person at the store), belongs. In the person name display section 412, the person name (common name) of the monitored person is displayed. In the remarks display section 413, reference information regarding the monitored person is displayed.

[0177] Also, on the notification screen 401, a date and time display section 414, a store name display section 415, and a camera name display section 416 are provided. In the date and time display section 414, the date and time (year, month, day, hour, minute, second) when the notification was made is displayed. In the store name display section 415, the name of the store related to the notification is displayed. In the camera name display section 416, the name of the face recognition camera 1 related to the notification is displayed.

[0178] Also, on the notification screen 401, a similarity display section 417 and an image display section 418 are provided. In the similarity display section 417, the similarity between the face image of the arriving person and the face image of the monitored person when it is determined that the face recognition related to the notification has been successful is displayed. In the image display section 418, the detected image at the time of notification, that is, the full body image and the face image of the arriving person are displayed. Also, in the image display section 418, the representative face image of the monitored person determined to be the same person as the arriving person is displayed. The user can visually compare the detected image (full body image and face image) at the time of notification with the representative face image of the monitored person to confirm that the arriving person is the same person as the monitored person, that is, that the current notification is a correct notification.

[0179] In this system, when face matching between any of the multiple face images registered for the person under surveillance and the face image of the customer entering the store is successful, the entry of the person under surveillance into the store is detected and a notification is issued. Therefore, the face image of the person under surveillance when the face matching for the notification is successful may be the representative face image or other registered face images. For this reason, in the example shown in FIG. 15, the representative face image of the person under surveillance is displayed on the image display unit 418, but instead of the representative face image, the face image itself used in the successful face matching may be displayed. Also, when face matching is successful with multiple face images, the face image with the highest matching score may be displayed.

[0180] In addition, the notification screen 401 is provided with a "Details" button 421, a "Live Video" button 422, and a "Play" button 423. When the user operates the "Details" button 421, the user transitions to a notification details screen (not shown). On the notification details screen, all the contents of the items that were only partially displayed on the notification screen 401 due to space limitations are displayed. When the user operates the "Live Video" button 422, the notification screen 401 is closed and the video surveillance screen 501 (see FIG. 15) is displayed, and on the video surveillance screen 501, a live video, that is, a real-time video captured by the face matching camera 1, is displayed. When displaying the real-time video on the surveillance terminal 4, it is possible to directly acquire and display the video captured by the face matching camera 1, but it is also possible to display the video via the recording server 11. When the user operates the "Play" button 423, the notification screen 401 is closed and the video surveillance screen 501 is displayed, and on the video surveillance screen 501, a recorded video, that is, a video stored in the recording server 11, is displayed. At this time, the video is played back starting from a predetermined time (for example, 10 seconds) before the notification time.

[0181] In addition, the notification screen 401 is provided with a "×" button 402. When the user views the notification screen 401 and confirms the content of the generated notification event, the user operates the "×" button 402. Thereby, the notification screen 401 is closed.

[0182] In addition, an alarm transmission summary display section 431 is provided on the alarm transmission screen 401. The alarm transmission summary display section 431 displays, as alarm transmission summary information, the total number of alarm transmissions (the total number of alarm pop-up accumulations) and the number of target persons. The total number of alarm transmissions represents the total number of alarm events in the accumulation state that the user has not confirmed via the pop-up displayed alarm transmission screen 401. The number of target persons represents the number of monitored persons related to the alarm events in the accumulation state. In the example shown in FIG. 15, there are 5 alarm events in the accumulation state, and 3 monitored persons related to those alarm events.

[0183] Here, on the alarm transmission screen 401, only information regarding the latest alarm event is displayed. For this reason, if a new alarm event occurs while the user does not operate the "×" button 402, that is, while the alarm transmission screen 401 is open, on the alarm transmission screen 401, the display content regarding the previous alarm event is successively overwritten by the display content regarding the newly occurred alarm event. Therefore, just by viewing the alarm transmission screen 401, the user cannot confirm the content of the alarm event before it is overwritten. Thus, in the present embodiment, an alarm transmission summary display section 431 is provided on the alarm transmission screen 401.

[0184] In addition, the alarm events in the accumulation state include the latest alarm event displayed on the currently displayed alarm transmission screen 401 and the alarm events that were overwritten without the user's confirmation previously. Therefore, the total number of alarm transmissions represents the number obtained by adding 1 for the currently displayed alarm event to the number of times the alarm transmission screen 401 has been overwritten due to newly occurring alarm events without the user operating the "×" button 402. Note that after the "×" button 402 is operated and the alarm transmission screen 401 is closed, if a new alarm transmission screen 401 pops up, the total number of alarm transmissions on that alarm transmission screen 401 becomes "1", and thereafter, each time a new alarm event occurs without the user operating the "×" button 402, the total number of alarm transmissions becomes a value incremented by 1. In the example shown in FIG. 15, there are 4 alarm events that have been overwritten by newly occurring alarm events without the user's confirmation.

[0185] When the user operates the transmission report accumulation summary display section 431, the screen transitions to the transmission report accumulation details screen 601 (see Fig. 16).

[0186] Next, the transmission report accumulation details screen 601 displayed on the monitoring terminal 4 will be described. Fig. 16 is an explanatory diagram showing the transmission report accumulation details screen 601.

[0187] On the left side of the transmission report accumulation details screen 601, a "List" tab 602 and a "Search" tab 603 are provided. The example shown in Fig. 16 is the case of the list mode displayed when the user selects the "List" tab 602. On the other hand, when the user operates the "Search" tab 603, the screen transitions to a search mode (not shown) in which the user can input search conditions and instruct the search for transmission report events. In the search mode, similar to the face search screen 101 (see Fig. 11), the user can input detection conditions. Specifically, the user can specify the search target period, the source face image, and the search target camera.

[0188] As shown in Fig. 16, in the list mode, an important person display section 611 is provided on the left side of the transmission report accumulation details screen 601. In the important person display section 611, the face images of important persons among the monitored persons are displayed in a list by category (for example, most important, important, attention). Specifically, for each category, a category mark and a category name are displayed, and the representative face image and person name of the important persons belonging to that category are also displayed. An important person is a person among the monitored persons for whom the checkbox 322 "Always display on the alarm list screen" is checked on the target person registration / editing screen 301 (see Fig. 14).

[0189] In addition, a transmission report accumulation summary display section 621 is provided on the transmission report accumulation details screen 601. The transmission report accumulation summary display section 621 displays summary information regarding the accumulation status of transmission report events for each monitored person. Specifically, the person name and the individual transmission report accumulation count (number of accumulations) are displayed for each monitored person who is the party to the transmission report event. In the example shown in FIG. 16, there are 5 transmission report events in the accumulation state, there are 3 monitored persons related to the transmission report events, and the individual transmission report accumulation counts of the 3 monitored persons are 2 times, 2 times, and 1 time, respectively. Note that the transmission report events in the accumulation state include the latest transmission report event and the transmission report event overwritten by a newly generated transmission report event without the user checking.

[0190] In addition, a transmission report event list display section 631 is provided on the transmission report accumulation details screen 601. In the transmission report event list display section 631, transmission report event-specific display sections 632 regarding each transmission report event in the accumulation state are arranged side by side. In the example shown in FIG. 16, five transmission report event-specific display sections 632 regarding the latest transmission report event and 4 transmission report events overwritten by newly generated transmission report events without the user checking are displayed.

[0191] The event notification type display section 632 is provided with a category display column 641, a notification date and time display column 642, a camera display column 643, an image display column 645, a person name display column 646, a remarks display column 647, and a similarity display column 648. In the category display column 641, a category mark and a category name representing the category to which the monitored person, who is the party to the event notification, belongs are displayed. In the notification date and time display column 642, the date and time when the event notification occurred is displayed. In the camera display column 643, the name of the face camera 1 that photographed the customer determined to be the same person as the monitored person is displayed. In the image display column 645, as images related to the event notification, the detected image at the time of notification, that is, the full body image and face image of the customer, and the representative face image of the monitored person determined to be the same person as the customer are displayed. In the person name display column 646, the name (common name) assigned to the monitored person is displayed. In the remarks display column 647, reference information regarding the monitored person is displayed. In the similarity display column 648, the similarity between the face image of the customer and the face image of the monitored person when it is determined that the face verification related to the notification has been successful is displayed.

[0192] In addition, the event notification type display section 632 is provided with a "Delete Alarm" button 651. When the user operates the "Delete Alarm" button 651, a deletion confirmation dialog (not shown) is displayed. When the user performs an operation to approve the alarm deletion in the deletion confirmation dialog, the corresponding event notification is deleted. At this time, the user browses the notification screen 401 (see FIG. 15) to determine the correctness of the notification, and if it is determined to be a false alarm, the user operates the "Delete Alarm" button 651.

[0193] In addition, the event notification type display section 632 is provided with a menu button 652. When the user operates the menu button 652, an operation menu (not shown) is displayed. The content of the operation menu is the same as that of the menu button 151 on the face search screen 101 (see FIG. 11).

[0194] In addition, the transmission report accumulation details screen 601 is provided with a "Filter Release" button 661. When the user operates the "Filter Release" button 661, the system transitions to the transmission report history list screen 701 (see FIG. 17).

[0195] Next, the transmission report history list screen 701 displayed on the monitoring terminal 4 will be described. FIG. 17 is an explanatory diagram showing the transmission report history list screen 701.

[0196] The transmission report history list screen 701 is the result of releasing the filter applied to the transmission report accumulation details screen 601, which filters and displays information on each transmission report event in the accumulation state. On the transmission report history list screen 701, all transmission report events that have occurred up to the present, including those in the accumulation state, are listed and displayed.

[0197] The transmission report history list screen 701 is provided with a transmission report event list display section 702. The transmission report event list display section 702 is provided with a display section 632 for each transmission report event related to each transmission report event that occurred in the past. The display section 632 for each transmission report event is the same as that of the transmission report accumulation details screen 601 (see FIG. 16).

[0198] In addition, the transmission report event list display section 702 is provided with a fixed display section 711 and a scroll display section 712. The fixed display section 711 displays the most recent predetermined number of (for example, 3) transmission report events. The scroll display section 712 displays transmission report events that occurred before the transmission report events displayed in the fixed display section 711. In the scroll display section 712, a scroll bar 713 is displayed. By operating the scroll bar 713, the user can scroll and display the display section 632 for each transmission report event and view transmission report events that are not within the screen display range.

[0199] Next, the screen displayed on the staff terminal 6 at the time of transmission report will be described. FIG. 18 is an explanatory diagram showing the mail screen 801 and the transmission report details screen 901 displayed on the staff terminal 6 at the time of transmission report.

[0200] When a notification is issued by the monitoring server 12, the worker terminal 6 receives an email from the monitoring server 12 notifying the worker (store manager or store clerk) of the occurrence of the notified event (the visit of the monitored person). The email screen 801 shown in FIG. 18(A) is displayed by the email application.

[0201] On the email screen 801 shown in FIG. 18(A), text notifying the occurrence of the notified event (the visit of the monitored person) is displayed. In addition, on the email screen 801, the name of the store where the notified event occurred, the date and time when the notified event occurred, the name of the face recognition camera 1 that captured the notified event, the name of the category to which the monitored person who is the subject of the notified event belongs, the personal name (common name) of the monitored person who is the subject of the notified event, and remarks (reference information) regarding the monitored person who is the subject of the notified event are displayed.

[0202] In addition, a URL is displayed on the email screen 801. When the user operates the URL, the notification details screen 901 shown in FIG. 18(B) is displayed by the browser.

[0203] On the notification details screen 901 shown in FIG. 18(B), a category display section 911, a personal name display section 912, a remarks display section 913, a date and time display section 914, a store name display section 915, a camera name display section 916, a similarity display section 917, and an image display section 918 are provided. The display contents of the category display section 911, the personal name display section 912, the remarks display section 913, the date and time display section 914, the store name display section 915, the camera name display section 916, the similarity display section 917, and the image display section 918 are the same as those of the category display section 411, the personal name display section 412, the remarks display section 413, the date and time display section 414, the store name display section 415, the camera name display section 416, the similarity display section 417, and the image display section 418 on the notification screen 401 (see FIG. 15) displayed on the monitoring terminal 4.

[0204] Note that the example shown in FIG. 18(B) is a case where the worker terminal 6 is a smartphone. Since the display is small, the notification screen 401 is vertically configured, and the user can view the entire notification screen 401 by performing a scroll operation.

[0205] In addition, the alert details screen 901 displayed on the staff terminal 6 may be provided with an alert accumulation summary display section (not shown) similar to the alert accumulation summary display section 431 provided on the alert screen 401 (see Figure 15) displayed on the monitoring terminal 4.

[0206] Second embodiment Next, a second embodiment will be described. Note that points not specifically mentioned here are the same as those in the above embodiment. Fig. 19 is an overall configuration diagram of a monitoring system according to the second embodiment.

[0207] In the first embodiment, a monitoring terminal 4 is installed in a store, and a security guard or a store manager stationed in the store operates the monitoring terminal 4 to view the screen displayed on the monitoring terminal 4. On the other hand, in the present embodiment, a monitoring terminal 21 is installed in a security company's business premises (such as a monitoring center), and a security guard of the security company operates the monitoring terminal 21 to view the screen displayed on the monitoring terminal 21. In addition, a warning light 22 is installed in the security company's business premises. In addition, a security guard of the security company (such as a patrol security guard) carries an employee terminal 23, and operates the employee terminal 23 to view the screen displayed on the employee terminal 23. If unnecessary, the employee terminal 6 and the warning light 5 on the store side may be omitted.

[0208] Third embodiment In the first embodiment, when the user operates the menu button 151, an operation menu is displayed, and when the user selects the item "Additionally register" from the "Register to target person list" in the operation menu, the face image is registered in association with the person to be monitored. However, some face images may not be suitable for registration because most of the face is hidden, etc. Therefore, the monitoring system according to the third embodiment performs a quality check on the face image to determine whether or not it can be registered.

[0209] Quality checks are performed based on characteristic states related to wearable items (such as glasses, hats, etc.) and the orientation of the face (such as sideways). FIG. 20 shows an example of a check item column 951 that accepts input of quality check items. The check item column 951 includes items such as sunglasses and masks. The check item column 951 can be displayed on the setting input screen of the face authentication server 13 or the like.

[0210] By selecting such check items, it is possible to prevent the registration of face images for which it is difficult to identify a person from a photo, and also by removing check items, it becomes possible to register face images that correspond to check items, such as face images of people who always wear sunglasses or masks and for which face registration was not possible.

[0211] Quality checks include two processes for face images: a face state detection process that detects characteristic states (hereinafter referred to as face states) related to wearable items (such as glasses, hats, etc.) and the orientation of the face (such as sideways), and a collation (hereinafter referred to as a check process) between the face state obtained by the face state detection process and the quality check items. Specifically, when it is detected by the face state process that sunglasses are being worn and the sunglasses in the check items are checked, it is determined that it is not suitable for registration, and the registration of the face image becomes impossible.

[0212] The face state detection process and the check process may be performed during the face search process, similar to the first embodiment.

[0213] FIG. 21 shows a flowchart when the face state detection process and the check process are performed during the face search process. When a search condition (for example, date and time) is input to the monitoring terminal 4, the search condition is transmitted to the monitoring server 12, and a request is transmitted from the monitoring server 12 to the personal information storage server 15 to transmit face images that match the search condition.

[0214] When the personal information storage server 15 receives a request including search conditions from the monitoring server 12, it extracts face images that match the search conditions (excluding the match with the face image to be verified), and transmits them to the monitoring server 12. The monitoring server 12 transmits the face images obtained from the personal information storage server 15 to the face authentication server 13 together with the request related to face information acquisition. When the search conditions include the face image to be verified (hereinafter referred to as the search face image), the monitoring server 12 also transmits the face image to be verified (or information for specifying the face image, face feature amounts, etc.) to the face authentication server 13.

[0215] When the face authentication server 13 (specifically, the processor 33) obtains a face image from the monitoring server 12 together with the request related to face information acquisition, it executes a face state detection process for each of the obtained face images. The face authentication server 13 may perform a check process together with the face state detection process and determine whether registration is possible.

[0216] When the search face image is included, the face authentication server 13 may also perform face authentication between the face images obtained from the monitoring server 12 and the face image to be verified. At this time, the face authentication server 13 executes the face state detection process only for the face images determined to match the face image to be verified.

[0217] The face authentication server 13 associates the face information obtained by the face state detection process and the determination result of whether registration is possible obtained by the check process with the face image, and transmits them to the monitoring server 12. When the face authentication server 13 performs collation with the face image to be verified, it may also transmit the collation result to the monitoring server 12.

[0218] The monitoring server 12 associates the face information and the determination result of whether registration is possible with the face image, and transmits them to the monitoring terminal 4. At this time, the monitoring server 12 may also transmit the collation result.

[0219] The monitoring terminal 4 displays the obtained face information and the determination result of whether registration is possible together with the face images that match the search conditions. At this time, the monitoring terminal 4 may also display the obtained collation result.

[0220] FIG. 22 shows an example of the face search screen 101 when the determination result of registration availability is transmitted in association with a face image. In FIG. 22, for a face image determined to be non-registerable, masking indicating non-registerability is applied and displayed. Also, as shown in FIG. 21, text 952 of "Non-registerable" or an icon indicating non-registerability may be displayed on the image determined to be non-registerable.

[0221] FIG. 23 shows an example of the subject registration editing screen 301 when the determination result of registration availability is transmitted in association with a face image. The subject registration editing screen 301 shown in FIG. 23 can be displayed when the item of "Register additionally" is selected in the operation menu 953 of the menu button 151 of the face search screen 101 shown in FIG. 24.

[0222] Also, similar to the first embodiment, as shown in FIG. 25, when the menu button 652 of the notification history list screen 701 is operated, the item of "Register additionally" can be selected in the operation menu 953 to additionally register a face image. The subject registration editing screen 301 shown in FIG. 22 can also be displayed when the item of "Register additionally" is selected in the operation menu 954 of the menu button 652 of the notification history list screen 701.

[0223] On the subject registration editing screen 301 shown in FIG. 23, text 955 indicating that the face image to be additionally registered is non-registerable is displayed in the face image registration section 312. However, the mode of notifying the availability of additional registration is not limited to text 955, and it may be a display by an icon or the like. In addition, when the face image to be registered is non-registerable, the items of "Register newly" and "Register additionally" may be configured to be unselectable in the operation menu 953 shown in FIG. 24 and the operation menu 954 shown in FIG. 25.

[0224] In addition, the mode of notifying the availability of additional registration may be, for example, a pop-up 956 shown in FIG. 26 or the like.

[0225] In the above embodiment, the face state detection process was performed when the face image was searched. However, the face state detection process may be performed before the associated information is saved. An example of the sequence diagram at that time is shown in FIG. 27. In the example shown in FIG. 27, after the personal information storage server 15 performs image accumulation, it transmits the face images stored in the face authentication server 13. Then, the face authentication server 13 performs a quality check on each face image.

[0226] Specifically, the face authentication server 13 first performs a face state detection process on the face image transmitted from the monitoring server 12 to detect the face state. Next, the face authentication server 13 performs a check process using the detected face state to determine whether registration is possible. The face authentication server 13 transmits the detected face state and the determination result of registration availability to the monitoring server 12 in association with the face image, and the monitoring server 12 stores the face state, the determination result of registration availability, and the associated information in association with the face image. Note that the face authentication server 13 may transmit the detected face state, the determination result of registration availability, and the personal information storage server 15, and the personal information storage server 15 may store the face state, the determination result of registration availability, and the associated information in association with the face image.

[0227] In this case, when a face image is searched on the monitoring terminal 4, the monitoring terminal 4 acquires the face state and the determination result of registration availability together with the associated information from the personal information storage server 15. The monitoring terminal 4 may display the face state and the determination result acquired from the personal information storage server 15 on the face search screen 101 (see, for example, FIG. 22). Further, when "additional registration" is selected in the operation menus 953 and 954, the monitoring terminal 4 may notify the determination result of registration availability by means of a pop-up 956, text 955 on the target person registration editing screen 301, an icon, or the like.

[0228] (Fourth Embodiment) In the above-described third embodiment, the quality check was performed during the face search process or before storing the associated information. However, the quality check may be performed at any timing as long as it is before face registration. Although the quality check is performed in the monitoring system according to the fourth embodiment in the same manner as in the third embodiment, different from the third embodiment, the quality check is performed immediately before face registration.

[0229] FIG. 28 shows a sequence diagram after the input reception of search conditions is performed in the monitoring system according to the fourth embodiment and the search results are displayed on the face search screen 101.

[0230] When a face image to be registered is selected on the face search screen 101 and the item "Register additionally" is selected in the operation menu, a registration request for the face image selected from the monitoring terminal 4 is sent to the monitoring server 12. In the fourth embodiment, even if the face state is not detected in the face search process and the face reference information (information related to the face state) is not displayed on the face search screen 101, it is acceptable. When the monitoring server 12 receives a registration request from the monitoring terminal 4, the monitoring server 12 sends the received registration request for the face image to the face authentication server 13.

[0231] When the face authentication server 13 receives a registration request for a face image from the face authentication server 13, the face authentication server 13 performs a face state detection process on the face image for which registration is requested to obtain the face state. Next, the face authentication server 13 performs a check process to compare the obtained face state with the quality check items, and determines whether face registration is possible.

[0232] When it is determined that face registration is possible, the face authentication server 13 performs face image registration and sends a message indicating that the registration was successful (registration result) to the monitoring server 12. When it is determined that face registration is impossible, the face authentication server 13 does not perform face image registration and sends a message indicating that the registration failed (registration result) to the monitoring server 12.

[0233] When the monitoring server 12 receives the registration result from the face authentication server 13, it transmits the registration result to the monitoring terminal 4. As a result, the registration result is displayed on the monitoring terminal 4. The monitoring terminal 4 may display the registration result by means of a pop-up 956 illustrated in FIG. 26 or the like.

[0234] As described above, embodiments have been described as examples of the technology disclosed in the present application. However, the technology in the present disclosure is not limited thereto, and can also be applied to embodiments in which changes, replacements, additions, omissions, etc. are made. It is also possible to combine the respective components described in the above embodiments to form a new embodiment.

Industrial Applicability

[0235] The monitoring system and monitoring method according to the present invention enable a user to quickly and appropriately make a determination for maintaining registration information regarding a monitoring target person with reference to the implementation status of past reports regarding each monitoring target person, and have the effect of suppressing the occurrence of false alarms. It is useful as a monitoring system and monitoring method in which a processor executes a process of performing a report regarding detection of a monitoring target person as a reported event based on an image of a visitor to a facility.

Explanation of Signs

[0236] 1: Camera for face illumination 2: In-store monitoring camera 3: Control terminal 4: Monitoring terminal (terminal device, notification device) 5: Warning light 6: Staff terminal 11: Recording server 12: Monitoring server (reported event history storage unit) 13: Face authentication server (registered image storage unit) 14: User management server 15: Personal information storage server (visitor image storage unit) 21: Monitoring terminal 22: Warning light 23: Staff terminal 101: Face search screen 201: Target Person List Screen 301: Target Person Registration / Edit Screen 401: Transmission Screen 601: Transmission Accumulation Details Screen 701: Transmission History List Screen 154: Facial Reference Information Display Column 241: Transmission Elapsed Period Display Column 242: Transmission Count Display Column 243: Registered Facial Image Count Display Column 251: Sort Button (Operation Section) 325: Transmission Count Display Column 331: Transmission Elapsed Period Display Column 332: Transmission Count Display Column 345: "Category" Input Box 346: "Alarm Sensitivity" Input Box 347: Expiration Date Setting Section 431: Transmission Accumulation Summary Display Section 621: Transmission Accumulation Summary Display Section 631: Transmission Event List Display Section

Claims

1. A monitoring system that executes a process of issuing an alert regarding detection of a monitored person as an alert event based on an image of a visitor to a facility, the monitoring system comprising: A camera for capturing images of the facility; a registration image storage unit that stores a registration face image for each person to be monitored; a notification history storage unit that stores notification history information regarding notifications made in the past for each monitored person; A terminal device operated by a user who manages information about each person to be monitored; an alarm device that performs a predetermined alarm operation in response to an instruction from the processor, The processor, A face matching process is executed to compare the face image of the visitor with the registered face image of the person to be monitored to determine whether or not the visitor is the same person, and based on the result of the face matching process, an alert is issued regarding the detection of the person to be monitored. Depending on the user's operation requesting viewing, A monitoring system characterized by generating summary information of the alert history based on the alert history information for each monitored person, and displaying a screen on the terminal device in which the summary information of the alert history for each monitored person is added to the registration information for each monitored person.

2. 2. The monitoring system according to claim 1, wherein the alarm history summary information indicates a period of time that has elapsed since the last alarm was issued.

3. The monitoring system according to claim 1 , wherein the alarm history summary information is the number of times an alarm has been issued in the past.

4. The processor, 2. The surveillance system according to claim 1, wherein a screen including information regarding the number of face images registered for each person to be monitored is displayed on the terminal device.

5. The processor, Displaying on the terminal device a screen including an operation unit for changing the order of information for each person to be monitored; The monitoring system according to claim 1, characterized in that the terminal device displays a screen in which the information for each target person displayed in the list is rearranged in response to a user's operation on the operation unit.

6. The processor, 2. The surveillance system according to claim 1, wherein a screen including the number of correct reports and false reports for each of the registered face images is displayed on the terminal device.

7. The processor, The monitoring system according to claim 1, characterized in that a category is set for each person to be monitored in response to a user's operation of selecting a category to which the person to be monitored belongs from a plurality of pre-set categories.

8. The processor, setting a notification condition for each person to be monitored in response to a user's operation specifying a notification condition used when determining whether or not to issue a notification based on a result of the face matching process; The monitoring system according to claim 1 , wherein a decision is made as to whether or not to issue an alarm based on the result of the face matching process and the alarm condition.

9. The processor, The surveillance system according to claim 8, characterized in that a threshold value for determining whether to issue an alarm is set as the alarm condition, based on a matching score representing the degree of similarity between the visitor's facial image and the facial image of the person being monitored, obtained in the facial matching process.

10. The processor, 2. The monitoring system according to claim 1, wherein the expiration date is set for each person to be monitored in response to a user's operation specifying an expiration date during which regular monitoring is to be performed.

11. The processor, The monitoring system according to claim 10, characterized in that the expiration date and the content of the action to be taken when ...

12. The processor, The monitoring system described in claim 1, characterized in that when the summary information of the alarm history of any monitored person corresponds to a predetermined warning level, a screen in which the summary information of the alarm history of that monitored person is highlighted is displayed on the terminal device.

13. A monitoring method for executing a process of issuing an alert regarding detection of a monitored person as an alert event based on an image of a visitor to a facility, the process comprising: Accumulating information on past alerts for each monitored person in an alert history storage unit; A face matching process is executed to compare the visitor's face image with the registered face image of the person to be monitored to determine whether or not the visitor is the same person, and based on the result of the process, an alert is issued regarding the detection of the person to be monitored. Depending on the user's operation requesting viewing, A monitoring method characterized by generating summary information on the alert history for each monitored person based on the alert history information for each monitored person, and displaying on a terminal device a screen in which the summary information on the alert history for each monitored person is added to the registration information for each monitored person.

Images