Person detection device, person tracking device, person tracking system, person detection method, person tracking method, person detection program, and person tracking program

Abstract

To allow a monitor to easily recognize accuracy of an authentication result.SOLUTION: A detection / tracking result determination function unit 34 acquires from a monitoring / tracking execution function unit 23, similarity to an image of a similarity calculation object person being an object image and an image of a specific person being a specific image extracted respectively from monitor images periodically acquired respectively by a plurality of monitor cameras 10, detects an image similar to the image of the specific person from the images of the similarity calculation object person extracted from the monitor images of the plurality of monitor cameras 10 on the basis of a plurality of thresholds stored in a camera information table storage unit 321, and detects one of a plurality of attention stages which the detected image of the similarity calculation object person belongs to. A detection alert output function unit 3521 generates a detection alert screen 42 for presenting the detected image similar to the image of the specific person together with a stage display indicating the detected attention stage.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] One aspect of the present invention relates to a person detection device, a person tracking device, a person tracking system, a person detection method, a person tracking method, a person detection program, and a person tracking program.

Background Art

[0002] In recent years, with the development of technologies for performing person authentication based on the similarity of facial and full-body feature amounts of a person extracted from an image, various usage methods of the authentication technology have been proposed. For example, Patent Document 1 proposes a technique of transmitting a face authentication result of a person extracted from a surveillance image acquired by a surveillance camera to an external terminal such as a surveillance device via a communication interface unit and displaying it on a screen display unit thereof.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Due to the type and performance of the surveillance camera that acquires the surveillance image, both the output result of face authentication and the output result of full-body matching are different. Therefore, it is difficult for the monitor to recognize the reliability of the authentication result from the authentication result displayed on the screen display unit of the surveillance device.

[0005] The present invention has been made paying attention to the above circumstances, and an object thereof is to enable the monitor to easily recognize the reliability of the authentication result.

Means for Solving the Problems

[0006] To achieve the above object, one aspect of the person detection device according to the present invention uses one similarity calculation method for a specific image The face image of a specific person for a target imageThe face image that is the similarity of the face image of the person to be calculated for similarity Based on the similarity calculated by the similarity calculation unit that calculates the similarity, for the target image with respect to one collation model, a plurality of Face image threshold values for classifying into a plurality of stages are stored in a threshold storage unit, and for each of the target images extracted from each of the monitoring images periodically acquired by a plurality of monitoring cameras, the Face verification similarity of the image of the person whose similarity is to be calculated, which is the target image, to the image of the specific person, which is the specific image, is Face acquired from the similarity calculation unit, and based on the plurality of Face threshold values stored in the threshold storage unit, among the images of the person whose similarity is to be calculated extracted from the monitoring images of each of the plurality of monitoring cameras, the Face image image similar to the image of the specific person is detected, and at the same time, it is detected which of the plurality of attention stages the detected image of the person whose similarity is to be calculated belongs to. A detection unit for detecting, and an output screen generation unit for generating an output screen for presenting the image similar to the image of the specific person detected by the detection unit together with a stage display indicating the attention stage detected by the detection unit are provided. Face verification One aspect of the person tracking device according to the present invention is an aspect of the person detection device according to the present invention, and from among the images of a plurality of the persons whose similarity is to be calculated and who are similar to the image of the specific person presented on the output screen together with the stage display by the person detection device, the Face designation of the image of the tracking target person to be tracked is received from a monitoring terminal operated by a monitor, and the designated Face image is registered as a tracking Face image, and the tracking Face image, which is the image of the tracking target person to be tracked and is the registered image, is used as the specific image, and the similarity of the image of the person whose similarity is to be calculated is calculated by the similarity calculation unit using the similarity calculation method. Face image of the Face

[0007] Face Face Face Face Face Face Face Face Face image ​​​​​​​​​​A registration unit that extracts similarity, and the detection unit obtains the Face image image of the person to be tracked detected from the Face similarity, and generates a tracking output screen for presenting the image of the person whose similarity is to be calculated and who is similar to the image of the person to be tracked, together with the stage display indicating the attention stage detected by the detection unit. Face That is, it is configured to include a tracking output screen generation unit.

[0008] Alternatively, one aspect of the person tracking device according to the present invention is to use a single similarity calculation method to calculate the similarity of a target image with respect to a specific image The face image of a specific person and classify the target image into a plurality of stages based on the similarity calculated by a similarity calculation unit, and store a plurality of The face image that is the similarity of the face image of the person to be calculated for similarity threshold values for this purpose in a threshold storage unit. A registration unit registers the image of the person to be tracked specified from a monitoring terminal operated by a monitor as the specific image. The similarity of the image of the person whose similarity is to be calculated, which is the target image extracted from each of the monitoring images periodically acquired by each of a plurality of monitoring cameras, with respect to the specific image is obtained from the similarity calculation unit, and based on the plurality of Face image threshold values stored in the threshold storage unit, among the images of the person whose similarity is to be calculated extracted from the monitoring images of each of the plurality of monitoring cameras, an image similar to the Face verification image of the person to be tracked is detected, and a detection unit that detects which of a plurality of attention stages the detected image of the person whose similarity is to be calculated belongs to, and the detection unit obtains the Face similarity from the similarity calculation unit, and generates a tracking output screen for presenting the image of the person whose similarity is to be calculated and who is similar to the image of the person to be tracked, together with the stage display indicating the attention stage. Face That is, it is configured to include a tracking output screen generation unit. Face image Face verification Face Face Face Face Face image Face Face

[0009] ​​​​​​​​​One aspect of the person tracking system according to the present invention includes one aspect of the person tracking device according to the present invention, the plurality of surveillance cameras, the surveillance terminal operated by the surveillant, and an analysis unit including the similarity calculation unit.

[0010] One aspect of the person detection method according to the present invention is a person detection method for detecting a specific person from surveillance images acquired by a plurality of surveillance cameras. A computer uses one similarity calculation method to calculate the similarity between a specific image The face image of a specific person and a target image The face image that is the similarity of the face image of the person to be calculated for similarity calculated by a similarity calculation unit, and based on the calculated Face image similarity, stores a plurality of Face verification threshold values for classifying the target image into a plurality of stages in a memory. For each of the surveillance images periodically acquired by each of the plurality of surveillance cameras, the similarity of the image of the similarity calculation target person, which is the target image extracted from each of the surveillance images, to the image of the specific person, which is the specific image, is obtained from the similarity calculation unit. Based on the plurality of Face threshold values stored in the memory, from the images of the similarity calculation target person extracted from the surveillance images of each of the plurality of surveillance cameras, Face images similar to the image of the specific person are detected, and it is detected which of the plurality of attention stages the detected image of the similarity calculation target person belongs to. An output screen is generated to present the images similar to the detected image of the specific person together with a stage display indicating the detected attention stage. Face verification Face Face Face Face Face Face

[0011] One aspect of the person tracking method according to the present invention is a person tracking method for tracking a tracking target person to be tracked from surveillance images acquired by a plurality of surveillance cameras. A computer, based on the output screen generated by one aspect of the person detection method according to the present invention and presented together with the stage display, for the images of a plurality of similarity calculation target persons similar to the image of the specific person Face Face ​​​​​​​​From among the images, the Face designation of the image of the person to be tracked is received from a monitoring terminal operated by a monitor, and the designated Face image is registered in the memory as a Face tracking image, and the Face tracking image, which is the image of the person to be tracked registered in the memory, is used as the Face specific image, and the similarity of the image of the person to be calculated for similarity calculation is extracted by the similarity calculation unit using the similarity calculation method, and the Face similarity of the image of the person to be calculated for similarity calculation extracted from each of the monitoring images of the plurality of monitoring cameras with respect to the Face image specific image, which is the image of the person to be tracked, is obtained from the similarity calculation unit, and based on the Face plurality of Face image thresholds stored in the memory, the Face verification image of the person to be calculated for similarity calculation extracted from each of the monitoring images of the plurality of monitoring cameras is detected for the Face image of the person to be tracked among the Face images of the persons to be calculated for similarity calculation similar to the Face image of the person to be tracked, and it is detected which of the plurality of attention levels the detected Face image of the person to be calculated for similarity calculation belongs to, and the Face image image of the person to be tracked detected from the similarity obtained from the similarity calculation unit is detected for the Face image of the person to be calculated for similarity calculation similar to the Face image, and a tracking output screen is generated to present the detected image of the person to be calculated for similarity calculation similar to the image of the person to be tracked together with a stage display indicating the detected attention level.

[0012] Alternatively, one aspect of the person tracking method according to the present invention is a person tracking method for tracking a person to be tracked from monitoring images acquired by a plurality of monitoring cameras, wherein a computer uses a single similarity calculation method to calculate the The face image of a specific person similarity of a target image with respect to a The face image that is the similarity of the face image of the person to be calculated for similarity specific image calculated by a similarity calculation unit, and based on the Face image similarity, the target image is classified into a plurality of Face verificationStore the threshold value in the memory, and perform the specified tracking of the person to be tracked designated from the monitoring terminal operated by the monitor Face The image is registered in the memory as the specific image, and is the image of the person to be calculated for similarity, which is the target image extracted from each of the monitoring images periodically acquired by each of the plurality of monitoring cameras Face Obtain the similarity of the image of the person to be calculated for similarity to the specific image from the similarity calculation unit, and the plurality of Face verification Based on the threshold values, from the images of the person to be calculated for similarity extracted from the monitoring images of each of the plurality of monitoring cameras Face Among the images, the Face Images similar to the image of the person to be tracked Face Detect the image, and detect which of the plurality of attention levels the detected image of the person to be calculated for similarity is, and from the similarity obtained from the similarity calculation unit, the Face Images similar to the image of the person to be tracked Face Generate a tracking output screen for presenting the image of the person to be calculated for similarity similar to the detected image of the person to be tracked together with the stage display indicating the detected attention level Face is what it is.

[0013] One aspect of the person detection program according to the present invention is a program for operating a computer as each part of one aspect of the person detection device according to the present invention.

[0014] One aspect of the person tracking program according to the present invention is a program for operating a computer as each part of one aspect of the person tracking device according to the present invention.

Effect of the Invention

[0015] According to each aspect of the present invention, the supervisor can easily recognize the reliability of the authentication result.

Brief Explanation of Drawings

[0016]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20A

Figure 20B

Figure 20C

Figure 21

Figure 22

DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0018] [One Embodiment] (1) Configuration (1-1) Overall Configuration FIG. 1 is a diagram showing an example of the overall configuration of a person tracking system 1 according to an embodiment of the present invention.

[0019] The person tracking system 1 is a system for tracking a specific person in a large facility having a wide monitoring range, such as a building having a plurality of floors such as an office building or a department store, or a commercial facility including a plurality of stores.

[0020] This person tracking system 1 includes a plurality of monitoring cameras 10, a video analysis function unit 20 provided corresponding to each monitoring camera 10, and a server device SV including a Web server 30 that functions as a person detection device and a person tracking device according to an embodiment of the present invention, and a monitoring terminal 40. Each monitoring camera 10, the server device SV, and the monitoring terminal 40 are connected via a network NET.

[0021] The network NET is an in-house network, for example, a wireless LAN (Local Area Network) or a wired LAN. The network NET may be a wide area network such as the Internet.

[0022] The plurality of monitoring cameras 10 are network cameras distributed in the large facility so that their shooting ranges do not overlap or partially overlap. The monitoring camera 10 may be a video camera that shoots a moving image within the shooting range, and each frame image constituting the moving image can be obtained as a monitoring image of the shooting range. Further, the monitoring camera 10 may be a still camera that periodically obtains a monitoring image by shooting a still image at a certain time interval.

[0023] Each of the plurality of video analysis functional units 20 arranged in the server device SV has various analysis functions for the surveillance images acquired from the corresponding surveillance cameras 10. Details of the analysis functions provided by this video analysis functional unit 20 will be described later. Note that the video analysis functional unit 20 may not be arranged inside the server device SV in this way, but may be arranged independently in a dedicated computing device or cloud, etc., and data can be exchanged between the corresponding surveillance camera 10 and the server device SV via the network NET. Further, if the surveillance camera 10 has a computing function, the video analysis functional unit 20 can also be arranged inside the camera. In the present embodiment, the surveillance camera 10 and the video analysis functional unit 20 are configured to correspond one-to-one, but one video analysis functional unit 20 may be assigned to a plurality of surveillance cameras 10.

[0024] The Web server 30 of the server device SV is composed of a server computer, and as functional units, includes a file server 31, a database (hereinafter abbreviated as DB) server 32, a Web application 35, a search human detection result storage functional unit 33, and a detection / tracking result determination functional unit 34. Details of each of these functional units will be described later.

[0025] The surveillance terminal 40 is composed of a computer such as a personal computer and can execute the program of the Web browser 41. A surveillance person such as a security guard who operates the surveillance terminal 40 by the Web browser 41 can view the person tracking results provided by the Web application 35 of the Web server 30. In FIG. 1, only one surveillance terminal 40 is shown, but the person tracking system 1 may include a plurality of surveillance terminals 40.

[0026] Further, the server device SV may have the function of the surveillance terminal 40.

[0027] (1-2) Video analysis functional unit 20 FIG. 2 is a block diagram showing an example of the configuration of the video analysis functional unit 20 shown in FIG. 1. The video analysis functional unit 20 includes an image acquisition module 21, a human detection information extraction functional unit 22, and a monitoring / tracking execution functional unit 23.

[0028] The image acquisition module 21 acquires frame images transmitted from the monitoring camera 10 via the network NET according to a predetermined communication protocol such as RTSP (Real Time Streaming Protocol). The image acquisition module 21 transmits the acquired frame images to the web server 30 via in-server communication, Websocket, etc., and stores them in the file server 31. When the video analysis functional unit 20 is arranged independently of the server device SV, the image acquisition module 21 transmits the acquired frame images to the web server 30 via the network NET. Also, the image acquisition module 21 sends the acquired frame images to the human detection information extraction functional unit 22.

[0029] The human detection information extraction functional unit 22 functions as an extraction unit that extracts a full-body image of a person from the input image, extracts a full-body feature amount, which is a feature amount of this full-body image, and a face image of the person from this full-body image, and further extracts a face feature amount, which is a feature amount of this face image. Specifically, the human detection information extraction functional unit 22 includes a full-body detection module 221, a region tracking module 222, a full-body feature amount extraction module 223, a face detection module 224, and a face feature amount extraction module 225.

[0030] The full-body detection module 221 is a full-body image extraction unit that detects the full body of each person shown in the frame image, which is the input image, and extracts a full-body image. The full-body detection module 221 can detect the full body of a person, for example, based on a pre-trained full-body image of a person by machine learning such as deep learning. The full-body detection module 221 transmits the extracted full-body image to the web server 30 via in-server communication, Websocket, etc., and stores it in the file server 31. When the video analysis function unit 20 is arranged independently of the server device SV, the full-body detection module 221 transmits the extracted full-body image to the web server 30 via the network NET. In addition, the full-body detection module 221 also sends the extracted full-body image to the full-body feature amount extraction module 223 and the face detection module 224. Furthermore, the full-body detection module 221 sends the frame image and the extracted full-body image to the region tracking module 222.

[0031] The region tracking module 222 detects the region position of the full-body image in the frame image input from the full-body detection module 221.

[0032] The full-body feature quantity extraction module 223 is a full-body feature quantity extraction unit that extracts full-body feature quantities from the input full-body image. The full-body feature quantity is an objective quantification of the characteristics of the body of the person included in the full-body image. For example, the full-body feature quantity can be a quantification of the attribute information of the person himself, such as body type, height, gender, age, etc. estimated from the image. The full-body feature quantity extraction module 223 can extract full-body feature quantities using a pre-trained model such as deep learning (e.g., Deep-person-reid, etc.). Also, the full-body feature quantity extraction module 223 can calculate these full-body feature quantities as local image feature quantities such as HOG (Histogram of Oriented Gradients) and SIFT (Scaled Invariance Feature Transform). The full-body feature quantity extraction module 223 transmits the extracted full-body feature quantities to the web server 30 via in-server communication or Websocket, etc., and stores them in the file server 31. When the video analysis function unit 20 is arranged independently from the server device SV, the full-body feature quantity extraction module 223 transmits the extracted full-body feature quantities to the web server 30 via the network NET. Furthermore, the full-body feature quantity extraction module 223 sends the extracted full-body feature quantities to the monitoring / tracking execution function unit 23.

[0033] The face detection module 224 is a face image extraction unit that detects the face of a person from the input full-body image and extracts a face image. The face detection module 224 can detect the face of a person based on, for example, a pre-trained face image of a person by machine learning such as deep learning. The face detection module 224 sends the extracted face image to the face feature quantity extraction module 225.

[0034] The face feature extraction module 225 is a face feature extraction unit that extracts face features from the input face image. The face features are an objective quantification of the features of the face of the person included in the face image. The face feature extraction module 225 can extract face features using a model pre-trained by deep learning or the like, for example, in the same manner as the whole body feature extraction module 223 that extracts whole body features. Also, the face detection module 224 may calculate these face features as local image features such as HOG or SIFT. The face detection module 224 transmits the extracted face features to the web server 30 via in-server communication, Websocket, etc., and stores them in the file server 31. When the video analysis function unit 20 is arranged independently of the server device SV, the face feature extraction module 225 transmits the extracted face features to the web server 30 via the network NET. Further, the face detection module 224 sends the extracted face features to the monitoring / tracking execution function unit 23.

[0035] Also, the monitoring / tracking execution function unit 23 functions as a similarity calculation unit that calculates the whole body image similarity, which is the similarity of the whole body features of two input images, and the face image similarity, which is the similarity of the face features of two input images. Specifically, the monitoring / tracking execution function unit 23 includes a whole body matching module 231 and a face matching module 232.

[0036] The whole body matching module 231 calculates the similarity between the whole body features input from the whole body feature extraction module 223 and the whole body features of the person to be tracked instructed from the monitoring terminal 40. The whole body features of the person to be tracked are, for example, pre-extracted by the whole body feature extraction module 223 and registered in the file server 31 of the web server 30. The whole body matching module 231 transmits the calculated similarity to the web server 30 via in-server communication, Websocket, etc. When the video analysis function unit 20 is arranged independently of the server device SV, the whole body matching module 231 transmits the calculated similarity to the web server 30 via the network NET.

[0037] The face matching module 232 calculates the similarity between the face feature amounts input from the face feature extraction module 225 and the face feature amounts of the person to be tracked instructed from the monitoring terminal 40. The face feature amounts of the person to be tracked are, for example, extracted in advance by the face feature extraction module 225 and registered in the file server 31 of the Web server 30. The face matching module 232 transmits the calculated similarity to the Web server 30 by means of in-server communication, Websocket, or the like. When the video analysis functional unit 20 is arranged independently from the server device SV, the face matching module 232 transmits the calculated similarity to the Web server 30 via the network NET.

[0038] (1-3)Web server 30 FIG. 3 is a block diagram showing the software configuration of the Web server 30. As described above, the Web server 30 includes, as functional units, a file server 31, a DB server 32, a search human detection result storage function unit 33, a detection / tracking result determination function unit 34, and a Web application 35.

[0039] The file server 31 stores various data files. The file server 31 can include a past search data storage unit 311, a detection history data storage unit 312, and a management data storage unit 313.

[0040] The past search data storage unit 311 stores past search data, which is data acquired by each of the plurality of video analysis function units 20. FIG. 4 is a diagram showing an example of the past search data stored in the past search data storage unit 311. The past search data can be stored in the past search data storage unit 311 for each full-body image detected by the full-body detection module 221 of each video analysis function unit 20 from the frame image. The past search data can include the frame image 3111 acquired by the image acquisition module 21 of the video analysis function unit 20, the full-body image 3112 extracted by the full-body detection module 221, the full-body feature amount 3113 extracted by the full-body feature amount extraction module 223, and the face feature amount 3114 extracted by the face feature amount extraction module 225. These frame image 3111, full-body image 3112, full-body feature amount 3113, and face feature amount 3114 can be associated, for example, by storing them in the same path or attaching the same character string to a part of the file name.

[0041] The detection history data storage unit 312 stores detection history data, which is data regarding each person detected as a specific surveillance target such as a suspicious person, a lost child, or a regular customer. FIG. 5 is a diagram showing an example of the detection history data of the detected person stored in the detection history data storage unit 312. The detection history data can include the detected face image 3121 of the detected person and the face feature amount 3122 corresponding to the detected face image 3121. These detected face image 3121 and face feature amount 3122 can be associated, for example, by storing them in the same path or attaching the same character string to a part of the file name.

[0042] Although not particularly shown in the drawings, data regarding the person to be monitored is pre-stored in the file server 31. This data regarding the person to be monitored can be stored in the file server 31, for example, by inputting the face image of the person to be monitored from the monitoring terminal 40 to the web server 30 via the network NET. Then, the web server 30 inputs the face image to the face feature extraction module 225 of one video analysis function unit 20 through in-server communication, Websocket, etc., and can obtain the face features from the face feature extraction module 225 and store them in the file server 31. Also, via the network NET, a plurality of full-body images 3112 of people stored in the past search data storage unit 311 can be viewed on the web browser 41 of the monitoring terminal 40, and by designating an arbitrary full-body image 3112 from the monitoring terminal 40, it is also possible to designate that person as the person to be monitored. In this case, the web server 30 inputs the designated full-body image 3112 to the face detection module 224 of one video analysis function unit 20 through in-server communication, Websocket, etc., and can obtain a face image from the face detection module 224 and store it in the file server 31 as the face image of the person to be monitored. Also, the web server 30 can obtain the face features 3114 from the past search data having the designated full-body image 3112 stored in the past search data storage unit 311 and store them in the file server 31.

[0043] Also, when storing data regarding the person to be monitored in the file server 31, a monitored person table describing information about the person to be monitored is also stored in the DB server 32. In this monitored person table, a suspicious person ID which is identification information for uniquely identifying the person to be monitored, a suspicious person name which is the name of the person to be monitored, a risk type indicating for what reason the person is being monitored, a description text of physical characteristics, behavior patterns, precautions, etc. regarding the person to be monitored, etc. can be described.

[0044] The detected face image 3121 in the detection history data can be stored in the detection history data storage unit 312 when the Web server 30 detects a person under surveillance by comparing data related to the person under surveillance from, for example, the surveillance images of any of the surveillance cameras 10. That is, the Web server 30 inputs the full-body image obtained from the surveillance image and stored in the past search data storage unit 311 into the face detection module 224 of one video analysis function unit 20 through in-server communication, Websocket, etc., obtains a face image from the face detection module 224, and can store it in the detection history data storage unit 312 as the detected face image 3121. This one video analysis function unit 20 may be the video analysis function unit 20 corresponding to the surveillance camera 10 that has obtained the surveillance image in which the person under surveillance is detected. When the face detection module 224 of the video analysis function unit 20 is buffering the extracted face image, the Web server 30 may also obtain the detected face image 3121 by requesting the face image buffered in the face detection module 224.

[0045] The management data storage unit 313 stores management data that is data related to each person to be tracked. FIG. 6 is a diagram showing an example of the management data for each person to be tracked stored in this management data storage unit 313. The management data for each person to be tracked can be stored in the management data storage unit 313 in response to a designation from the surveillance terminal 40. The management data can include a registered face image 3131 that is the face image of the designated person to be tracked, a face feature amount 3132 that is a feature extracted from the registered face image 3131, a registered full-body image 3133 that is the full-body image of the person to be tracked, and a full-body feature amount 3134 that is a feature extracted from the registered full-body image 3133. These registered face image 3131, face feature amount 3132, registered full-body image 3133, and full-body feature amount 3134 can be associated, for example, by storing them in the same path or attaching the same character string to a part of the file name.

[0046] For example, when a person under surveillance is detected, the Web server 30 causes the surveillance browser 41 of the surveillance terminal 40 to display the detected face image 3121 in the detection history data for the person under surveillance stored in the detection history data storage unit 312 for viewing by the surveillant. When a designation is made from the surveillance terminal 40 to set the person as a tracking target, the Web server 30 stores the detected face image 3121 in the detection history data as a registered face image 3131 in this management data. Further, the Web server 30 can store the face feature amount 3122 in the detection history data as a face feature amount 3132 in this management data. Also, the Web server 30 acquires the full-body image that is the source of the detected face image 3121 in the designated detection history data from the past search data stored in the past search data storage unit 311, and also acquires the full-body feature amount from the past search data, and stores them as a registered full-body image 3133 and a full-body feature amount 3134 in this management data.

[0047] Alternatively, the Web server 30 may cause the Web browser 41 of the monitoring terminal 40 to view the full-body images 3112 of a plurality of persons stored in the past search data storage unit 311, receive a designation of an arbitrary full-body image from the monitoring terminal 40, and set the person as a designated tracking target. In this case, the Web server 30 inputs the designated full-body image 3112 to the face detection module 224 of one video analysis function unit 20 by means of in-server communication, Websocket, or the like, obtains a face image from the face detection module 224, and can store it as a registered face image 3131 in the management data storage unit 313. Also in this case, for the face feature amount 3132, the registered full-body image 3133, and the full-body feature amount 3134, those stored in the past search data storage unit 311 may be obtained and stored. Alternatively, for the face image and the full-body image of the tracking target, similar to the face image of the monitored person, they may be input to the Web server 30 from the monitoring terminal 40 via the network NET and stored as the registered face image 3131 and the registered full-body image 3133 in the management data storage unit 313. In this case, the Web server 30 may send the input face image and full-body image to the video analysis function unit 20 by means of in-server communication, Websocket, or the like, and obtain the face feature amount 3132 and the full-body feature amount 3134.

[0048] The DB server 32 stores various data tables. The DB server 32 may include a camera information table storage unit 321, a past search data table storage unit 322, a detection history table storage unit 323, a management table storage unit 324, a tracking table storage unit 325, and the like.

[0049] The camera information table storage unit 321 is provided corresponding to each of the plurality of surveillance cameras 10, and stores in advance a camera information table 3211 in which various information regarding the corresponding surveillance camera 10 is described. FIG. 7 is a diagram showing an example of the content described in the camera information table 3211 for each surveillance camera 10 stored in this camera information table storage unit 321. In the camera information table 3211, for example, camera ID, camera name, aspect ratio, camera position X, camera position Y, camera angle, face matching thresholds 1 to 3, full body matching thresholds 1 to 3, and so on are described.

[0050] Here, the camera ID is identification information that uniquely identifies the corresponding surveillance camera 10. The camera name is the name of the corresponding surveillance camera 10. This can be a name associated with the installation position of the corresponding surveillance camera 10 in a large facility. The aspect ratio is the aspect ratio of the image acquired by the corresponding surveillance camera 10. The camera position X and the camera position Y are the XY coordinates in the large facility indicating the installation position of the corresponding surveillance camera 10. The camera angle indicates the installation orientation of the corresponding surveillance camera 10.

[0051] Also, the face matching thresholds 1 to 3 and the full body matching thresholds 1 to 3 are thresholds for comparing with the similarity calculated by the full body matching module 231 and the face matching module 232 of the video analysis function unit 20. The face matching threshold 1 is the first threshold corresponding to the face image similarity for classifying whether the face image of a person extracted from the surveillance image of the corresponding surveillance camera 10 is the face image of the surveillance target person or the tracking target person. When the face image similarity calculated by the face matching module 232 is greater than this first threshold, the face matching threshold 1, the Web server 30 can determine that the face image of the person extracted from the surveillance image of the corresponding surveillance camera 10 is the face image of the surveillance target person or the tracking target person. The face matching thresholds 2 and 3 correspond to face image similarities greater than the face matching threshold 1, and are the second thresholds for classifying the extracted face image of the person into multiple stages when the face image of the person is the face image of the surveillance target person or the tracking target person. That is, the magnitudes of these thresholds have the relationship of [face matching threshold 1 < face matching threshold 2 < face matching threshold 3]. Similarly, the full body matching threshold 1 is the first threshold corresponding to the full body image similarity for classifying whether the full body image of a person extracted from the surveillance image of the corresponding surveillance camera 10 is the full body image of the surveillance target person or the tracking target person. When the full body image similarity calculated by the full body matching module 231 is greater than this full body matching threshold 1, the Web server 30 can determine that the full body image of the person extracted from the surveillance image of the corresponding surveillance camera 10 is the full body image of the surveillance target person or the tracking target person. The full body matching thresholds 2 and 3 correspond to full body image similarities greater than the full body matching threshold 1, and are the second thresholds for classifying the extracted full body image of the person into multiple stages when the full body image of the person is the full body image of the surveillance target person or the tracking target person. That is, the magnitudes of these thresholds have the relationship of [full body matching threshold 1 < full body matching threshold 2 < full body matching threshold 3].

[0052] The values of the respective thresholds are generally determined by the matching models used by the face matching module 232 in the monitoring / tracking execution function unit 23 of the video analysis function unit 20 that calculates the similarity and the matching model used by the full body matching module 231. Since each threshold depends on the type, performance, installation conditions, etc. of the monitoring camera 10 itself, it is adjusted to an appropriate value and described in the camera information table 3211 during the period from when the monitoring camera 10 is installed until the actual operation starts. Note that the face matching thresholds 1 to 3 and the full body matching thresholds 1 to 3 may be separately described in the camera information table 3211 as different values for the monitored person and the tracked person. Also, of course, the respective thresholds are not limited to three levels. That is, it is sufficient to have at least one second threshold.

[0053] The past search data table storage unit 322 stores a past search data table 3221 in which various data related to the past search data is described, corresponding to each of the past search data stored in the past search data storage unit 311. When the web server 30 stores past search data in the past search data storage unit 311, it creates this past search data table 3221 and stores it in the past search data table storage unit 322. FIG. 8 is a diagram showing an example of the description content of the past search data table 3221 stored in the past search data table storage unit 322. In the past search data table 3221, for example, detection ID, detection date and time, detection camera ID, tracking ID, frame ID, detection coordinate information X, detection coordinate information Y, detection coordinate information Width, detection coordinate information Height, etc. are described.

[0054] Here, the detection ID is identification information assigned to each full-body image 3112 in the frame image 3111 detected by the full-body detection module 221 of the video analysis function unit 20. The detection date and time is the date and time when the full-body detection module 221 detected the full-body image 3112. The detection camera ID is the camera ID of the surveillance camera 10 that acquired the frame image 3111. The tracking ID is identification information for associating full-body images 3112 of the same person across multiple frame images. The frame ID is identification information for uniquely identifying the frame image 3111. The detection coordinate information X, the detection coordinate information Y, the detection coordinate information Width, and the detection coordinate information Height are information indicating the region position of the full-body image 3112 in the frame image 3111 detected by the region tracking module 222, and indicate, for example, the XY coordinates in the frame image at the upper left corner of the full-body image 3112 and the image width and image height from there.

[0055] Note that the Web server 30 can assign the tracking ID in the past search data table 3221, for example, as follows. The Web server 30 compares the region position of the full-body image given from the human detection information extraction function unit 22 of the video analysis function unit 20 with the region positions of the respective full-body images in the past search data table 3221 in the previous frame image of the surveillance camera 10 stored in the past search data table storage unit 322, and determines the identity of the person in the full-body image by considering overlapping regions and the like. If the same person does not appear in the previous frame image, the Web server 30 assigns a new tracking ID. If the same person appears in the previous frame image, the Web server 30 inherits the tracking ID from the past search data table 3221 corresponding to the full-body image 3112 of that person. Note that the Web server 30 may determine the identity of the person by comparing the full-body feature amounts instead of the position of the full-body image, and determine whether a new tracking ID needs to be assigned.

[0056] The detection history table storage unit 323 stores a detection history table 3231 in which various data regarding the detection history data is described, corresponding to each of the detection history data stored in the detection history data storage unit 312. When storing the detection history data in the detection history data storage unit 312, the web server 30 creates this detection history table 3231 and stores it in the detection history table storage unit 323. FIG. 9 is a diagram showing an example of the description content of the detection history table 3231 stored in this detection history table storage unit 323. In the detection history table 3231, for example, a detection ID, a detection date and time, a detection camera ID, a suspicious person ID, a face authentication score, a face horizontal angle, a face vertical angle, a tracking ID, and so on are described.

[0057] Here, the detection ID, the detection date and time, the detection camera ID, and the tracking ID are as described for the past search data table 3221. The suspicious person ID is the suspicious person ID of the detected surveillance target person described in a surveillance target person table (not shown). The face authentication score is the similarity with the face image of the surveillance target person calculated by the face matching module 232 of the video analysis function unit 20. This similarity stores the real value of the similarity calculated by the face matching module 232, but can be rewritten into a three-level face image similarity label ID for display of the detection alert screen on the web browser 41 as described later. The face horizontal angle and the face vertical angle indicate the orientation of the face in the detected face image 3121 of the corresponding detection history data.

[0058] The management table storage unit 324 stores a management table 3241 in which various data related to the management data is described, corresponding to each of the management data stored in the management data storage unit 313. When the Web server 30 stores management data in the management data storage unit 313, it creates this management table 3241 and stores it in the management table storage unit 324. FIG. 10 is a diagram showing an example of the description content of the management table 3241 stored in this management table storage unit 324. In the management table 3241, for example, a suspicious person ID, a suspicious person name, a risk level type, a description, a query face image path, a face feature amount file path, a monitoring status, a pinning flag, a long-term stay flag, and so on are described.

[0059] Here, the suspicious person ID, the suspicious person name, the risk level type, and the description are transcribed with information about the monitored person designated as the tracking target, which is described in a monitored person table (not shown). The query face image path and the face feature amount file path indicate the save paths of the registered face image 3131 and the face feature amount 3132 of the management data in the management data storage unit 313. The monitoring status is attribute information arbitrarily set from the monitoring terminal 40 for the tracking target other than the risk level type. The pinning flag is a flag set to indicate that the person has been designated as the tracking target. The long-term stay flag is a flag set when a person who has not been designated as the tracking target has stayed in the facility for a specified time or longer. That is, when there is a person who is not outside the monitoring target but has stayed for a long time, the Web server 30 can also store the face image and the face feature amount of that person in the management data storage unit 313 and store the management table in the management table storage unit 324. As a result, it becomes possible to discover new suspicious persons and candidates for new customers.

[0060] The tracking table storage unit 325 stores a tracking table in which the similarity of a person extracted from a surveillance image of any one of the surveillance cameras 10, which is determined by the web server 30 to be a full-body image or a face image of a person to be tracked, based on the full-body image similarity or the face image similarity calculated by the full-body matching module 231 or the face matching module 232 of the video analysis function unit 20, is described. FIG. 11 is a diagram showing an example of the description content of the tracking table 3251 stored in the tracking table storage unit 325. The tracking table 3251 includes records consisting of a detection ID, a full-body image similarity, and a face image similarity for each suspicious person ID of the person to be tracked, and each record is added every time the web server 30 determines that it is a full-body image or a face image of the person to be tracked. The full-body image similarity and the face image similarity store the real values of the similarities calculated by the full-body matching module 231 and the face matching module 232 of the video analysis function unit 20, but can be rewritten into three-level full-body image similarity label IDs and face image similarity label IDs for the display of the tracking alert screen on the web browser 41, as will be described later.

[0061] The search person detection result storage function unit 33 and the detection / tracking result determination function unit 34 are provided as backend functions of the web server. The search person detection result storage function unit 33 receives the outputs of the respective function units of the person detection information extraction function unit 22 of the video analysis function unit 20, and creates a search data table to be stored in the past search data table storage unit 322. The detection / tracking result determination function unit 34 receives the outputs of the respective function units of the monitoring / tracking execution function unit 23 of the video analysis function unit 20, detects that a person identical to the pre-stored face image of the monitoring target person appears in the monitoring image of any of the monitoring cameras 10, creates a detection history table to be stored in the detection history table storage unit 323, and stores the detected face image 3121 and face feature amount 3122 of that person extracted by the respective function units of the person detection information extraction function unit 22 in the detection history data storage unit 312. The detection / tracking result determination function unit 34 also receives the outputs of the respective function units of the monitoring / tracking execution function unit 23 of the video analysis function unit 20, detects that a person identical to the full-body image and / or face image of the tracking target person designated by the monitor appears in the monitoring image of any of the monitoring cameras 10, and adds a record to the tracking table of the tracking target person stored in the tracking table storage unit 325 in response to the detection.

[0062] The web application 35 is an application program that performs various processes in response to requests from the web browser 41 of the monitoring terminal 40 and creates browsing data indicating the processing results. The web application 35 provides functions as a login function unit 351, a monitoring function unit 352, a tracking function unit 353, and a past search function unit 354 to the web server 30.

[0063] The login function unit 351 receives a login from the web browser 41, performs authentication, and permits a legitimate user to use the functions provided by the web application 35.

[0064] When the detection / tracking result determination function unit 34 detects a person identical to the face image of the person under surveillance, the monitoring function unit 352 generates an alert and presents it to the web browser 41. To this end, the monitoring function unit 352 includes a detection alert output function unit 3521 that generates a detection alert screen for presentation to the web browser 41. The detection alert screen is an output screen for presenting an image of a person detected as similar to the person under surveillance, together with a stage display indicating the attention stage based on the similarity degree.

[0065] The tracking function unit 353 presents to the web browser 41 the tracking results of real-time tracking of a person across a plurality of surveillance cameras 10 using the face image and full-body image of a person identical to the face image of the person under surveillance detected by the detection / tracking result determination function unit 34. To this end, the tracking function unit 353 includes an image registration function unit 3531 and a tracking alert output function unit 3532. The image registration function unit 3531 receives the designation of the face image of the tracking target person to be tracked by a designation operation from the monitor on the web browser 41, stores the registered face image 3131 and face feature amount 3132 in the management data storage unit 313, and creates and stores a management table 3241 in the management table storage unit 324. The tracking alert output function unit 3532 presents to the web browser 41 a tracking alert screen, which is an output screen for presenting an image of a person detected as similar to the tracking target person, together with a stage display indicating the attention stage based on the similarity degree, when the detection / tracking result determination function unit 34 detects a person identical to the full-body image and / or face image of the tracking target person.

[0066] The past search function unit 354 receives the designation of time and location from the web browser 41, displays a list of persons who visited the designated time and location, and accepts the designation of the person to be searched from among them. Then, the past search function unit 354 presents to the web browser 41 the result of searching for similar persons stored in the past search data storage unit 311 by the detection / tracking result determination function unit 34 from the face image and full-body image of the designated person.

[0067] FIG. 12 is a block diagram showing the hardware configuration of the web server 30. The web server 30 includes, for example, a hardware processor 301A such as a CPU (Central Processing Unit), a program memory 301B, a storage device 302, and a communication interface device 303. The program memory 301B, the storage device 302, and the communication interface device 303 are connected to the hardware processor 301A via a bus 304.

[0068] The communication interface device 303 includes, for example, one or more wired or wireless communication interface units, and enables the transmission and reception of various information between the video analysis functional unit 20 and the monitoring terminal 40 in accordance with the communication protocol used in the network NET.

[0069] The program memory 301B uses, as a storage medium, a combination of a non-volatile memory such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive) that can be written to and read from at any time, and a non-volatile memory such as a ROM (Read Only Memory). By being executed by a hardware processor 301A such as a CPU, it stores programs necessary for executing various control processes according to an embodiment of this invention. That is, the hardware processor 301A can function as a search human detection result storage function unit 33 and a detection / tracking result determination function unit 34 as shown in FIG. 3 by reading and executing the programs stored in the program memory 301B. Note that these processing function units may be realized by separate hardware processors. That is, the Web server 30 may include a plurality of hardware processors. Also, at least a part of these processing function units may be realized in the form of other various hardware circuits including integrated circuits such as an ASIC (Application Specific Integrated Circuit), an FPGA (field-programmable gate array), and a GPU (Graphics Processing Unit). Further, the programs stored in the program memory 301B can include the programs of a Web application 35 as shown in FIG. 3.

[0070] The storage device 302 uses, as a storage medium, a combination of a non-volatile memory such as an HDD or an SSD that can be written to and read from at any time, and a volatile memory such as a RAM (Random Access Memory), and is used to store various data acquired and created in the process of performing person tracking processing. Also, a file server 31 and a DB server 32 as shown in FIG. 3 can be configured in the storage device 302.

[0071] (2) Operation Next, the operation of the person tracking system configured as described above will be explained.

[0072] (2-1) Monitoring operation When the web browser 41 is launched on the monitoring terminal 40 and an access operation to the web server 30 is performed, the web server 30 performs an authentication operation by the login function unit 351 provided by the web application 35. If it is confirmed that the user is a legitimate user, the web server 30 generates a monitoring screen for viewing on the web browser 41 by the monitoring function unit 352 provided by the web application 35 and transmits it to the web browser 41. The monitoring screen can be generated as follows.

[0073] Each of the plurality of monitoring cameras 10 periodically acquires a monitoring image and inputs the acquired monitoring image to the corresponding video analysis function unit 20. The image acquisition module 21 of the video analysis function unit 20 acquires a frame image that is a monitoring image from the corresponding monitoring camera 10, and the human detection information extraction function unit 22 extracts the full-body image of the person shown in the frame image, the region position of the full-body image, the full-body feature amount, and the face feature amount. Then, the video analysis function unit 20 transmits those frame images, full-body images, region positions of the full-body images, full-body feature amounts, and face feature amounts to the web server 30 via in-server communication, Websocket, or the like.

[0074] The web server 30 accumulates the frame images, full-body images, full-body feature amounts, and face feature amounts transmitted from each of the plurality of video analysis function units 20 as past search data in the past search data storage unit 311 of the file server 31. At this time, the search human detection result storage function unit 33 of the web server 30 creates a past search data table including the region positions of the full-body images transmitted from each of the plurality of video analysis function units 20 and stores it in the past search data table storage unit 322 of the DB server 32. In this way, every time a monitoring image is acquired by each monitoring camera 10, the past search data is accumulated in the past search data storage unit 311, and the past search data table associated with those past search data is accumulated in the past search data table storage unit 322.

[0075] The monitoring function unit 352 can generate a monitoring screen by arranging the frame images from each monitoring camera 10 that are thus accumulated in the past search data storage unit 311 on one screen. By viewing this monitoring screen displayed on the web browser 41 of the monitoring terminal 40, a monitor such as a security guard can grasp the real-time situation of each part within the facility.

[0076] Also, in the person tracking system, in parallel with the real-time situation monitoring of each part, a visiting detection operation for the monitored person is performed. This monitored person visiting detection operation includes monitoring processing by the monitoring / tracking execution function unit 23 of each video analysis function unit 20, detection result determination processing by the detection / tracking result determination function unit 34, and detection alert output processing by the detection alert output function unit 3521 of the monitoring function unit 352.

[0077] FIG. 13 is a flowchart showing the processing procedure of the monitoring processing by the monitoring / tracking execution function unit 23. The monitoring / tracking execution function unit 23 performs the processing shown in this flowchart every time an update of the monitored person is reported from the monitoring function unit 352 of the web server 30 via, for example, in-server communication or Websocket.

[0078] First, the monitoring / tracking execution function unit 23 accesses the file server 31 of the web server 30 via in-server communication or Websocket or the like, and acquires the monitoring face feature amounts, which are the face feature amounts of each of a plurality of monitored persons previously stored in the file server 31 (step S101).

[0079] Thereafter, the monitoring / tracking execution function unit 23 acquires the face image and its feature amount at time t from the person detection information extraction function unit 22 as the comparison face image and the comparison face feature amount (step S102). When the monitoring image of the corresponding monitoring camera 10 includes full-body images of a plurality of persons, the monitoring / tracking execution function unit 23 acquires the comparison face image and the comparison face feature amount for each person.

[0080] Then, the monitoring / tracking execution functional unit 23 calculates, by means of the face matching module 232, the face image similarity, which is the similarity with the monitoring face feature amounts, for each of the face feature amounts for verification from the person detection information extraction functional unit 22 (step S103).

[0081] After that, the monitoring / tracking execution functional unit 23 transmits the set of the calculated face image similarities of each monitored person and the face image for verification to the detection / tracking result determination functional unit 34 of the web server 30 via in-server communication, Websocket, or the like (step S104).

[0082] Then, after adding the time interval t1 to the time t, that is, after updating to the next processing time (step S105), the monitoring / tracking execution functional unit 23 repeats the above processing from the above step S102. Here, the time interval t1 can be an arbitrary time interval, for example, several seconds. Alternatively, the time interval t1 may be a time interval that is an integral multiple of the acquisition interval of the monitoring images of the monitoring camera 10.

[0083] FIG. 14 is a flowchart showing the processing procedure of the detection result determination process by the detection / tracking result determination functional unit 34 of the web server 30. The detection / tracking result determination functional unit 34 can periodically perform the processing shown in this flowchart, for example, every time interval t1 in the monitoring / tracking execution functional unit 23 of the video analysis functional unit 20.

[0084] First, the detection / tracking result determination functional unit 34 determines whether or not it has received the set of the similarity and the face image for verification from the monitoring / tracking execution functional unit 23 (step S201). If it determines that it has not received the set of the similarity and the image for verification, the detection / tracking result determination functional unit 34 ends this detection / tracking result determination process.

[0085] On the other hand, when it is determined that a set of similarity and verification face images has been received, the detection / tracking result determination functional unit 34 acquires the face verification threshold 1 (step S202). That is, the detection / tracking result determination functional unit 34 acquires the face verification threshold 1 from the camera information table 3211 regarding the surveillance camera 10 corresponding to the video analysis functional unit 20, which is the transmission source of the set of the similarity and verification face images, stored in the camera information table storage unit 321.

[0086] Then, the detection / tracking result determination functional unit 34 compares each of the received face image similarities with the acquired face verification threshold 1, and determines whether there is a face image similarity greater than the face verification threshold 1 (step S203). When it is determined that there is no face image similarity greater than the face verification threshold 1, the detection / tracking result determination functional unit 34 ends this detection / tracking result determination process.

[0087] On the other hand, when it is determined that there is a face image similarity greater than the face verification threshold 1, the detection / tracking result determination functional unit 34 discriminates that the person having the face feature amount is a surveillance target person. Therefore, in this case, the detection / tracking result determination functional unit 34 stores the detected face image 3121 and the face feature amount 3122 as detection history data in the detection history data storage unit 312, generates a detection history table 3231, and stores it in the detection history table storage unit 323 (step S204). The face authentication score of the generated detection history table 3231 will be described with a face image similarity having a value greater than the face verification threshold 1. Then, thereafter, the detection / tracking result determination functional unit 34 ends this detection / tracking result determination process.

[0088] FIG. 15 is a flowchart showing the processing procedure of the detection alert output process by the detection alert output functional unit 3521 of the surveillance functional unit 352. The detection alert output functional unit 3521 periodically performs the processing shown in this flowchart, for example, at intervals of time t1 in the monitoring / tracking execution functional unit 23 of the video analysis functional unit 20.

[0089] The detection alert output function unit 3521 first obtains face matching thresholds 1 to 3 from the camera information table 3211 for each monitoring camera 10 stored in the camera information table storage unit 321 (step S301).

[0090] Also, the detection alert output function unit 3521 obtains the information of the new detection history table 3231 from the detection history table storage unit 323 (step S302). That is, the detection alert output function unit 3521 extracts and obtains those in which the real number of face similarity rather than the face image similarity label ID is recorded as the face image score in the detection history table 3231.

[0091] Then, the detection alert output function unit 3521 determines whether the value of the face image similarity described as the face authentication score in the obtained new detection history table 3231 is greater than the face matching threshold 3 (step S303). If it is determined that the value of the face image similarity is greater than the face matching threshold 3, the detection alert output function unit 3521 rewrites the value of the face image similarity in the detection history table 3231 of the detection history table storage unit 323 from a real number to the face image similarity label ID, here ID = 3 (step S304).

[0092] On the contrary, if it is determined that the value of the face image similarity is not greater than the face matching threshold 3, the detection alert output function unit 3521 further determines whether the value of the face image similarity is greater than the face matching threshold 2 (step S305). If it is determined that the value of the face image similarity is greater than the face matching threshold 2, the detection alert output function unit 3521 rewrites the value of the face image similarity in the detection history table 3231 of the detection history table storage unit 323 from a real number to the face image similarity label ID, here ID = 2 (step S306).

[0093] Also, when it is determined that the value of the face image similarity is not greater than the face matching threshold 2, the detection alert output function unit 3521 rewrites the value of the face image similarity in the detection history table 3231 of the detection history table storage unit 323 from a real number to a face image similarity label ID, here ID = 1 (step S307). This is because the detection history table 3231 is not created if the value of the face image similarity is not greater than the face matching threshold 1. Therefore, if the value of the face image similarity is not greater than the face matching threshold 2, it is certain that the value of the face image similarity is greater than the face matching threshold 1.

[0094] As described above, if the value of the face image similarity in the detection history table 3231 is rewritten to the face image similarity label ID, the detection alert output function unit 3521 creates a new detected person card that is a screen image (step S308). This detected person card is a screen image including the detected face image 3121 stored as detection history data in the detection history data storage unit 312 and a caution level display indicating the caution level based on the face image similarity label ID in the corresponding detection history table 3231. Details of this detected person card will be described later.

[0095] The detection alert output function unit 3521 updates the detection alert screen for display on the Web browser 41 with the newly created detected person card (step S309). Then, the detection alert output function unit 3521 transmits this updated detection alert screen to the Web browser 41 via the network NET and displays it there (step S310). In the Web browser 41, the detection alert screen is displayed in an alert screen display area provided in a part of the monitoring screen or in a window separate from the monitoring screen. Then, the detection alert output function unit 3521 ends this detection alert output process.

[0096] FIG. 16 is a diagram showing an example of a detection alert screen 42 displayed on a web browser 41. The detection alert screen 42 includes detection person cards 421 for each detected person. This detection person card 421 can include, as its display content, a monitored person face image 4211, a detected face image 4212, a warning message 4213, a suspicious person ID 4214, a warning level display 4215, detection information 4216, and a tracking start button 4217.

[0097] Here, the monitored person face image 4211 is a face image of the monitored person previously stored in the file server 31, and the detected face image 4212 is the detected face image 3121 stored in the detection history data storage unit 312 of the person detected as the monitored person. The warning message 4213 is a message for prompting the attention of the monitor, and the identification display of the blinking indicator light may be performed. The suspicious person ID 4214 is identification information transferred from the detection history table 3231.

[0098] The warning level display 4215 indicates the similarity between the face image of the monitored person and the face image of the detected person. That is, the similarity is shown not as a numerical value but as a plurality of levels based on the face image similarity label ID in the detection history table 3231. Here, it is shown as a three-level warning level display 4215. In this warning level display 4215, the number of the identified squares represented by hatching in FIG. 16 corresponds to the face image similarity label ID. For example, in the detection person card 421 of the person whose suspicious person ID 4214 is "00098842", since the face image similarity label ID in the detection history table 3231 is "3", three squares are identified and displayed in the warning level display 4215. In the detection person card 421 of the person whose suspicious person ID 4214 is "00059820", since the face image similarity label ID in the detection history table 3231 is "2", two squares are identified and displayed.

[0099] The detection information 4216 includes detection camera location information based on the detection camera ID in the past search data table 3221 or the detection history table 3231, and detection date and time information based on the detection date and time as well. The tracking start button 4217 is a button to be pressed when tracking the person.

[0100] Note that each time a new person is detected as a person to be monitored, this detected person card 421 is shifted downward so that the old detected person card 421 is displayed at the top of the detection alert screen 42. A new detected person card 421 is not added for the same person. By selecting one detected person card 421, the history of the person stored in the detection history data storage unit 312 and the detection history table storage unit 323 is displayed in a list.

[0101] (2-2) Tracking operation When the monitor presses the tracking start button 4217 on the detection alert screen 42, the Web server 30 uses the tracking function unit 353 provided by the Web application 35 to utilize the video analysis function unit 20 and the detection / tracking result determination function unit 34 to present the tracking result of tracking the person to the Web browser 41. This tracking operation includes an image registration process by the image registration function unit 3531 of the tracking function unit 353, a tracking process by the monitoring / tracking execution function unit 23 of the video analysis function unit 20, a tracking result determination process by the detection / tracking result determination function unit 34, and a tracking alert output process by the tracking alert output function unit 3532 of the tracking function unit 353.

[0102] Figure 17 is a flowchart showing the processing procedure of the image registration process by the image registration function unit 3531 of the tracking function unit 353. The image registration function unit 3531 periodically performs the processing shown in this flowchart, for example, in synchronization with the acquisition interval of the monitoring images of the monitoring camera 10.

[0103] The image registration function unit 3531 first determines whether or not a pressing operation of the tracking start button 4217 on the detection alert screen 42 displayed on the Web browser 41 by the monitor has been performed on the monitoring terminal 40 (step S401). If it is determined that the pressing operation of the tracking start button 4217 has not been performed, the image registration function unit 3531 ends this image registration process.

[0104] On the other hand, if it is determined that the pressing operation of the tracking start button 4217 has been performed, the image registration function unit 3531 acquires the face image of the person as a tracking face image (step S402). That is, the image registration function unit 3531 acquires the detected face image 3121 from the detection history data storage unit 312 as a tracking face image based on the tracking ID in the detection history table storage unit 323 of the detection alert screen 42 having the suspicious person ID. For example, the image registration function unit 3531 can acquire the detected face image 3121 specified by the detection ID described in the detection history table 3231 having the tracking ID and having a detection date and time within a specific period, for example, the latest one. The specific period is not limited to the latest, and may be the oldest, the latest or the oldest within a specified period, or a season (such as spring, summer, autumn, or winter). Alternatively, the image registration function unit 3531 may generate a selection screen including the detected face images 3121 specified by the detection IDs described in each of the detection history tables 3231 having the tracking ID in a list format, display it on the Web browser 41, and let the monitor select the detected face image 3121 to be used as the tracking face image. Then, the image registration function unit 3531 stores the acquired tracking face image in the management data storage unit 313 as a registered face image 3131. In addition, the image registration function unit 3531 creates a management table 3241 based on the information about the corresponding monitoring target person described in the monitoring target person table (not shown) previously stored in the DB server 32, and stores it in the management table storage unit 324. The management table 3241 includes a query face feature path indicating the storage path of the registered face image 3131 in the management data storage unit 313.

[0105] Then, the image registration function unit 3531 acquires, as a tracking full-body image, the full-body image corresponding to the detected face image 3121 acquired from the detection history data storage unit 312 as the tracking face image (step S403). That is, the image registration function unit 3531 acquires the full-body image 3112 from the past search data storage unit 311 as the tracking full-body image based on the detection ID described in the detection history table 3231 for the detected face image 3121 acquired as the tracking face image. The image registration function unit 3531 stores the acquired tracking full-body image in the management data storage unit 313 as the registered full-body image 3133. Also, the image registration function unit 3531 adds a query full-body feature path indicating the storage path of the registered full-body image 3133 in the management data storage unit 313 to the management table 3241 stored in the management table storage unit 324.

[0106] Furthermore, the image registration function unit 3531 acquires the tracking face image feature amount and the tracking full-body image feature amount (step S404). That is, the image registration function unit 3531 acquires the face feature amount 3122 from the detection history data storage unit 312 as the tracking face feature amount based on the detection ID, and also acquires the full-body feature amount 3113 from the past search data storage unit 311 as the tracking full-body feature amount. The image registration function unit 3531 stores the acquired tracking face feature amount and tracking full-body feature amount in the management data storage unit 313 as the face feature amount 3132 and the full-body feature amount 3134. Also, the image registration function unit 3531 adds a face feature amount file path and a full-body feature amount file path indicating the feature amount storage paths to the management table 3241 stored in the management table storage unit 324.

[0107] Then, the image registration function unit 3531 transmits, as tracking images, the registered face image 3131 and the registered full-body image 3133 stored in the management data storage unit 313, and transmits the face feature amount 3132 and the full-body feature amount 3134 as tracking feature amounts to each of the plurality of video analysis function units 20 via in-server communication, Websocket, etc. (step S405). After that, the image registration function unit 3531 ends this image registration process.

[0108] Figure 18 is a flowchart showing the processing procedure of the tracking process by the monitoring / tracking execution function unit 23 of the video analysis function unit 20. The monitoring / tracking execution function unit 23 performs the processing shown in this flowchart every time it receives a tracking image and feature amounts from the tracking function unit 353 and the image registration function unit 3531 of the web server 30 via, for example, in-server communication or Websocket.

[0109] First, the monitoring / tracking execution function unit 23 acquires the tracking image and feature amounts received from the image registration function unit 3531 of the web server 30 (step S111).

[0110] After that, the monitoring / tracking execution function unit 23 acquires, from the human detection information extraction function unit 22, the full-body image and its feature amounts at time T as the comparison full-body image and the comparison full-body feature amounts (step S112). When the monitoring image of the corresponding monitoring camera 10 includes full-body images of a plurality of persons, the monitoring / tracking execution function unit 23 acquires the comparison full-body image and the comparison full-body feature amounts for each person.

[0111] Then, the monitoring / tracking execution function unit 23 calculates, by means of the full-body comparison module 231, the full-body image similarity, which is the similarity between the comparison full-body feature amounts from the human detection information extraction function unit 22 and the tracking full-body feature amounts from the image registration function unit 3531, for each of them (step S113).

[0112] In addition, the monitoring / tracking execution function unit 23 acquires, from the human detection information extraction function unit 22, the face image and its feature amounts at time T as the comparison face image and the comparison face feature amounts (step S114). When the monitoring image of the corresponding monitoring camera 10 includes face images of a plurality of persons, the monitoring / tracking execution function unit 23 acquires the comparison face image and the comparison face feature amounts for each person. Note that there are cases where, depending on the orientation of a person, a full-body image can be extracted but a face image cannot. Therefore, the number of comparison full-body images acquired in step S112 above and the number of comparison face images acquired in this step S114 do not necessarily match. Also, there may be cases where no face image can be extracted for any person and no comparison face image (and comparison face feature amounts) is acquired.

[0113] Then, the monitoring / tracking execution function unit 23 calculates, by means of the face matching module 232, the face image similarity, which is the similarity with the face feature amount for tracking from the image registration function unit 3531, for each of the face feature amounts for verification from the person detection information extraction function unit 22 (step S115).

[0114] After that, the monitoring / tracking execution function unit 23 transmits a set of the similarities for those full-body images and face images and the verification images to the detection / tracking result determination function unit 34 of the web server 30 via in-server communication, Websocket, etc. (step S116). Note that the set of the similarity and the verification images may include both the set of the full-body image similarity and the verification full-body image and the set of the face image similarity and the verification face image, or may only include the set of the full-body image similarity and the verification full-body image without the set of the face image similarity and the verification face image.

[0115] Then, the monitoring / tracking execution function unit 23 determines whether or not a pressing operation of the tracking stop button on the tracking alert screen displayed on the web browser 41 by the monitor is performed on the monitoring terminal 40 (step S117). Details of the tracking alert screen having the tracking stop button will be described later. If it is determined that the pressing operation of the tracking stop button has been performed, the monitoring / tracking execution function unit 23 ends this monitoring / tracking process.

[0116] If it is determined that the pressing operation of the tracking stop button has not been performed, the monitoring / tracking execution function unit 23 adds the time interval t1 to the time T, that is, after updating to the next processing time (step S118), and repeats the above processing from step S112. Here, the time interval t1 is as described above. Of course, a time interval different from this time interval t1 may be used.

[0117] FIG. 19 is a flowchart showing the processing procedure of the tracking result determination process by the detection / tracking result determination functional unit 34 of the Web server 30. The detection / tracking result determination functional unit 34 periodically performs the processes shown in this flowchart, for example, in synchronization with the processing interval of the tracking process of the monitoring / tracking execution functional unit 23 of the video analysis functional unit 20.

[0118] First, the detection / tracking result determination functional unit 34 determines whether or not it has received a set of similarity and comparison images from the monitoring / tracking execution functional unit 23 (step S221). If it is determined that the set of similarity and comparison images has not been received, the detection / tracking result determination functional unit 34 ends this tracking result determination process.

[0119] On the other hand, if it is determined that the set of similarity and comparison images has been received, the detection / tracking result determination functional unit 34 acquires the face matching threshold 1 and the full body matching threshold 1 (step S222). That is, the detection / tracking result determination functional unit 34 acquires the face matching threshold 1 and the full body matching threshold 1 from the camera information table 3211 regarding the monitoring camera 10 corresponding to the video analysis functional unit 20, which is the transmission source of the set of similarity and comparison images, stored in the camera information table storage unit 321.

[0120] Then, for each of the received full-body image similarities, the detection / tracking result determination function unit 34 compares it with the acquired full-body matching threshold 1 and determines whether it is greater than the full-body matching threshold 1 or there is a full-body image similarity (step S223). When it is determined that there is a full-body image similarity greater than the full-body matching threshold 1, the detection / tracking result determination function unit 34 determines that the person having the full-body feature amount is the tracking target person. Therefore, in this case, the detection / tracking result determination function unit 34 saves the determined large full-body image similarity together with the corresponding full-body image for verification received in step S221 in a predetermined storage area of the storage device 302, and registers the full-body image similarity in the tracking table 3251 (step S224). When registering this full-body image similarity, if the tracking table 3251 of the corresponding suspicious person ID is not yet stored in the tracking table storage unit 325, the tracking table 3251 is generated, and the generated tracking table 251 is stored in the tracking table storage unit 325. Also, when the tracking table 3251 of the corresponding suspicious person ID is already stored in the tracking table storage unit 325, instead of generating a new tracking table 3251, a record is added. The full-body image similarity registered in the tracking table 3251 is a full-body image similarity with a value greater than the full-body matching threshold 1.

[0121] After that, or when it is determined in step S223 that there is no overall image similarity greater than the overall matching threshold 1, the detection / tracking result determination functional unit 34 compares each of the received face image similarities with the obtained face matching threshold 1, and determines whether there is a face image similarity greater than the face matching threshold 1 (step S225). When it is determined that there is a face image similarity greater than the face matching threshold 1, the detection / tracking result determination functional unit 34 determines that the person having the face feature amount is the tracking target person. Therefore, in this case, the detection / tracking result determination functional unit 34 stores the determined large face image similarity together with the corresponding face image for verification received in step S221 in a predetermined storage area of the storage device 302, and registers the face image similarity in the tracking table 3251 (step S226). When registering this face image similarity, if the tracking table 3251 of the corresponding suspicious person ID is not yet stored in the tracking table storage unit 325, the tracking table 3251 is generated, and the generated tracking table 251 is stored in the tracking table storage unit 325. When the tracking table 3251 of the corresponding suspicious person ID is already stored in the tracking table storage unit 325, instead of generating a new tracking table 3251, a record is added. The face image similarity registered in the tracking table 3251 is an overall image similarity with a value greater than the face matching threshold 1.

[0122] After that, or when it is determined in step S225 that there is no face image similarity greater than the face matching threshold 1, the detection / tracking result determination functional unit 34 ends this tracking result determination process.

[0123] FIGS. 20A to 20C are flowcharts showing the processing procedure of the tracking alert output process by the tracking alert output functional unit 3532 of the tracking functional unit 353. The tracking alert output functional unit 3532 periodically performs the processing shown in this flowchart, for example, at time intervals t1 in the monitoring / tracking execution functional unit 23 of the video analysis functional unit 20.

[0124] First, the tracking alert output function unit 3532 determines whether a narrowing-down designation operation has been performed on the tracking alert screen displayed on the Web browser 41 by the monitor on the monitoring terminal 40 (step S501). If it is determined that the narrowing-down designation operation has been performed, the tracking alert output function unit 3532 proceeds to step S519, which will be described later.

[0125] On the other hand, if it is determined that the narrowing-down designation operation has not been performed, the tracking alert output function unit 3532 acquires the face matching thresholds 1 to 3 and the full-body matching thresholds 1 to 3 from the camera information table 3211 for each monitoring camera 10 stored in the camera information table storage unit 321 (step S502).

[0126] After that, the tracking alert output function unit 3532 acquires a new tracking result from the tracking table 3251 stored in the tracking table storage unit 325 (step S503). That is, the tracking alert output function unit 3532 extracts and acquires those in which the real numbers of the similarity rather than the image similarity ID are recorded as the full-body image similarity and the face image similarity in the tracking table 3251.

[0127] Then, the tracking alert output function unit 3532 determines whether the value of the face image similarity in the tracking table 3251 is greater than the face matching threshold 3 (step S504). If it is determined that the value of the face image similarity is greater than the face matching threshold 3, the tracking alert output function unit 3532 rewrites the value of the face image similarity in the tracking table 3251 of the tracking table storage unit 325 from a real number to the face image similarity label ID, here ID = 3 (step S505).

[0128] On the other hand, when it is determined that the value of the face image similarity is not greater than the face matching threshold 3, the tracking alert output function unit 3532 further determines whether the value of the face image similarity is greater than the face matching threshold 2 (step S506). When it is determined that the value of the face image similarity is greater than the face matching threshold 2, the tracking alert output function unit 3532 rewrites the value of the face image similarity in the tracking table 3251 of the tracking table storage unit 325 from a real number to a face image similarity label ID, here ID = 2 (step S507).

[0129] Also, when it is determined that the value of the face image similarity is not greater than the face matching threshold 2, the tracking alert output function unit 3532 further determines whether the value of the face image similarity is greater than the face matching threshold 1 (step S508). When it is determined that the value of the face image similarity is greater than the face matching threshold 1, the tracking alert output function unit 3532 rewrites the value of the face image similarity in the tracking table 3251 of the tracking table storage unit 325 from a real number to a face image similarity label ID, here ID = 1 (step S509).

[0130] In the process related to the detection operation in the monitoring function unit 352, only the face image similarity is used. Therefore, if the value of the face image similarity is not greater than the face matching threshold 2, it is certain that the value of the face image similarity is not greater than the face matching threshold 1. On the other hand, in the process related to the tracking operation of the tracking function unit 353, both the face image similarity and the whole body similarity are used, and there are cases where only one of the face image similarity and the whole body similarity is registered in the tracking table 3251. That is, when the value of the face image similarity is not greater than the face matching threshold 2, there are cases where a face image similarity value greater than the face matching threshold 1 is registered and cases where a face image similarity value equal to or less than the face matching threshold 1 is registered (or no face image similarity value is registered). Therefore, just because the value of the face image similarity is not greater than the face matching threshold 2, it cannot be determined that the value of the face image similarity is greater than the face matching threshold 1.

[0131] If the value of the face image similarity in the tracking table 3251 is rewritten as the face image similarity label ID as described above, or if it is determined in step S508 that the value of the face image similarity is not greater than the face matching threshold 1, the tracking alert output function unit 3532 proceeds to the determination operation of the full body image similarity.

[0132] That is, the tracking alert output function unit 3532 determines whether the value of the full body image similarity in the tracking table 3251 is greater than the full body matching threshold 3 (step S510). If it is determined that the value of the full body image similarity is greater than the full body matching threshold 3, the tracking alert output function unit 3532 rewrites the value of the full body image similarity in the corresponding tracking table 3251 of the tracking table storage unit 325 from a real number to the full body image similarity label ID, here ID = 3 (step S511).

[0133] On the other hand, if it is determined that the value of the full body image similarity is not greater than the full body matching threshold 3, the tracking alert output function unit 3532 further determines whether the value of the full body image similarity is greater than the full body matching threshold 2 (step S512). If it is determined that the value of the full body image similarity is greater than the full body matching threshold 2, the tracking alert output function unit 3532 rewrites the value of the full body image similarity in the corresponding tracking table 3251 of the tracking table storage unit 325 from a real number to the full body image similarity label ID, here ID = 2 (step S513).

[0134] Also, if it is determined that the value of the full body image similarity is not greater than the full body matching threshold 2, the tracking alert output function unit 3532 further determines whether the value of the full body image similarity is greater than the full body matching threshold 1 (step S514). If it is determined that the value of the full body image similarity is greater than the full body matching threshold 1, the tracking alert output function unit 3532 rewrites the value of the full body image similarity in the corresponding tracking table 3251 of the tracking table storage unit 325 from a real number to the full body image similarity label ID, here ID = 1 (step S515).

[0135] If the value of the full-body image similarity in the tracking table 3251 is rewritten to the full-body image similarity label ID as described above, or if it is determined in step S514 that the value of the full-body image similarity is not greater than the full-body matching threshold 1, the tracking alert output function unit 3532 creates a new tracking information card that is a screen image (step S308). This tracking information card is a screen image including the full-body image for verification and the face image for verification stored in a predetermined storage area of the storage device 302 in steps S224 and S226, and a caution level display indicating the caution level based on the full-body image similarity label ID and the face image similarity label ID in the corresponding tracking table 3251. Details of this tracking information card will be described later.

[0136] The tracking alert output function unit 3532 updates the tracking alert screen for display on the Web browser 41 with the newly created tracking information card (step S517). Then, the tracking alert output function unit 3532 transmits this updated tracking alert screen to the Web browser 41 via the network NET and displays it there (step S518). In the Web browser 41, the tracking alert screen is displayed in an alert screen display area provided in a part of the monitoring screen or in a window separate from the monitoring screen. Then, the tracking alert output function unit 3532 ends this tracking alert output process.

[0137] On the other hand, if it is determined in step S501 that a narrowing-down specification operation has been performed, the tracking alert output function unit 3532 acquires the specified narrowing-down level (step S519). This narrowing-down level is an index for narrowing down the tracking information cards displayed on the tracking alert screen.

[0138] The tracking alert output function unit 3532 extracts the suspicious person ID corresponding to the specified narrowing-down level from the tracking table 3251 stored in the tracking table storage unit 325 (step S520).

[0139] Then, based on the detection IDs of the suspicious person IDs extracted by it, the tracking alert output function unit 3532 creates the number of tracking information cards that can be displayed on the tracking alert screen, including the full-body image 3112 stored as past search data in the past search data storage unit 311 and the face image extracted from the full-body image 3112 by the human detection information extraction function unit 22 of the video analysis function unit 20 (step S521).

[0140] After that, similar to step S516 above, the tracking alert output function unit 3532 creates a new tracking information card (step S522), and similar to step S517 above, updates the tracking alert screen with the created tracking information card (step S523). Then, the tracking alert output function unit 3532 proceeds to the process of step S518, sends the updated tracking alert screen to the Web browser 41, and causes it to be displayed there.

[0141] FIG. 21 is a diagram showing an example of the tracking alert screen 43 displayed on the Web browser 41. The tracking alert screen 43 includes a tracking target person information area 431, a tracking information card 432, and a narrowing-down instruction area 433.

[0142] The tracking target person information area 431 is an area for displaying information about the tracking target person. Each time a tracking target person is added, it is additionally displayed at the topmost position of the tracking alert screen 43. The old tracking target person information area 431 is sequentially shifted downward. Here, the display content is different between the topmost, that is, the latest tracking target person information area 431 and the other tracking target person information areas 431. That is, the topmost tracking target person information area 431 includes a warning message 4311, a suspicious person ID 4312, a registered image 4313, tracking information 4314, and a tracking stop button 4315. In contrast, the tracking target person information areas 431 other than the topmost include only the registered image 4313 and the tracking information 4314.

[0143] Here, the alert message 4311 is a message for prompting the attention of the monitor, and for example, it can display the risk level type of the monitored person designated as the target to be tracked. This risk level type can be transferred from the management table 3241. The suspicious person ID 4312 is identification information transferred from the management table 3241. The registered image 4313 is the registered face image 3131 and the registered full-body image 3133 stored in the management data storage unit 313 of the target to be tracked. The tracking information 4314 is information regarding the target to be tracked, and the content of the tracking information 4314 in the top-level target to be tracked information area 431 is different from that in other target to be tracked information areas 431. That is, the tracking information 4314 in the top-level target to be tracked information area 431 includes the suspicious person ID, the risk level type, and the explanatory text transferred from the management table 3241. The tracking information 4314 in the target to be tracked information areas 431 other than the top level includes the tracking status instead of the explanatory text. The tracking stop button 4315 is a button to be pressed when ending the tracking of the person.

[0144] Also, each time the tracking alert output function unit 3532 creates a new card, the tracking information card 432 is additionally displayed at the top of the tracking alert screen 43. The old tracking information cards 432 are sequentially shifted downward. The tracking information card 432 includes, as its display content, the detection date and time information 4321, the detection camera location information 4322, the detected face image 4323, the face attention stage display 4324, the detected full-body image 4325, the full-body attention stage display 4326, and the detected background image 4327.

[0145] Here, the detection date and time information 4321 is the date and time information detected from the surveillance images of any of the surveillance cameras 10 for the person to be tracked. The detected camera location information 4322 is the camera location information based on the detected surveillance camera ID. The detected face image 4323 is the face image for verification of the person to be tracked that has been detected and saved. The face attention level display 4324 indicates the level of similarity between the face image of the person to be tracked and the face image of the detected person. That is, the similarity is not a real number, but a plurality of levels based on the face image similarity label ID in the tracking table 3251. Here, it is shown as a three-level attention level display. In this face attention level display 4324, the number of hatched and identified squares in FIG. 21 corresponds to the face image similarity label ID. For example, in the topmost tracking information card 432, since the face image similarity label ID in the tracking table 3251 was "2", two squares are identified and displayed in the face attention level display 4324. In the second tracking information card 432 from the top, since the face image similarity label ID in the tracking table 3251 was "3", three squares are identified and displayed in the face attention level display 4324.

[0146] Similarly, the detected full-body image 4325 is the full-body image for verification of the person to be tracked that has been detected and saved. The full-body attention level display 4326 indicates the level of similarity between the full-body image of the person to be tracked and the full-body image of the detected person. That is, the similarity is not a real number, but a plurality of levels based on the full-body image similarity label ID in the tracking table 3251. Here, it is shown as a three-level attention level display. Similar to the face attention level display 4324, in this full-body attention level display 4326, the number of hatched and identified squares in FIG. 21 corresponds to the full-body image similarity label ID. For example, in the topmost tracking information card 432, since the full-body image similarity label ID in the tracking table 3251 was "3", three squares are identified and displayed in the full-body attention level display 4326. In the second tracking information card 432 from the top, since the full-body image similarity label ID in the tracking table 3251 was "2", two squares are identified and displayed in the full-body attention level display 4326.

[0147] The detection background image 4327 is a frame image that is the original surveillance image from which the face image for verification and / or the full-body image for verification has been extracted. This frame image uses the frame image 3111 stored in the past search data storage unit 311.

[0148] Note that the tracking information card 432 does not necessarily include both the detected face image 4323 and the face attention level display 4324, and the detected full-body image 4325 and the full-body attention level display 4326, and may include only one of them.

[0149] The narrowing-down instruction area 433 is an area for receiving an instruction to narrow down the tracking information card 432 displayed on the tracking alert screen 43. This narrowing-down instruction area 433 includes a face image similarity selection button 4331, a face image similarity selection reset button 4332, a full-body image similarity selection button 4333, and a full-body image similarity selection reset button 4334.

[0150] Here, the face image similarity selection button 4331 includes three buttons corresponding to the levels of the face attention level display 4324. Only the tracking information card 432 including the face attention level display 4324 at or above the specified level corresponding to the pressing operation of this face image similarity selection button 4331 by the monitor is displayed on the tracking alert screen 43. In the example of FIG. 21, since the level on the face image similarity selection button 4331 is not specified, the tracking information card 432 including the face attention level display 4324 at level 1 or above is displayed on the tracking alert screen 43. The face image similarity selection reset button 4332 is a button to be pressed when resetting the level specification on the face image similarity selection button 4331.

[0151] Similarly, the full-body image similarity selection button 4333 includes three buttons corresponding to the levels of the full-body attention stage display 4326. Only the tracking information card 432 that includes the full-body attention stage display 4326 at or above the specified level in response to the pressing operation of this full-body image similarity selection button 4333 by the monitor will be displayed on the tracking alert screen 43. In the example of FIG. 21, since the level on the full-body image similarity selection button 4333 is not specified, the tracking information card 432 that includes the full-body attention stage display 4326 at level 1 or above is displayed on the tracking alert screen 43. The full-body image similarity selection reset button 4334 is a button that is pressed when resetting the level specification on the full-body image similarity selection button 4333.

[0152] FIG. 22 is a diagram showing another example of the tracking alert screen 43 displayed on the web browser 41. This example shows the case where 3 levels are specified on the face image similarity selection button 4331 and 1 level is specified on the full-body image similarity selection button 4333. Here, the pressed button is identified and displayed as shown by hatching in FIG. 21. The specification by the face image similarity selection button 4331 and the specification by the full-body image similarity selection button 4333 are in an AND condition, and only the tracking information card 432 that matches the levels specified by both will be displayed on the tracking alert screen 43.

[0153] (2-3) Past search operation The tracking operation by the tracking function unit 353 is a function of tracking the tracking target person in real time. On the other hand, there are cases where it is desired to investigate whether a specific person has visited the facility in the past.

[0154] Therefore, the past search function unit 354 receives the specification of time and location from the web browser 41, and extracts the full-body image 3112 of the person who visited the specified time and location from the past search data storage unit 311 based on the past search data table 3221 stored in the past search data table storage unit 322. Then, a selection screen that displays the extracted full-body images 3112 in a list is created and displayed on the web browser 41.

[0155] Upon receiving the selection and designation of the person to be searched from the web browser 41, the past search function unit 354 uses the selected and designated full-body image 3112 as the full-body image for tracking. Also, instead of the surveillance image from the real-time surveillance camera 10, the past search function unit 354 performs the same operation as the tracking function unit 353 on the frame image 3111 stored in the past search data storage unit 311 for the designated time and location.

[0156] As a result, the past search function unit 354 can search for a person similar to the selected and designated person among the past visitors stored in the past search data storage unit 311, and display the result on the web browser 41 as a tracking alert screen 43.

[0157] (3) Effect As described in detail above, in the Web server 30 as a person detection device according to an embodiment, the monitoring / tracking execution function unit 23 of the video analysis function unit 20, which is a similarity calculation unit that calculates two or more similarities of a target image with respect to a specific image using each of two or more similarity calculation methods having different determination criteria, stores, for each similarity calculation method, a plurality of thresholds for classifying the target image into a plurality of stages in the camera information table storage unit 321 as a threshold storage unit. The detection / tracking result determination function unit 34 as a detection unit acquires at least one similarity of an image of a similarity calculation target person, which is a target image extracted from each of the monitoring images periodically acquired by each of the plurality of monitoring cameras 10, with respect to an image of a specific person, which is a specific image, from the monitoring / tracking execution function unit 23, and based on the plurality of thresholds stored in the camera information table storage unit 321 corresponding to the similarity calculation method used for calculating at least one similarity in the monitoring / tracking execution function unit 23, detects an image similar to the image of the specific person from among the images of the similarity calculation target person extracted from the monitoring images of each of the plurality of monitoring cameras 10, and also detects in which of the plurality of attention stages the detected image of the similarity calculation target person is. In this way, by classifying similarities that are difficult to compare with each other due to different determination criteria of the similarity calculation methods into a plurality of stages independent of the similarity calculation methods, it is possible to reduce the degree of similarity to the same evaluation criteria regardless of which similarity calculation method is used. More specifically, in the camera information table storage unit 321, for example, for each of at least one collation model, a plurality of thresholds for determining in which of the plurality of attention stages the similarity is located are stored, and the detection / tracking result determination function unit 34 acquires the similarity between the image of the similarity calculation target person extracted from each of the monitoring images periodically acquired by each of the plurality of monitoring cameras 10 and the image of the specific person from the monitoring / tracking execution function unit 23 that calculates the similarity using, for example, at least one collation model, and detects in which of the attention stages the acquired similarity is based on the plurality of thresholds stored in the camera information table storage unit 321. With such detected warning levels, it becomes easier for the monitor to recognize the reliability of the authentication result. Also, since the threshold value for each monitoring camera 10 is stored in the camera information table storage unit 321, the variation in the acquired similarity among the monitoring cameras 10 can be reduced.

[0158] Further, the Web server 30 may further include a detection alert output function unit 3521 as an output screen generation unit that generates a detection alert screen 42, which is an output screen for presenting an image similar to the image of the specific person detected by the detection / tracking result determination function unit 34 together with a stage display indicating the warning level detected by the detection / tracking result determination function unit 34. More specifically, based on the discrimination result of the detection / tracking result determination function unit 34, the detection alert output function unit 3521 generates a detection alert screen 42 for presenting, together with the warning level, the image of the similarity calculation target person similar to the image of the specific person among the images of the similarity calculation target persons extracted from the monitoring images of each of the plurality of monitoring cameras 10. In this way, since the similarity, which is the authentication result, is presented as the warning level display 4215, even a non-expert monitor can easily and somewhat accurately intuitively recognize the similarity of the images. Therefore, it becomes easier for the monitor to recognize the reliability of the authentication result.

[0159] Note that the detection / tracking result determination function unit 34 may obtain the similarity between the face image of the similarity calculation target person extracted from each of the monitoring images of the plurality of monitoring cameras 10 and the face image of the specific person from the monitoring / tracking execution function unit 23, and the detection alert output function unit 3521 may generate a detection alert screen 42 for presenting, together with the warning level display 4215, the face image of the similarity calculation target person similar to the face image of the specific person. This makes it easier for the monitor to recognize the reliability of the authentication result of the face image. Note that instead of the face image, the full-body image of the specific person and the face image of the similarity calculation target person may be used.

[0160] Here, the plurality of thresholds can include a first threshold for classifying whether the image of the person for whom similarity is to be calculated, which is the target image, is the image of a specific person, and at least one second threshold for classifying the image of the person for whom similarity is to be calculated into a plurality of levels when the image of the person for whom similarity is to be calculated is the image of a specific person. More specifically, the camera information table storage unit 321 stores a face matching threshold 1 corresponding to the similarity at which the image of the person for whom similarity is to be calculated is estimated to be the image of a specific person as the first threshold, and face matching thresholds 2 and 3 corresponding to similarities each greater than the face matching threshold 1 as the second thresholds. The detection / tracking result determination functional unit 34 detects an image of a person for whom similarity is to be calculated having a similarity greater than the face matching threshold 1, and the detection alert output functional unit 3521 can generate a detection alert screen 42 that presents the image of the person for whom similarity is to be calculated detected by the detection / tracking result determination functional unit 34 based on the face matching threshold 1, together with an attention level display 4215 indicating the attention level determined by the detection / tracking result determination functional unit 34 based on the face matching thresholds 2 and 3. Thereby, the image of the person for whom similarity is to be calculated to be presented on the detection alert screen 42 can be easily detected by the face matching threshold 1, and for each of the detected images, it becomes possible to determine what kind of attention level display 4215 should be set based on the face matching thresholds 2 and 3.

[0161] In addition, as a person tracking device according to an embodiment, the Web server 30 receives, from a monitoring terminal 40 operated by a monitor, a designation of an image of a person to be tracked from among images of a plurality of similarity calculation target persons similar to the image of a specific person presented on the detection alert screen 42 with step display in addition to the configuration of the Web server 30 as the person detection device. The designated image is registered as a tracking image, and the tracking image, which is the image of the person to be tracked for which tracking has been registered, is used as a specific image by the monitoring / tracking execution function unit 23. An image registration function unit 3531, which is a registration unit, causes the similarity of the images of the similarity calculation target persons to be extracted at least twice using at least two or more similarity calculation methods. A tracking alert output function unit 3532, which is a tracking output screen generation unit, generates a tracking alert screen 43, which is a tracking output screen, for presenting each of the images of two or more similarity calculation target persons similar to the image of the person to be tracked detected from at least two or more similarities acquired by the detection / tracking result determination function unit 34 from the monitoring / tracking execution function unit 23, together with a step display indicating the caution level detected by the detection / tracking result determination function unit 34. More specifically, the image registration function unit 3531 receives, from the monitoring terminal 40 operated by the monitor, a designation of the face image of the tracking target person to be tracked from among the images of a plurality of similarity calculation target persons similar to the image of the specific person presented on the detection alert screen 42 with step display, and registers the designated face image and the full body image of the tracking target person corresponding to the face image as the tracking face image and the tracking full body image. Further, the camera information table storage unit 321 stores, as the first threshold values, a face matching threshold value 1 corresponding to the similarity at which the face image of the similarity calculation target person is estimated to be the tracking face image registered by the image registration function unit 3531, and a full body matching threshold value 1 corresponding to the similarity at which the full body image of the similarity calculation target person is estimated to be the tracking full body image registered by the image registration function unit 3531, and stores, as at least one second threshold value, a face matching threshold value 2 and a face matching threshold value 3 corresponding to the similarity of the face image greater than the face matching threshold value 1, and a full body matching threshold value 2 and a full body matching threshold value 3 corresponding to the similarity of the full body image greater than the full body matching threshold value 1. Then, the detection / tracking result determination function unit 34 obtains, from the monitoring / tracking execution function unit 23, the similarity between the face image of the similarity calculation target person extracted from each of the monitoring images of the plurality of monitoring cameras 10 calculated using, for example, the first matching model and the tracking face image registered by the image registration function unit 3531, and the similarity between the full body image of the similarity calculation target person and the tracking full body image registered by the image registration function unit 3531 calculated using, for example, the second matching model, and detects at least one of the face image of the similarity calculation target person having a similarity greater than the face matching threshold value 1 among the obtained similarities of the face images and the full body image of the similarity calculation target person having a similarity greater than the full body matching threshold value 1 among the obtained similarities of the full body images. The tracking alert output function unit 3532 generates a tracking alert screen 43 that presents at least one of the face image and the full body image of the similarity calculation target person detected by the detection / tracking result determination function unit 34 based on at least one of the face matching threshold value 1 and the full body matching threshold value 1, together with a step display indicating the caution level determined based on at least one of the corresponding face matching threshold values 2 and 3 and the full body matching threshold values 2 and 3. As a result, even when two or more similarity calculation methods having different criteria from each other, for example, a first collation model for face images and a second collation model for full-body images, output similarity values with different ranges or scales from each other, on the tracking alert screen 43, the similarity is displayed by the same stage display, so that the monitor can easily recognize the identity of the similarity.

[0162] In addition, a Web server 30 as a person tracking device may further include a file server 31 as an image storage unit that stores a full-body image, a full-body feature amount, and a face feature amount extracted by a person detection information extraction function unit 22 of a video analysis function unit 20, which is an extraction unit that extracts a full-body image of a person from each of the surveillance images, extracts a full-body feature amount, which is a feature amount thereof, and a face image of the person from the full-body image, and extracts a face feature amount, which is a feature amount thereof, from the face image, together with the surveillance images. An image registration function unit 3531 registers the full-body image, the full-body feature amount, and the face feature amount stored in the file server 31 corresponding to the face image of the tracking target person designated from the surveillance terminal 40, and the face image from which the face feature amount was extracted, as a tracking full-body image, a tracking full-body feature amount, a tracking face feature amount, and a tracking face image. A detection / tracking result determination function unit 34 receives, from a surveillance / tracking execution function unit 23 that calculates a full-body image similarity, which is a similarity between the full-body feature amount extracted by the person detection information extraction function unit 22 and the tracking full-body feature amount, and a face image similarity, which is a similarity between the face feature amount extracted by the person detection information extraction function unit 22 and the tracking face feature amount, for each of the verification images, which are the surveillance images stored in the file server 31 from the time of registration by the image registration function unit 3531, a set of the calculated full-body image similarity and face image similarity, the tracking face image, and the tracking full-body image, determines whether at least one of the face image similarity and the full-body image similarity is greater than at least one of the corresponding face verification threshold 1 and the full-body verification threshold 1, detects at least one of the face image and the full-body image of the similarity calculation target person having a similarity greater than at least one of the face verification threshold 1 and the full-body verification threshold 1, and a detection alert output function unit 3521 generates a tracking alert screen 43, which is a tracking output screen, that presents at least one of the face image and the full-body image of the similarity calculation target person detected by the detection / tracking result determination function unit 34, a stage display indicating a caution stage determined based on at least one of the corresponding face verification threshold 2 and face verification threshold 3 and the full-body verification threshold 2 and full-body verification threshold 3, and the position information of the surveillance camera 10 that acquired the surveillance image as the verification image, together with the tracking full-body image and the tracking face image registered by the image registration function unit 3531. As a result, even when the face image of the person to be tracked cannot be acquired by the surveillance camera 10, the person to be tracked can be tracked using the full-body image.

[0163] Note that the detection alert output function unit 3521 can generate a tracking alert screen 43 as a narrowed-down output screen that presents, along with stage display, those having a similarity of at least one or more of the face matching threshold 2 and the face matching threshold 3 and the full-body matching threshold 2 and the full-body matching threshold 3 corresponding to the specified stage among at least one of the face image and the full-body image of the similarity calculation target person detected by the detection / tracking result determination function unit 34, in response to the stage designation from the surveillance terminal 40. When false detections increase during tracking, the number of displayed items increases, and results with low accuracy may also be displayed, making it difficult to identify the person to be tracked. The Web server 30 as a person tracking device according to an embodiment can encourage viewing only of highly accurate tracking results by enabling setting of the standard for the display stage of the stage display indicating the attention stage.

[0164] Alternatively, in the Web server 30 as a person tracking device according to an embodiment, in the camera information table storage unit 321 as a threshold storage unit, for each of two or more similarity calculation methods having different judgment criteria, the similarity of the target image to the specific image is calculated. Based on each of the similarities calculated by the monitoring / tracking execution function unit 23 of the video analysis function unit 20, which is a similarity calculation unit that calculates two or more similarities, a plurality of thresholds for classifying the target image into a plurality of stages are stored for each similarity calculation method. The image registration function unit 3531 as a registration unit registers an image of a person to be tracked, specified from the monitoring terminal 40 operated by the monitor, as a specific image. Then, the detection / tracking result determination function unit 34 as a detection unit acquires at least one similarity of the image of the similarity calculation target person, which is a target image extracted from each of the monitoring images periodically acquired by each of the plurality of monitoring cameras 10, to the specific image, from the monitoring / tracking execution function unit 23, and based on the plurality of thresholds stored in the camera information table storage unit 321 corresponding to the similarity calculation method used for calculating at least one similarity in the monitoring / tracking execution function unit 23, from among the images of the similarity calculation target person extracted from the monitoring images of each of the plurality of monitoring cameras 10, detects an image similar to the image of the person to be tracked, and also detects which of the plurality of attention stages the detected image of the similarity calculation target person is in. The tracking alert output function unit 3532 as a tracking output screen generation unit generates a tracking alert screen 43 as a tracking output screen for presenting each of two or more images of the similarity calculation target person similar to the image of the person to be tracked detected by the detection / tracking result determination function unit 34 from at least two or more similarities acquired from the monitoring / tracking execution function unit 23, together with a stage display indicating the attention stage detected by the monitoring / tracking execution function unit 23. More specifically, the camera information table storage unit 321 stores a plurality of thresholds for determining at which of a plurality of attention levels the similarity is for each of, for example, two or more collation models. The image registration function unit 3531 registers an image of a tracking target person to be tracked specified from the monitoring terminal 40 operated by the monitor. Then, the detection / tracking result determination function unit 34 obtains two or more similarities from the monitoring / tracking execution function unit 23 that calculates the similarity between the image of the tracking target person registered by the image registration function unit 3531 and the images of the similarity calculation target persons extracted from each of the monitoring images periodically acquired by each of the plurality of monitoring cameras 10 using each of the two or more collation models, and determines based on the plurality of thresholds stored in the camera information table storage unit 321 at which of the attention levels each of the two or more obtained similarities is. The tracking alert output function unit 3532 generates a tracking alert screen 43 for presenting, together with a stage display indicating each attention level, the image of the similarity calculation target person similar to the image of the tracking target person among the images of the similarity calculation target persons extracted from the monitoring images of each of the plurality of monitoring cameras 10 based on the determination result of the detection / tracking result determination function unit 34. Thereby, even when two or more similarity calculation methods having different judgment criteria, for example, two or more collation models, output similarity values with different ranges or scales from each other, the similarity is displayed by the same stage display on the tracking alert screen 43, so that the monitor can easily recognize the identity of the similarity.

[0165] Here, the tracking alert output function unit 3532 can generate the tracking alert screen 43 as a narrowed-down output screen that presents, together with a stage display, those having a similarity equal to or higher than the threshold corresponding to the specified stage among the images of the similarity calculation target persons detected by the detection / tracking result determination function unit 34 in response to the stage number specification from the monitoring terminal 40. In this way, by making it possible to set the reference of the display stage of the stage display indicating the attention level, it is possible to encourage viewing only of highly accurate tracking results.

[0166] In addition, the camera information table storage unit 321 stores a face matching threshold 1 or a full body matching threshold 1, which is a first threshold corresponding to the similarity at which the image of the person whose similarity is to be calculated is estimated to be the image of the person to be tracked, and at least one second threshold corresponding to a similarity greater than the face matching threshold 1 or the full body matching threshold 1, namely, a face matching threshold 2 and a face matching threshold 3, or a full body matching threshold 2 and a full body matching threshold 3. The detection / tracking result determination function unit 34 detects an image of the person whose similarity is to be calculated having a face similarity or a full body similarity greater than the face matching threshold 1 or the full body matching threshold 1. The tracking alert output function unit 3532 can generate a tracking alert screen 43 that presents the image of the person whose similarity is to be calculated detected by the detection / tracking result determination function unit 34 based on the face matching threshold 1 or the full body matching threshold 1, together with a stage display indicating the attention stage determined by the detection / tracking result determination function unit 34 based on the face matching thresholds 2 and 3 or the full body matching thresholds 2 and 3. Thereby, it becomes possible to easily detect an image of the person whose similarity is to be calculated to be presented on the tracking alert screen 43 by the face matching threshold 1 or the full body matching threshold 1, and for each of the detected images, it becomes possible to determine what kind of attention stage display 4215 should be set based on the face matching thresholds 2 and 3, or the full body matching thresholds 2 and 3.

[0167] Moreover, the person tracking system according to an embodiment can include a person tracking device according to an embodiment, a plurality of surveillance cameras 10, a surveillance terminal 40 operated by a surveillant, and a video analysis function unit 20 as an analysis unit including a person detection information extraction function unit 22.

[0168] Moreover, a person detection method according to an embodiment is a person detection method for detecting a specific person from surveillance images acquired by a plurality of surveillance cameras 10. A Web server 30, which is a computer, calculates two or more similarities of a target image with respect to a specific image using each of two or more similarity calculation methods having different judgment criteria. Based on each of the similarities calculated by a surveillance / tracking execution function unit 23 of a video analysis function unit 20, which is a similarity calculation unit, a plurality of thresholds for classifying the target image into a plurality of stages are stored in a camera information table storage unit 321 of a DB server 32 provided in a storage device 302, which is a memory, for each similarity calculation method. At least one similarity of an image of a similarity calculation target person, which is a target image extracted from each of the surveillance images periodically acquired by each of the plurality of surveillance cameras 10, with respect to an image of a specific person, which is a specific image, is acquired from the surveillance / tracking execution function unit 23. Based on the plurality of thresholds stored in the camera information table storage unit 321 corresponding to the similarity calculation method used for calculating at least one similarity in the surveillance / tracking execution function unit 23, an image similar to the image of the specific person is detected from the images of the similarity calculation target persons extracted from the surveillance images of each of the plurality of surveillance cameras 10, and it is also detected in which of a plurality of attention stages the detected image of the similarity calculation target person is. More specifically, for example, for each of at least one matching model, a plurality of thresholds for determining in which of a plurality of attention stages the similarity is located are stored in the camera information table storage unit 321. The similarity between an image of a similarity calculation target person extracted from each of the surveillance images periodically acquired by each of the plurality of surveillance cameras 10 and an image of a specific person is acquired from the surveillance / tracking execution function unit 23, which calculates the similarity using at least one matching model. Based on the plurality of thresholds stored in the camera information table storage unit 321, it is detected in which of the attention stages the acquired similarity is located. Due to such detected attention stages, the supervisor can easily recognize the reliability of the authentication result. Also, since the threshold for each surveillance camera 10 is stored in the camera information table storage unit 321, the variation in the acquired similarity among the surveillance cameras 10 can be reduced.

[0169] Also, a person tracking method according to an embodiment is a person tracking method for tracking a person to be tracked from surveillance images acquired by a plurality of surveillance cameras 10. A Web server 30, which is a computer, generates a detection alert screen 42, which is an output screen for presenting an image similar to the image of a specific person detected by the person detection method according to the above embodiment, together with a stage display indicating the detected attention level. From among the images of a plurality of similarity calculation target persons similar to the image of the specific person presented on the detection alert screen 42 together with the stage display, the Web server 30 accepts, from a monitoring terminal 40 operated by a monitor, the designation of the image of the person to be tracked, and registers the designated image in a management data storage unit 313 of a file server 31 provided in a storage device 302, which is a memory, as a tracking image. Then, using the tracking image, which is the image of the person to be tracked registered in the management data storage unit 313, as a specific image, the monitoring / tracking execution functional unit 23 extracts at least two similarities of the images of the similarity calculation target persons using at least two or more similarity calculation methods. The monitoring / tracking execution functional unit 23 acquires at least two similarities of the images of the similarity calculation target persons extracted from the surveillance images of each of the plurality of surveillance cameras 10 with respect to the image of the person to be tracked, which is the specific image. Based on a plurality of threshold values stored in the management data storage unit 313 corresponding to the similarity calculation methods used for calculating each of the similarities in the monitoring / tracking execution functional unit 23, the monitoring / tracking execution functional unit 23 detects two or more images of similarity calculation target persons similar to the image of the person to be tracked from among the images of the similarity calculation target persons extracted from the surveillance images of each of the plurality of surveillance cameras 10, and detects which of the plurality of attention levels the detected images of the similarity calculation target persons are in. The monitoring / tracking execution functional unit 23 generates a tracking alert screen 43 as a tracking output screen for presenting each of the two or more images of similarity calculation target persons similar to the image of the person to be tracked detected from the at least two or more similarities acquired from the monitoring / tracking execution functional unit 23, together with a stage display indicating the detected attention level. More specifically, from among the images of a plurality of similarity calculation target persons extracted from each of the monitoring images of a plurality of monitoring cameras 10 presented on the detection alert screen 42 together with the attention level, the monitoring terminal 40 operated by the monitor receives the designation of the face image of the tracking target person to be tracked, and registers the designated face image and the full-body image of the tracking target person corresponding to the face image in the management data storage unit 313 as the tracking face image and the tracking full-body image. Then, from the monitoring / tracking execution function unit 23, for example, the similarity between the face image of the similarity calculation target person extracted from each of the monitoring images of the plurality of monitoring cameras 10 calculated using the first collation model and the registered tracking face image, and, for example, the similarity between the full-body image of the similarity calculation target person calculated using the second collation model and the registered tracking full-body image, are obtained. Also, in the camera information table storage unit 321, as the first threshold values, a face collation threshold value 1 corresponding to the similarity at which the face image of the similarity calculation target person is presumed to be the registered tracking face image and a full-body collation threshold value 1 corresponding to the similarity at which the full-body image of the similarity calculation target person is presumed to be the registered tracking full-body image, and, as at least one second threshold value, a face collation threshold value 2 and a face collation threshold value 3 corresponding to the similarity of the face image greater than the face collation threshold value 1 and a full-body collation threshold value 2 and a full-body collation threshold value 3 corresponding to the similarity of the full-body image greater than the full-body collation threshold value 1 are stored. Then, from among the obtained similarities, at least one of the face image of the similarity calculation target person having a similarity greater than the face collation threshold value 1 and the full-body image of the similarity calculation target person having a similarity greater than the full-body collation threshold value 1 is detected, and at least one of the face image and the full-body image of the similarity calculation target person detected based on at least one of the face collation threshold value 1 and the full-body collation threshold value 1 is presented together with a stage display indicating the attention level determined based on at least one of the corresponding face collation threshold values 2 and 3 and the full-body collation threshold values 2 and 3 as a tracking alert screen 43 as a tracking output screen. As a result, even when two or more similarity calculation methods having different judgment criteria, for example, a first collation model for face images and a second collation model for full-body images, output similarity values with different ranges or scales from each other, on the tracking alert screen 43, the similarity is displayed by the same stage display, so that the monitor can easily recognize the identity of the similarity.

[0170] Alternatively, a person tracking method according to an embodiment is a person tracking method for tracking a tracking target person to be tracked from surveillance images acquired by a plurality of surveillance cameras 10. A Web server 30, which is a computer, calculates two or more similarities of a target image with respect to a specific image using each of two or more similarity calculation methods having different judgment criteria. Based on each of the similarities calculated by the monitoring / tracking execution function unit 23 of the video analysis function unit 20, a plurality of thresholds for classifying the target image into a plurality of stages are stored in the camera information table storage unit 321 of the DB server 32 provided in the storage device 302, which is a memory, for each similarity calculation method. An image of a tracking target person to be tracked designated from a monitoring terminal 40 operated by a monitor is registered in the management data storage unit 313 of the file server 31 provided in the storage device 302, which is a memory, as a specific image. At least one similarity of an image of a similarity calculation target person, which is a target image extracted from each of the surveillance images periodically acquired by each of the plurality of surveillance cameras 10, with respect to the specific image is obtained from the monitoring / tracking execution function unit 23. Based on a plurality of thresholds stored in the camera information table storage unit 321 corresponding to the similarity calculation method used for calculating at least one similarity in the monitoring / tracking execution function unit 23, an image similar to the image of the tracking target person is detected from the images of the similarity calculation target persons extracted from the surveillance images of each of the plurality of surveillance cameras 10, and it is detected which of the plurality of attention stages the detected image of the similarity calculation target person is in. A tracking alert screen 43 as a tracking output screen for presenting each of two or more images of similarity calculation target persons similar to the image of the tracking target person detected from at least two or more similarities obtained from the monitoring / tracking execution function unit 23, together with a stage display indicating the detected attention stage, is generated. More specifically, for example, for each of two or more collation models, a plurality of thresholds for determining at which of a plurality of attention levels the similarity is located are stored in the camera information table storage unit 321. An image of a tracking target to be tracked designated from the monitoring terminal 40 operated by the monitor is registered in the management data storage unit 313, and the similarity between the registered image of the tracking target person and the images of the persons whose similarity is to be calculated extracted from each of the monitoring images periodically acquired by each of the plurality of monitoring cameras 10 is calculated using each of the two or more collation models. The monitoring / tracking execution function unit 23 acquires two or more similarities, determines at which attention level the acquired similarity is located based on the plurality of stored thresholds, and based on the result of the determination, presents, together with a stage display indicating each attention level, an image of a person whose similarity is to be calculated and is similar to the image of the tracking target person among the images of the persons whose similarity is to be calculated extracted from the monitoring images of each of the plurality of monitoring cameras 10. A tracking alert screen 43 is generated. Accordingly, even when two or more similarity calculation methods having different judgment criteria from each other, for example, two or more collation models, output similarity values having different ranges or scales from each other, on the tracking alert screen 43, the similarities are displayed by the same stage display, so that the monitor can easily recognize the identity of the similarities.

[0171] [Other Embodiments] In the above-described embodiment, it has been described that a full-body image of a person is extracted from the monitoring image acquired by the monitoring camera 10 and a face image of the person is extracted from the extracted full-body image, but the present invention is not limited thereto. When a person's face is captured in the monitoring image, the face image may be extracted without extracting the full-body image.

[0172] In addition, in one embodiment, the whole-body image feature amount and the face image feature amount are the quantification of the attribute information of the person himself / herself, but it is not limited thereto. The whole-body feature amount may include, for example, the quantification of the accessory information associated with the person, such as clothing, belongings (types and colors of bags, baby strollers, etc.), ornaments (sunglasses, masks, etc.). Further, the whole-body feature amount may include the quantification of the internal information of the person obtained from the image such as fever information, pulse, etc. or obtained from other sensors. Also, the face feature amount may include the quantification of the accessory information associated with the face, such as ornaments (glasses, hats, etc.).

[0173] In addition, the whole-body image feature amount and the face image feature amount are only the quantification of the attribute information of the person himself / herself, and the quantification of those accessory information and the quantification of the internal information of the person may be used as another feature amount and used as one of the detection criteria similar to the whole-body image feature amount and the face image feature amount.

[0174] Also, in the above-described embodiment, the past search data storage unit 311 of the file server 31 stores the frame image 3111, the whole-body image 3112, the whole-body feature amount 3113, and the face feature amount 3114 as the past search data, but the face image from which the face feature amount 3114 is extracted may also be stored.

[0175] Also, the order of the processing steps shown in the flowcharts of FIGS. 13 to 15 and FIGS. 17 to 20C is an example and is not limited to this order. For example, in FIG. 18, the processing of steps S112 and S113 and the processing of steps S114 and S115 may be in the reverse order or may be performed in parallel. Also, for the processing of steps S504 to S509 in FIG. 20A and the processing of steps S510 to S515 in FIG. 20B, the order may be reversed or may be performed in parallel. Thus, each processing step may change the processing order and the like as long as there is no conflict with the preceding or subsequent processing steps.

[0176] Also, the detection alert screen 42 shown in FIG. 16 and the tracking alert screen shown in FIG. 21 (and FIG. 22) are not limited to this layout and display content. For example, the attention level display 4215, the face attention level display 4324, and the whole body attention level display 4326 are represented by the number of discriminatively displayed multiple squares. However, for example, the attention level may be represented by discriminative display that changes the color of one square, such as yellow, orange, or red. Alternatively, instead of the number or color of the squares, the attention level may be represented by letters such as A to C or 1 to 3.

[0177] In short, the present invention is not limited to the above-described embodiments as they are, and at the implementation stage, the components can be modified and embodied without departing from the gist thereof. Also, various inventions can be formed by appropriately combining a plurality of components disclosed in the above-described embodiments. For example, some components may be deleted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

Description of Reference Numerals

[0178] 1…Person tracking system 10…Surveillance camera 20…Video analysis functional unit 21…Image acquisition module 22…Human detection information extraction functional unit 23…Surveillance / tracking execution functional unit 30…Web server 31…File server 32…Database (DB) server 33…Search human detection result storage functional unit 34…Detection / tracking result determination functional unit 35…Web application 40…Surveillance terminal 41…Web browser 42…Detection alert screen 43…Tracking alert screen 221…Whole body detection module 222…Region tracking module 223…Whole body feature amount extraction module 224…Face detection module 225…Face feature extraction module 231…Full body matching module 232…Face matching module 251…Tracking table 301A…Hardware processor 301B…Program memory 302…Memory device 303…Communication interface device 304…Bus 311…Data storage unit for past searches 312…Detection history data storage unit 313…Management data storage unit 321…Camera information table storage unit 322…Data table storage unit for past searches 323…Detection history table storage unit 324…Management table storage unit 325…Tracking table storage unit 351…Login function unit 352…Monitoring function unit 353…Tracking function unit 354…Past search function unit 421…Detected person card 431…Tracking target person information area 432…Tracking information card 433…Narrowing instruction area 3111…Frame image 3112…Full body image 3113,3134…Full body feature quantities 3114,3122,3132…Face feature quantities 3121,4212…Detected face image 3131…Registered face image 3133…Registered full body image 3211…Camera information table 3221…Data table for past searches 3231…Detection history table 3241…Management table 3251…Tracking table 3521…Detected alert output function unit 3531… Image registration function unit 3532… Tracking alert output function unit 4211… Monitored person's face image 4213… Attention-arousing message 4214, 4312… Suspicious person ID 4215… Attention level display 4216… Detection information 4217… Tracking start button 4311… Attention-arousing message 4313… Registered image 4314… Tracking information 4315… Tracking stop button 4321… Detection date and time information 4322… Detection camera location information 4323… Detected face image 4324… Face attention level display 4325… Detected full-body image 4326… Full-body attention level display 4327… Detected background image 4331… Face image similarity selection button 4332… Face image similarity selection reset button 4333… Full-body image similarity selection button 4334… Full-body image similarity selection reset button NET… Network SV… Server device

Claims

1. a threshold value storage unit that stores a plurality of face matching threshold values ​​for classifying a target image into a plurality of stages for one matching model based on the facial image similarity calculated by a similarity calculation unit that calculates a facial image similarity, which is a similarity of a facial image of a similarity calculation target person, which is a target image, to a facial image of a specific person, which is a specific image, using one similarity calculation method; a detection unit which acquires from the similarity calculation unit a facial image similarity of a facial image of a similarity calculation subject, which is the target image, extracted from each of the surveillance images periodically acquired by each of a plurality of surveillance cameras, to a facial image of a specific person, which is the specific image, and detects a facial image similar to the facial image of the specific person from among the facial images of the similarity calculation subjects extracted from the surveillance images of each of the plurality of surveillance cameras based on the plurality of face matching thresholds stored in the threshold storage unit, and detects which of a plurality of attention stages the detected facial image of the similarity calculation subject is in; an output screen generating unit that generates an output screen for presenting the face image similar to the face image of the specific person detected by the detection unit together with a stage display that indicates the attention stage detected by the detection unit; A human detection device comprising:

2. A human detection device as described in Claim 1, wherein a facial image of the person to be similarity calculated is extracted from a whole-body image of the person extracted from each of the surveillance images.

3. 3. The human detection device according to claim 1, wherein the plurality of face matching thresholds include a first face matching threshold for classifying whether or not a face image of the similarity calculation subject, which is the target image, is a face image of the specific person, and at least one second face matching threshold for classifying the face image of the similarity calculation subject into the plurality of stages when the face image of the similarity calculation subject is the face image of the specific person.

4. The human detection device according to claim 1 ; a registration unit that receives, from a monitoring terminal operated by a monitor, a designation of a face image of a tracking target person to be tracked from among face images of a plurality of the similarity calculation targets similar to the face image of the specific person presented on the output screen by the human detection device together with the graded display, registers the designated face image as a tracking face image, and causes the similarity calculation unit to extract the face image similarity of the face image of the similarity calculation target person using the similarity calculation method, with the tracking face image being the registered face image of the tracking target person to be tracked as the specific image; a tracking output screen generation unit that generates a tracking output screen for presenting a face image of a similarity calculation target person similar to the face image of the tracking target person detected by the detection unit from the face image similarity acquired from the similarity calculation unit, together with a stage display that indicates the attention stage detected by the detection unit; A person tracking device comprising:

5. a threshold value storage unit that stores a plurality of face matching threshold values ​​for classifying the target image into a plurality of stages based on the facial image similarity calculated by a similarity calculation unit that calculates a facial image similarity, which is a similarity of a facial image of a similarity calculation target person, which is a target image, to a facial image of a specific person, which is a specific image, using one similarity calculation method; and a registration unit that registers a face image of a tracking target person to be tracked, which is designated by a monitoring terminal operated by a monitoring person, as the specific image; a detection unit which acquires from the similarity calculation unit a facial image similarity to the specific image of a facial image of a similarity calculation subject, which is the target image extracted from each of the surveillance images periodically acquired by each of a plurality of surveillance cameras, and detects a facial image similar to the facial image of the tracking subject from among the facial images of the similarity calculation subject extracted from the surveillance images of each of the plurality of surveillance cameras based on the plurality of face matching thresholds stored in the threshold storage unit, and detects which of a plurality of attention stages the detected facial image of the similarity calculation subject is in; a tracking output screen generation unit that generates a tracking output screen for presenting a face image of a similarity calculation target person similar to the face image of the tracking target person detected by the detection unit from the face image similarity acquired from the similarity calculation unit, together with a stage display indicating the attention stage; and A person tracking device comprising:

6. 6. The person tracking device according to claim 4 or 5, wherein the tracking output screen generation unit receives a number of stages specified by the monitoring terminal and generates a narrowed-down output screen that presents, together with a display of the stages, face images of the person to be similarity calculated that have a similarity equal to or greater than a face matching threshold corresponding to the specified stage, among the face images of the person detected by the detection unit.

7. A person tracking device according to any one of claims 4 to 6, The plurality of surveillance cameras; The monitoring terminal operated by the monitor; an analysis unit including the similarity calculation unit; A person tracking system comprising:

8. A person detection method for detecting a specific person from surveillance images captured by a plurality of surveillance cameras, comprising the steps of: The computer a similarity calculation unit calculates a facial image similarity, which is a similarity between a facial image of a specific person, which is a target image, and a facial image of a specific person, which is a specific image, by using one similarity calculation method, and stores a plurality of facial matching thresholds in a memory for classifying the target image into a plurality of stages based on the facial image similarity calculated by the similarity calculation unit; obtaining, from the similarity calculation unit, a similarity between a facial image of a person to be subjected to similarity calculation, which is the target image, extracted from each of the surveillance images periodically obtained by each of a plurality of surveillance cameras, and a facial image of a specific person, which is the specific image; detects a face image similar to a face image of the specific person from among face images of the similarity calculation subject extracted from the surveillance images of the plurality of surveillance cameras based on the plurality of face matching thresholds stored in the memory, and detects which of a plurality of attention stages the detected face image of the similarity calculation subject belongs to; generating an output screen for presenting the face image similar to the detected face image of the specific person together with a stage display indicating the detected attention stage; Person detection methods.

9. A person tracking method for tracking a target person to be tracked from surveillance images acquired by a plurality of surveillance cameras, comprising: The computer receiving, from a monitoring terminal operated by a monitor, designation of a face image of a tracking target person to be tracked from among face images of a plurality of similarity calculation targets similar to the face image of the specific person presented together with the graded display on the output screen generated by the person detection method according to claim 8; The designated face image is registered in the memory as a face image for tracking; a facial image for tracking, which is a facial image of the tracking target person to be tracked and registered in the memory, being set as the specific image, and the similarity calculation unit is caused to extract the facial image similarity of the facial image of the similarity calculation target person using the similarity calculation method, and the facial image similarity of the similarity calculation target person extracted from the surveillance images of each of the multiple surveillance cameras with respect to the facial image of the tracking target person, which is the specific image, is obtained from the similarity calculation unit; detects a face image of the similarity calculation subject similar to the face image of the tracking subject from among the face images of the similarity calculation subject extracted from the surveillance images of each of the plurality of surveillance cameras based on the plurality of face matching thresholds stored in the memory, and detects which of the plurality of attention stages the detected face image of the similarity calculation subject belongs to, and generates a tracking output screen on which is presented the face image of the similarity calculation subject similar to the face image of the tracking subject detected from the face image similarity acquired from the similarity calculation unit, together with a stage display indicating the detected attention stage. Person tracking methods.

10. A person tracking method for tracking a target person to be tracked from surveillance images acquired by a plurality of surveillance cameras, comprising: The computer a similarity calculation unit calculates a facial image similarity, which is a similarity between a facial image of a specific person, which is a target image, and a facial image of a specific person, which is a specific image, by using one similarity calculation method, and stores a plurality of facial matching thresholds in a memory for classifying the target image into a plurality of stages based on the facial image similarity calculated by the similarity calculation unit; A face image of a person to be tracked, which is designated by a monitoring terminal operated by a monitor, is registered in the memory as the specific image; obtaining, from the similarity calculation unit, a similarity between the specific image and a face image of a person to be subjected to similarity calculation, the face image being the target image extracted from each of the surveillance images periodically obtained by each of a plurality of surveillance cameras; detects a face image similar to the face image of the tracking target from among the face images of the similarity calculation target extracted from the surveillance images of each of the plurality of surveillance cameras based on the plurality of face matching thresholds stored in the memory, and detects which of a plurality of attention stages the detected face image of the similarity calculation target belongs to, and generates a tracking output screen for presenting the face image of the similarity calculation target similar to the face image of the tracking target detected from the similarity acquired from the similarity calculation unit together with a stage display indicating the detected attention stage. Person tracking methods.

11. A human detection program that causes a computer to operate as each unit of the human detection device according to any one of claims 1 to 3.

12. A person tracking program for causing a computer to operate as each unit of the person tracking device according to any one of claims 4 to 6.

Images