Management device

Abstract

The objective of this invention is to provide a management device that can contribute to the identification of unspecified individuals. [Solution] The management device comprises a first determination unit, a second determination unit, an extraction unit, and a display unit. The first determination unit determines whether or not an unspecified person is approaching at least one subject in the video footage captured by a surveillance camera installed in the facility. The second determination unit determines whether or not the face image of the unspecified person can be recognized from the video footage if the first determination unit has detected the approach of the unspecified person to the subject. If the second determination unit determines that the face image of the unspecified person cannot be recognized, the extraction unit compares the video footage with video data previously taken in the facility, identifies the same person as the unspecified person from among at least one person appearing in the video footage, and extracts an image of the same person from the video footage. The display unit displays the image extracted by the extraction unit.

Description

Technical Field

[0001] The present invention relates to a management device.

Background Art

[0002] Normally, since an unspecified number of people visit facilities such as medical facilities and commercial facilities, security in the facility is an important issue for the facility manager. Patent Document 1 discloses a lost child detection technique as a type of security technology in a facility. This lost child detection technique is as follows.

[0003] When a caregiver and a child etc. enter the facility, the face images of the caregiver and the child are individually captured by a registration camera. At that time, when the caregiver inputs the email address of his / her own mobile phone, that email address is registered in association with the face images of the caregiver and the child. A plurality of surveillance cameras are installed in the facility, and the child is monitored by the surveillance cameras. The video of the surveillance camera is transferred to a face matching server, and the face matching server recognizes the caregiver and the child in the video by matching the video with the face images of the caregiver and the child. The face matching server calculates the positions of the caregiver and the child based on the video, and then measures the time when the two are separated from each other. When the state where the caregiver and the child are separated from each other continues for a certain period of time, the face matching server detects this. Then, the face matching server sends an email to the email address of the caregiver. Thereby, the caregiver recognizes that the child is lost.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The technology described in Patent Document 1 is effective in the early detection of lost children and, consequently, in the early discovery of lost children. However, the technology in Patent Document 1 cannot prevent malicious individuals with the intention of kidnapping or other crimes from approaching children. In particular, malicious individuals almost always conceal their faces to avoid being discovered while committing a crime, and the technology in Patent Document 1 cannot identify such individuals.

[0006] Therefore, the objective of one or more embodiments of the present invention is to provide a management device that can contribute to the identification of unspecified individuals, such as malicious actors. [Means for solving the problem]

[0007] To solve the above problems, according to one aspect of the present invention, the management device comprises a first determination unit, a second determination unit, an extraction unit, and a display unit. The first determination unit determines whether or not an unspecified person is approaching at least one subject in the video footage captured by a surveillance camera installed in the facility. The second determination unit determines whether or not the face image of the unspecified person can be recognized from the video footage if the first determination unit has detected the approach of the unspecified person to the subject. If the second determination unit determines that the face image of the unspecified person cannot be recognized, the extraction unit compares the video footage with video data previously taken in the facility, identifies the same person as the unspecified person from among the at least one person appearing in the video footage, and extracts an image of the same person from the video footage. The display unit displays the image extracted by the extraction unit. [Effects of the Invention]

[0008] According to one aspect of the present invention, even if the face of an unspecified person is not visible in the video footage captured by the surveillance camera, if the same person appears in past videos within the facility, that person's image will be extracted from the video and displayed. Therefore, this management device contributes to the identification of the unspecified person. [Brief explanation of the drawing]

[0009] [Figure 1]Figure 1 is a block diagram in which the components of the monitoring system are represented as blocks. [Modes for carrying out the invention]

[0010] Embodiments will be described below with reference to the drawings. The features and technical effects of the embodiments will be understood from the following detailed description and drawings. However, the scope of the present invention is not limited to the embodiments disclosed below. The scope of the present invention is not limited to the examples in the drawings, as the drawings are provided for illustrative purposes only.

[0011] In the following description, "facility" refers to a facility where a large number of people gather, such as a medical facility, amusement facility, commercial facility, exhibition facility, and event facility. For example, a facility could be a hospital, a shopping complex, amusement park, zoo, or sports viewing facility. A facility may be indoors, outdoors, or both indoors and outdoors. The target individuals are those whose safety is confirmed using the monitoring system 1 described below. Target individuals include children, such as infants, elementary school students, and junior high school students. Elderly individuals may also be included. Individuals with insufficient decision-making capacity may also be included. A chaperone is someone who leads a group of people, and is primarily a guardian responsible for their protection and supervision. Usually, the group's parent is the chaperone. However, someone who assists, supports, or helps the group may also be the chaperone. A target group refers to a group consisting of a target person and a chaperone who escorts that target person. Each target group may have one or more target people. For example, siblings may belong to the same target group as target people. Each target group may have one or more chaperones. For example, both parents may belong to the same target group as chaperones.

[0012] <1. Overview of the Monitoring System> Figure 1 is a block diagram of monitoring system 1. The monitoring system 1 monitors individuals who visit the facility and contributes to identifying unspecified individuals who approach them. The monitoring system 1 comprises a registration device 10, a management device 20, one or more surveillance cameras 21, a first input device 22, a first display device 23, a terminal device 24, an exit detection unit 25, and a storage device 30.

[0013] <2. Registered Devices> The registration device 10 is installed at the entrance of the facility. The registration device 10 includes a registration terminal device 11, at least one registration camera 12, at least one second input device 13, and at least one second display device 14. The registration terminal device 11, registration camera 12, second input device 13, and second display device 14 may be integrated into a single unit.

[0014] The registered camera 12 includes an image sensor, a lens, and an image processing circuit. The registered camera 12 captures images of the area in front of it and outputs the captured images to the registered terminal device 11.

[0015] The second input device 13 is installed near the registration camera 12 at the entrance of the facility. The second input device 13 is an input device such as a keyboard, mouse, touch panel, touchpad, stylus, pointing device, key, or push button. When the second input device 13 is operated, it outputs a signal corresponding to that operation to the registration terminal device 11.

[0016] The second display device 14 is, for example, a liquid crystal display device or an organic EL display device. The second display device 14 displays video according to the video signal transferred from the registration terminal device 11.

[0017] The registration terminal device 11 is a computer having a CPU (Central Processing Unit), GPU (Graphics Processing Unit), RAM (Random Access Memory), a storage medium, a bus, a bus controller, and an interface circuit, etc. The registration camera 12 is installed at the entrance of the facility.

[0018] The storage medium of the registration terminal device 11 has a semiconductor storage device or a magnetic storage device. This storage medium stores a program executable by the CPU of the registration terminal device 11. When this CPU executes the program, the registration terminal device 11 exhibits various functions and performs processing based on these functions.

[0019] The registration terminal device 11 generates a video signal of a screen for realizing a GUI (Graphical User Interface) and outputs the video signal to the second display device 14. Then, the second display device 14 displays a screen according to the video signal.

[0020] The registration terminal device 11 transfers the video input from the registration camera 12 to the second display device 14. Then, the second display device 14 displays the video input from the registration terminal device 11.

[0021] The registration terminal device 11 is connected to a local area network. The registration terminal device 11 communicates with the management device 20 via the local area network. The registration terminal device 11 may communicate with the management device 20 via the local area network and the Internet.

[0022] <3. The First Input Device> The first input device 22 is connected to the management device 20. The first input device 22 is an input device such as a keyboard, a mouse, a touch panel, a touch pad, a stylus, a pointing device, a key, and a push button. When the first input device 22 is operated, the first input device 22 outputs a signal corresponding to the operation to the management device 20.

[0023] <4. Second display device> The first display device 23 is, for example, a liquid crystal display device or an organic EL display device. The first display device 23 displays video according to the video signal transferred from the management device 20.

[0024] <5. Surveillance Camera 21> The surveillance camera 21 includes an image sensor, a lens, and an image processing circuit. The surveillance camera 21 is installed within the facility. If there are multiple surveillance cameras 21, they are installed in various locations within the facility and are distributed throughout the facility. The surveillance camera 21 is connected to the management device 20. The surveillance camera 21 may be connected to the management device 20 via a serial bus such as USB (registered trademark). The surveillance camera 21 may be connected to the management device 20 via a local area network. The surveillance camera 21 may be connected to the management device 20 via a local area network and the internet.

[0025] The surveillance camera 21 captures its surroundings and immediately outputs the captured video to the management device 20. The management device 20 immediately records video data 31 based on the video input from the surveillance camera 21 to the storage device 30. The video input from the surveillance camera 21 to the management device 20 is current, while the video data 31 is past. Alternatively, the surveillance camera 21 may immediately output the video to the storage device 30, thereby immediately recording the video data 31 based on that video to the storage device 30. Video encoding may be performed in the surveillance camera 21, the management device 20, or the storage device 30.

[0026] To mitigate the effects of failures such as power outages and network malfunctions, the surveillance camera 21 may have storage such as a semiconductor memory device or a magnetic memory device. In this case, before and during a failure, the surveillance camera 21 photographs its surroundings and stores the captured images as video data in the storage. After the failure is resolved, the surveillance camera 21 reads the video data from its storage and transfers it to the management device 20 or storage device 30, either by the administrator operating the surveillance camera 21 or by the surveillance camera 21 detecting the resolution of the failure. The management device 20 then transfers the video data received from the surveillance camera 21 to the storage device 30. The storage device 30 records the video data received from the surveillance camera 21 or the management device 20 as video data 31. As a result, the video data stored in the surveillance camera 21's storage is copied to the storage device 30 as video data 31. Alternatively, the storage of the surveillance camera 21 may be detachable from the surveillance camera 21, and the administrator may manually copy the video data 31 from the surveillance camera 21's storage to the storage device 30.

[0027] Furthermore, the combination of the registered camera 12 and the registered terminal device 11 may have the same functionality as the surveillance camera 21. In other words, the registered camera 12 and the registered terminal device 11 may operate continuously, and the registered terminal device 11 may immediately transfer the video captured by the registered camera 12 to the management device 20. In this case, the management device 20 immediately records video data 31 based on the video input from the registered terminal device 11 to the storage device 30.

[0028] <6. Storage device 30> The storage device 30 is a semiconductor memory device, a magnetic memory device, a NAS (Network Attached Storage), a data server, a file server, or a cloud computing system. The storage device 30 may be connected to the management device 20 via a serial bus. The storage device 30 may be connected to the management device 20 via a local area network. The storage device 30 may be connected to the management device 20 via a local area network and the Internet. The storage device 30 stores data transferred from the management device 20, reads stored data, and transfers that data to the management device 20.

[0029] The storage device 30 stores one or more video data 31. The video data 31 is data encoded from video footage captured by the surveillance camera 21. The video data 31 is associated with the surveillance camera 21 that captured the video footage of that video data 31.

[0030] The storage device 30 stores one or more registered data 32. The registered data 32 is registered by the registration terminal device 11. Registration data 32 is data created for each target group. Registration data 32 contains an identification number, a facial image of the subject, and a facial image of the group leader. The identification number is a unique value assigned to the target group. The identification number is a value unique to the target group. For each subject belonging to the target group, the subject's facial image is included in registration data 32. The faces of the subjects belonging to the target group are visible in the subject's facial image. For each group leader belonging to the target group, the group leader's facial image is included in registration data 32.

[0031] The storage device 30 pre-stores one or more facial image data 33. The facial image data 33 are still images or videos showing the faces of the facility's staff. Facial image data 33 is prepared for each staff member. A staff identification number is associated with the facial image data 33. This allows the facial image data 33 to be assigned to a staff member.

[0032] The storage device 30 pre-stores one or more schedule data sets 34. The schedule data is time-series data of the facility staff's schedules. Staff schedules consist of time-series data such as arrival time, departure time, break time, on-duty time, off-duty time, time spent performing specific tasks, and time spent in specific locations within the facility. Schedule data 34 is prepared for each staff member. A staff identification number is associated with the schedule data 34, thereby assigning the schedule data 34 to a staff member.

[0033] <7. Management device> The management device 20 has at least one computer, which includes a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), RAM (Random Access Memory), a storage medium, a bus, a bus controller, and interface circuits. The management device 20 may consist of a single computer. The management device 20 may consist of multiple computers capable of distributed or parallel processing. The management device 20 may be a cloud computing system. The management device 20 may be installed in a control room within the facility. The management device 20 may be installed in a data center outside the facility.

[0034] The storage medium of the management device 20 is a semiconductor memory device or a magnetic memory device. This storage medium stores a program that can be executed by the CPU of the management device 20. When this CPU executes the program, the management device 20 performs various functions and processes based on these functions.

[0035] The management device 20 is connected to a local area network. The management device 20 communicates with the registered terminal device 11 via the local area network. The management device 20 may also communicate with the registered terminal device 11 via the local area network and the internet.

[0036] The management device 20 records data to the storage device 30. The management device 20 reads data from the storage device 30.

[0037] The management device 20 generates video signals for various screens to implement a GUI (Graphical User Interface) and outputs these video signals to the first display device 23. The first display device 23 then displays a screen according to the video signals.

[0038] The management device 20 decodes the video input from the registered camera 12 as needed and transfers the video to the first display device 23. The first display device 23 then displays the video input from the management device 20.

[0039] The management device 20 encodes the video input from the registered camera 12 as needed and records the video data 31 based on that video in the storage device 30.

[0040] <8. Terminal Devices> The terminal device 24 is composed of a general-purpose computer system. A general-purpose computer system refers to a computer system such as a mobile phone, smartphone, tablet computer, laptop computer, or desktop computer on which a general-purpose OS (Operating System) is installed. Examples of general-purpose OSs include Windows®, Android®, iOS®, macOS®, Linux®, or Unix®. The terminal device 24 has a display device and an input device. The terminal device 24 is connected to the management device 20 via a local area network, the internet, or both. The terminal device 24 stores an application program. The application program allows the terminal device 24 to access the management device 20 and also allows the terminal device 24 to function as a console for the management device 20. The terminal device 24 functions as a console for the management device 20. The terminal device 24 may be installed in a control room or similar location within the facility. The terminal device 24 may be portable or stationary.

[0041] <9. Exit Detection Unit> The exit detection unit 25 is installed at the exit of the facility. The exit detection unit 25 detects the departure of the subject and their chaperone and outputs a signal to the management device 20 to that effect. The exit detection unit 25 may also be a reader that reads the RFID (Radio Frequency Identification) provided to the subject and their chaperone. The exit detection unit 25 may also be a camera with an image recognition function that recognizes the subject and their chaperone based on the registered data 32.

[0042] <10. Functions and Operation> The functions and operation of the registration device 10, the management device 20, and the terminal device 24 will be described below.

[0043] <10-1. Monitoring> The surveillance camera 21 is always in operation. Therefore, the surveillance camera 21 continuously films its surroundings and immediately outputs the captured video to the management device 20. The management device 20 continuously records video data 31 based on the video input from the registered camera 12 to the storage device 30. If the combination of the registered camera 12 and the registered terminal device 11 has the same functionality as the surveillance camera 21, the management device 20 continuously records video data 31 based on the video captured by the registered camera 12 input from the registered terminal device 11 to the storage device 30.

[0044] An administrator or other user operates the first input device 22 to give a command to the management device 20 to display the surveillance video. The management device 20 then transfers the video input from the surveillance camera 21 to the display device 14, and the display device 14 displays the video input from the management device 20. Alternatively, an administrator or other user may operate the terminal device 24 to issue a command to the management device 20 to display the surveillance video. In this case, the management device 20 transfers the video input from the surveillance camera 21 to the terminal device 24, and the terminal device 24 displays the video input from the management device 20.

[0045] <10-2. Generating a database of images of suspicious individuals> The database of images of suspicious individuals is generated by the generation unit of the management device 20. Specifically, it is generated as follows:

[0046] The management device 20 identifies suspicious individuals captured in the video by analyzing the footage input from the surveillance camera 21. For example, if the management device 20 detects suspicious behavior from a person captured in the video, it identifies that person as a suspicious individual. Suspicious behavior includes things like entering a restricted area, speaking to children or other minors, entering the facility from an entrance other than the main entrance, or visiting the facility at night or during other times when the facility is not in operation. The control device 20 calculates the degree of suspiciousness based on the suspicious behavior of the suspicious person. The degree of suspiciousness is the degree to which the suspicious person is suspicious. The management device 20 extracts images of suspicious individuals from the video and records these images as image data 35 in the storage device 30. Furthermore, the management device 20 associates a degree of suspiciousness with the image data 35 and records the degree of suspiciousness in the storage device 30. Image data 35 may be a video or a still image. Image data 35 may be an image of a suspicious person's face. Image data 35 may be an image that includes the face of that person.

[0047] If the suspicious person's face is not visible in the image data 35, the management device 20 accesses the storage device 30 and seeks the video data 31. While seeking the video data 31, the management device 20 compares and analyzes the image data 35 and the video data 31, identifies the person who is the same as the suspicious person from among the people appearing in the video of the video data 31, and extracts the face image of that person. Then, the management device 20 associates the data 36 of that face image with the image data 35 and stores it in the storage device 30.

[0048] Based on the above, the management device 20 generates a database of suspicious person image data 35 and facial image data 36.

[0049] <10-3. Registration> When the subject and their chaperone enter the facility, the subject and their chaperone are registered by the registration device 10. Specifically, this is as follows:

[0050] The registration terminal device 11 outputs a video signal of the registration screen to the display device 14, which then displays the registration screen according to the video signal. The registration screen is a screen that requests consent from the group leader for the following: (1) for the facility manager to register the subject and group leader in association with each other; (2) for the facility manager to monitor the subject and group leader by photographing them with cameras within the facility; and (3) for the facility manager to store images of the subject and group leader obtained through monitoring for a certain period of time. Such requests for consent are made from the perspective of protecting personal information.

[0051] If the group leader agrees to the terms presented on the registration screen, the group leader performs an operation to indicate their agreement on the second input device 13. The registration terminal device 11 then recognizes the agreement and activates the registration camera 12. The registration camera 12 captures the area in front of it and outputs the captured video as a video signal to the registration terminal device 11. The registration terminal device 11 then transfers the video signal input from the registration camera 12 to the display device 14. The display device 14 immediately displays the video according to the video signal input from the registration terminal device 11. When the group leader turns their face towards the registration camera 12, the registration terminal device 11 detects the group leader's face image from the video captured by the registration camera 12 and extracts the group leader's face image from that video. Subsequently, when the subject turns their face towards the registration camera 12, the registration terminal device 11 detects the subject's face image from the video captured by the registration camera 12 and extracts the subject's face image from that video. Subsequently, the registration terminal device 11 associates the facial images of the group leader and the participants with each other and transmits the facial images of the group leader and the participants to the management device 20. The management device 20 receives the facial images of the group leader and the participants. The management device 20 assigns identification numbers to the facial images of the group leader and the participants, associates the facial images of the group leader and the participants with the identification numbers with each other, and records them as registration data 32 in the storage device 30.

[0052] <10-3-1. Correspondence with facial images of past group leaders> Before recording the facial images of the group leader and the subject in the storage device 30, the management device 20 may refer to the registration data 32 already registered in the storage device 30. Specifically, the management device 20 refers to the registration data 32 in the storage device 30, compares the subject's latest facial image with the registration data 32, and searches the storage device 30 for registration data 32 that has the same facial image as the subject's latest facial image. The subject's latest facial image is extracted from the video footage taken by the registration camera 12 when the subject enters the facility. The facial image identical to the subject's latest facial image is extracted from the video footage taken by the registration camera 12 when the subject entered the facility in the past. If there is registration data 32 that has the same facial image as the subject's latest facial image, the management device 20 associates the subject's latest facial image, the current group leader's facial image, and the past group leader's facial images included in the registration data 32 with each other and updates the registration data 32. Therefore, the updated registration data 32 contains two or more facial images of the chaperones, and these facial images show the faces of the same or different chaperones. For example, the current chaperone's facial image is the subject's mother's facial image, and the past chaperone's facial image is the father's facial image. In the updated registration data 32, current information is associated with the current chaperone's facial image, and past information is associated with the past chaperone's facial image.

[0053] The relationship between past group leaders, the participants, and the current group leaders may be dissolved. Furthermore, past group leaders may lose their legal capacity. In such cases, the current group leader will notify the administrator upon entering the facility. The administrator can then operate the first input device 22 or terminal device 24 to specify registration data 32 containing the participant's face image, the current group leader's face image, and the past group leader's face image, and instruct the management device 20 to delink the past group leader's face image. In this case, the management device 20 decouples the past group leader's face image from the registration data 32 and updates the registration data 32. This delinks the past group leader's face image from the registration data 32.

[0054] <10-4. Determining Safety Status> After registration, the management device 20 extracts facial images of people from the video input from each surveillance camera 21 by performing image recognition processing on the video. Furthermore, the management device 20 instantly recognizes and tracks the target person in the video input from each surveillance camera 21 by comparing the facial images of people with the facial images of the target person. Similarly, the management device 20 instantly recognizes and tracks the group leader in the video input from each surveillance camera 21 by comparing the facial images of people with the facial images of the group leader. Note that if the target person is not captured by any of the surveillance cameras 21, the management device 20 will not recognize the target person. Similarly, if the group leader is not captured by any of the surveillance cameras 21, the management device 20 will not recognize the group leader.

[0055] The management device 20 determines whether the tracking leader is leading the tracking subject in the video footage from each surveillance camera 21. Specifically, the management device 20 determines whether the distance from the tracking subject to the tracking leader exceeds a first threshold for a certain period of time in the video footage from each surveillance camera 21. In other words, the management device 20 determines whether the leader has been outside the detection area, which has a radius of the first threshold centered on the subject, for a certain period of time. This certain period of time will be referred to as the time-out duration below.

[0056] The management device 20 performs the aforementioned determination process for each individual. If there are two or more group leaders, the management device 20 may determine whether the distance to the target person exceeded the first threshold for any of the group leaders for a duration equal to the time-out period. If there are two or more group leaders, the management device 20 may determine whether the distance to the subject exceeded the first threshold for the duration of the time-out period for any one of the group leaders. In this case, the first threshold, the time-out period, or both may differ for each group leader. For example, if the first threshold used for the current group leader is greater than the first threshold used for past group leaders, past group leaders are more likely to determine the subject to be dangerous than the current group leaders. For example, if the time-out period used for the current group leader is longer than the time-out period used for past group leaders, past group leaders are more likely to determine the subject to be dangerous than the current group leaders. The reason for setting the first threshold or time-out period as described above is the low reliability of past registration data 32.

[0057] If the determination process is positive, that is, if the distance from the subject being tracked to the accompanying person being tracked in the video footage of each surveillance camera 21 exceeds the first threshold for the duration of the timeout, the management device 20 determines that the subject is in danger because the accompanying person is no longer leading the subject. If the determination process is negative, that is, if the distance from the subject to the accompanying person in the video footage of each surveillance camera 21 does not exceed the first threshold for the duration of the timeout, the management device 20 determines that the subject is safe because the accompanying person is leading the subject. Note that the distance from the subject to the accompanying person in the video footage of each surveillance camera 21 exceeding the first threshold corresponds to the accompanying person moving outside the detection area with the first threshold as the radius, centered on the subject.

[0058] The determination that a person is in danger is based on scenarios such as the person getting lost or the person separating from the chaperone while the chaperone is not paying attention.

[0059] The following is a specific example of the process for determining the safety of the person concerned. First, the management device 20 instantly calculates the distance from the subject to the leader in the video footage from each surveillance camera 21 and instantly compares that distance with a first threshold. If the distance from the subject to the leader exceeds the first threshold, the management device 20 starts timing. If, during timing, the distance from the subject to the leader falls below the first threshold before the timing time reaches the timeout duration, the management device 20 terminates timing and resets the timing time. After that, the management device 20 performs the same process. If the timing time reaches the timeout duration without the distance from the subject to the leader falling below the first threshold, the management device 20 determines that the subject is in danger. Until the management device 20 determines that the subject is in danger, the management device 20 determines that the subject is safe.

[0060] The first threshold may be a constant, or it may be a variable that changes due to some factor. If the first threshold is a variable and there are multiple subjects belonging to the target group, the control device 20 calculates the distance between each pair of subjects in that target group. The control device 20 then determines the first threshold based on the distance between the furthest pairs of subjects. The longer the distance, the smaller the first threshold. Therefore, the longer the distance, the more likely the subject is to be judged as dangerous. If the first threshold is a variable and there are multiple subjects belonging to the target group, the management device 20 estimates the age of each subject in that target group based on the images from each surveillance camera 21. The management device 20 determines the first threshold based on the highest age among these subjects. The lower the highest age, the smaller the first threshold. Therefore, subjects with a lower highest age are more likely to be judged as at risk.

[0061] The timeout duration can be a constant, or it can be a variable that changes due to some factor. If the time-out duration is a variable and there are multiple subjects belonging to the target group, the management device 20 calculates the distance between each pair of subjects in that target group. The management device 20 then determines the time-out duration based on the distance between the furthest pairs of subjects. The longer the distance, the shorter the time-out duration. Therefore, the longer the distance, the more likely the subject is to be judged as dangerous. If the time-out duration is a variable and there are multiple subjects belonging to the target group, the management device 20 estimates the age of each subject in that target group based on the video footage from each surveillance camera 21. The management device 20 determines the time-out duration based on the oldest age among these subjects. The lower the oldest age, the shorter the time-out duration. Therefore, subjects with a lower oldest age are more likely to be judged as being at risk. If the time-out duration is a variable and there are multiple leaders belonging to the target group, the management device 20 may determine the time-out duration based on the number of leaders whose distance to the target is less than or equal to the first threshold. The fewer such leaders there are, the shorter the time-out duration. Therefore, the fewer such leaders there are, the more likely the target is to be judged as being in danger.

[0062] While the management device 20 recognizes and tracks the subject in the video input from each surveillance camera 21, the management device 20 continuously performs the process of determining the safety of the subject. If the management device 20 determines that the subject is in danger, the management device 20 performs the following process.

[0063] <10-5. Detection of approach by unspecified individuals> While the management device 20 recognizes and tracks subjects in the video input from each surveillance camera 21, the management device 20 performs image recognition processing on the video input from each surveillance camera 21 to instantly recognize and track unspecified individuals in the video.

[0064] The management device 20 determines whether an unspecified person being tracked is approaching the subject being tracked in the video footage from each surveillance camera 21. Specifically, the management device 20 determines whether the distance from the subject being tracked to the unspecified person being tracked in the video footage from each surveillance camera 21 is less than or equal to a second threshold. In other words, the management device 20 determines whether the unspecified person has entered a second detection area with the second threshold as the radius, centered on the subject.

[0065] If the determination is positive, that is, if the distance from the subject being tracked to the unspecified person being tracked in the video footage of each surveillance camera 21 is less than or equal to the second threshold, the management device 20 detects the approach of the unspecified person to the subject. If the determination is negative, that is, if the distance from the subject being tracked to the unspecified person being tracked in the video footage of each surveillance camera 21 exceeds the second threshold, the management device 20 does not detect the approach of the unspecified person to the subject.

[0066] The second threshold may be a constant or a variable that changes due to some factor. If the second threshold is a variable, the management device 20 instantly calculates the distance from the subject to the guide in the video footage from each surveillance camera 21. The management device 20 then sets the second threshold higher the longer the distance. Therefore, as the guide moves away from the subject, the approach of an unspecified person to the subject becomes easier to detect.

[0067] While the management device 20 recognizes and tracks the subject in the video input from each surveillance camera 21, the management device 20 continuously performs the process of determining whether or not an unspecified person is approaching. If the management device 20 detects that an unspecified person is approaching the subject, the management device 20 performs the following process.

[0068] <10-6. Recognition of the faces of unspecified individuals> The management device 20 performs image recognition processing on the video input from each surveillance camera 21. This allows the management device 20 to determine whether or not it can recognize the face image of an unspecified person from the video. If the face of an unspecified person is visible in the video, the management device 20 recognizes the face image of the unspecified person. If the face of an unspecified person is not visible in the video, or if the face of an unspecified person is unclear, the management device 20 cannot recognize the face image of the unspecified person.

[0069] <10-7. Video Analysis> If the management device 20 cannot recognize the face image of an unspecified person, the management device 20 accesses the storage device 30 and seeks the video data 31. While seeking the video data 31, the management device 20 compares the video footage from each surveillance camera 21 with the video data 31 and analyzes them to identify the same person as the unspecified person from among the people appearing in the video of the video data 31, and extracts the image of that person. The image of the same person may be a video or a still image. There may be one image or multiple images of the same person. The image of the same person may be a face image of that person. The image of the same person may be an image that includes the face of that person.

[0070] Specifically, the management device 20 identifies the same person as an unspecified person from among the characters appearing in the video data 31 in the following manner.

[0071] The control device 20 extracts feature quantities of an unspecified person from the video footage of each surveillance camera 21 using skeletal detection technology or the like. The feature quantities of an unspecified person are, for example, feature quantities of the whole or part of the body of the unspecified person captured in the video, or feature quantities of the movements of the whole or part of the body of the unspecified person. The feature quantities of the whole or part of the body of an unspecified person refer to elements other than the face, such as the height of the unspecified person, the length of the torso, the length of the legs, and the length of the arms. The whole or part of the body of an unspecified person may also include the clothing worn by the unspecified person. The whole or part of the body of an unspecified person may also include accessories worn by the unspecified person. Accessories refer to, for example, hats, face masks, helmets, scarves, necklaces, and wristbands. The whole or part of the body of an unspecified person may also include luggage carried by the unspecified person. The movements of the whole or part of the body of an unspecified person refer to walking, for example, lifting and lowering the legs, swinging the arms, swaying the upper body, and shaking the head. Motion characteristics include, for example, stride length, walking speed, height of leg lift, speed of leg lift, speed of leg lowering, magnitude of arm swing, speed of arm swing, magnitude of upper body sway, speed of upper body sway, magnitude of head sway, and speed of head sway. The management device 20 uses skeletal detection technology or the like to extract feature quantities for each character from the video data 31, for each character that appears in the video. The character's feature quantities are, for example, feature quantities of the character's whole or part of their body as seen in the video, or feature quantities of the character's whole or part of their body movements. Furthermore, the management device 20 extracts images of the characters that appear in the video from the video data 31, for each character. For each character from which an image has been successfully extracted, the management device 20 associates the extracted image with its feature quantities. Subsequently, the management device 20 identifies the person who is the same as the unspecified person by comparing the feature quantities of the unspecified person extracted from the video footage of each surveillance camera 21 with the feature quantities of each character extracted from the video data 31. For example, the management device 20 may calculate the deviation between the feature quantities of the unspecified person and the feature quantities of each character, and identify the character with the smallest deviation as the same person as the unspecified person. Here, "smallest deviation" refers to the 1st, 2nd, or 3rd place, etc. The management device 20 may also calculate the deviation between the feature quantities of the unspecified person and the feature quantities of each character, and identify the character with a feature whose deviation is greater than or equal to a predetermined value as the same person as the unspecified person. Note that the identified same person is not limited to one person; there may be two or more. As described above, from among the characters appearing in video data 31, the same person as an unspecified person is identified, and the image of that same person is extracted and identified.

[0072] <10-7-1. De-identification> If a character identified as being the same as an unspecified person is a staff member of the facility, the identification of that character as an unspecified person may be removed as follows:

[0073] The management device 20 compares the extracted image of a person identified as identical to an unspecified person with the facial image data 33 to determine whether or not that person is a staff member of the facility. Specifically, the management device 20 extracts characteristic quantities of the person and the staff from the extracted image and image data, respectively, and determines whether or not that person is a staff member of the facility based on these characteristic quantities. If the determination is positive, that is, if the person is a staff member of the facility, the management device 20 de-identifies the person as being the same as an unspecified person. If the determination is negative, that is, if the person is not a staff member of the facility, the management device 20 maintains the identification of the person who has been identified as being the same as an unspecified person.

[0074] Furthermore, even if a person identified as being the same as an unspecified person is a staff member of the facility, the connection between that person and the target may be weak. In such cases, the management device 20 may determine, based on the schedule data 34, whether or not to de-identify the person as being the same as an unspecified person. For example, if a staff member is off duty or performing duties unrelated to the target, the management device 20 can recognize this from the schedule data 34, and therefore maintains the identification of the person as being the same as an unspecified person. For example, if a staff member is on duty or performing duties related to the target, the management device 20 can recognize this from the schedule data 34, and therefore de-identifies the person as being the same as an unspecified person.

[0075] <10-8. Display of extracted images of characters identical to unspecified individuals> After identifying a person identical to the unspecified person, the management device 20 displays the extracted image associated with the person identical to the unspecified person on the first display device 23. For example, the management device 20 may display the extracted image of the person identical to the unspecified person on the first display device 23 in parallel with the immediate display of the video input from each surveillance camera 21 on the first display device 23. The management device 20 may combine the extracted image of the person with the video input from each surveillance camera 21 and immediately display the combined video on the first display device 23. The management device 20 may combine the extracted image of the person near the unspecified person in the video input from each surveillance camera 21 and immediately display the combined video on the first display device 23. If there are multiple extracted images, the management device 20 may arrange these extracted images in order of clarity and display them on the first display device 23. If there are multiple extracted images, the management device 20 may sequentially display these extracted images on the first display device 23 in order of clarity. If there are multiple extracted images, the management device 20 may arrange these extracted images in ascending or descending order of recording time and display them on the first display device 23. If there are multiple extracted images, the management device 20 may sequentially display these extracted images on the first display device 23 in ascending or descending order of recording time. If there are multiple extracted images, the management device 20 may classify the extracted images by the pattern of the clothing of the people in the images and display the extracted images on the first display device 23 according to the classification. The management device 20 may display the extracted images on the first display device 23 along with information that identifies the location of the surveillance camera 21 that acquired the source video data 31 from which the extracted images were taken. The information that identifies the location of the surveillance camera 21 is, for example, a map of the facility. An administrator or the like may operate a terminal device 24, and the terminal device 24 may communicate with the management device 20, causing the terminal device 24 to display extracted images of people who are the same as an unspecified person.

[0076] Furthermore, if a character identified as being the same as an unspecified person is de-identified, the extracted image of that character will not be displayed.

[0077] When an image of a person identical to an unspecified individual is displayed on the first display device 23 or terminal device 24, facility managers and chaperones can see the image and identify the unspecified individual.

[0078] <10-9. Display of images of suspicious individuals identical to unspecified individuals> The management device 20 identifies an unspecified person from among the suspicious individuals by comparing the video footage from each surveillance camera 21 with the image data 35 in the database. The identification method is as follows: First, the management device 20 extracts the features of the unspecified person from the video footage from each surveillance camera 21 using techniques such as skeletal detection. The management device 20 also extracts the features of the suspicious person for each image data 35 in the database by performing feature extraction processing on each image data 35. The management device 20 identifies an unspecified person from among the suspicious individuals by comparing the features of the unspecified person extracted from the video footage from each surveillance camera 21 with the features of each suspicious person extracted from the image data 35. In this process, the management device 20 performs feature extraction processing and matching processing on the image data 35 in order of decreasing suspiciousness.

[0079] After identifying a suspicious person who is identical to an unspecified person, the management device 20 displays an image on the first display device 23 according to the image data 35 and face image data 36 of the suspicious person who is identical to the unspecified person. This image may be displayed together with an extracted image of the person who is identical to the unspecified person.

[0080] <10-10. Deleting Registration> The management device 20 may delete the registered data 32 from the storage device 30 after a certain period of time has elapsed since the registration of the registered data 32. This certain period of time may be, for example, six months, one year, or two years.

[0081] When the subject and their chaperone leave the facility through the exit, the exit detection unit 25 detects their departure and outputs a signal to the management device 20. Upon receiving this signal, the management device 20 may delete the registered data 32 from the storage device 30.

[0082] The administrator may use the first input device 22 or terminal device 24 to issue a command to the management device 20 to delete the registered data 32, causing the management device 20 to delete the registered data 32 from the storage device 30.

[0083] Thus, the deletion of registration data 32 is carried out from the perspective of protecting personal information.

[0084] <11. Technically advantageous effects> The facial images of the subject and their chaperone are registered as registration data 32. The registration of the facial images of the subject and their chaperone contributes to enabling the management device 20 to recognize and track the subject and their chaperone from the people captured in the video footage taken by the surveillance camera 21. In turn, the registration of the facial images of the subject and their chaperone contributes to enabling the management device 20 to determine the safety of the subject.

[0085] If the group leader remains outside the detection area, which has a radius of a first threshold centered on the subject, for a certain period of time, the management device 20 recognizes and tracks the unspecified person in the video footage captured by each surveillance camera 21. The management device 20 determines whether or not the unspecified person is approaching the subject in the video. If the management device 20 detects the approach of the unspecified person, and if the management device 20 is unable to recognize the face of the unspecified person even after performing image recognition processing on the video, the management device 20 compares the video footage from the recorded video data 31 with the video footage from the surveillance cameras 21, analyzes them, and identifies the same person as the unspecified person from among the people appearing in the video footage 31. The management device 20 extracts the image of the identified person from the video data 31. That image is displayed on the first display device 23. Facility managers and group leaders can then view the image and identify the unspecified person. [Explanation of Symbols]

[0086] 1. Monitoring System 12 Registered Cameras 10. Registration device 11. Registered terminal device 20 Management device 21 surveillance cameras 22 First Input Device 23. First display device 30 Storage devices 31 Video Data 32. Registered Data 33. Facial image data 34 Schedule Data 35 Image data 36. Facial image data

Claims

1. A first determination unit determines whether or not an unspecified person is approaching at least one subject in the video footage captured by surveillance cameras installed within the facility, When the first determination unit detects the approach of the unspecified person to the target person, a second determination unit determines whether or not the face image of the unspecified person can be recognized from the video, If the second determination unit determines that it cannot recognize the face image of the unspecified person, the extraction unit compares the video with video data previously taken within the facility, identifies the same person as the unspecified person from among at least one person appearing in the video data, and extracts an image of the same person from the video. The extraction unit includes a display unit that displays the extracted image, A control device equipped with the following features.

2. The extraction unit, The features of the unspecified person are extracted from the video, the features of the characters are extracted from the video data, the same person is identified from among the characters by comparing the features of the unspecified person and the features of the characters, and an image of the same person is extracted from the video data. The control device according to claim 1.

3. The aforementioned image of the same person is an image that includes the face of the same person. The control device according to claim 1 or 2.

4. The system further includes a third determination unit that uses the facial image of the subject and the facial image of at least one chaperone to recognize and track the subject and the chaperone in the video, and determines whether it is safe or not based on whether the chaperone is leading the subject in the video. If the third determination unit determines that it is unsafe, the first determination unit determines whether or not the unspecified person approached the subject. The control device according to claim 1 or 2.

5. The third determination unit determines whether the distance from the subject to the leader in the video exceeded a threshold for a time-out duration. The third determination unit determines that it is unsafe if the distance from the subject to the guide exceeds the threshold for the duration of the timeout, and determines that it is safe if the distance from the subject to the guide does not exceed the threshold for the duration of the timeout. The control device according to claim 4.

6. The number of the aforementioned subjects is multiple, The longer the distance between the subjects, the smaller the threshold value the third determination unit sets. The control device according to claim 5.

7. The number of the aforementioned subjects is multiple, The lower the age of the highest age among the subjects, the smaller the threshold value set by the third determination unit. The control device according to claim 6.

8. The number of the aforementioned subjects is multiple, The longer the distance between the subjects, the shorter the time-out duration is determined by the third determination unit. The control device according to claim 5.

9. The number of the aforementioned subjects is multiple, The lower the age of the oldest person among the subjects, the shorter the time-out duration will be determined by the third determination unit. The control device according to claim 5.

10. The number of group leaders is multiple, The fewer the number of guides whose distance to the subject is below the threshold, the shorter the time-out duration is determined by the third determination unit. The control device according to claim 5.

11. The number of group leaders is multiple, The third determination unit determines whether the state in which the distance to the subject exceeds the threshold for any of the group leaders continued for the duration of the time-out period. The control device according to claim 5.

12. The number of group leaders is multiple, The third determination unit determines whether the distance to the subject exceeded the threshold for any one of the group leaders for the duration of the time-out period. The control device according to claim 5.

13. The threshold, the time-out duration, or both of these differ for each group leader. The control device according to claim 12.

14. The facial images of the subject and the accompanying person were registered by being captured by a registration camera installed at the entrance of the facility. The control device according to claim 4.

15. The registration terminal device presents a registration screen to the group leader requesting their consent, and when the group leader inputs their consent to the registration terminal device, the registration terminal device causes the registration camera to capture images of the subject's face and the group leader's face. The control device according to claim 14.

16. The number of group leaders is multiple, The facial images of the aforementioned subjects and the facial images of any of the first aforementioned chaperones were registered by being captured by a registration camera installed at the entrance of the facility. Any other second facial image of the aforementioned group leader is a facial image that was previously registered and associated with the same facial image as the subject's facial image. The control device according to claim 4.

17. The third determination unit determines whether the distance from the subject to the leader in the video exceeded a threshold for a time-out duration. If the distance from the subject to the guide exceeds the threshold for the duration of the time-out period, the third determination unit determines that it is unsafe, and if the distance from the subject to the guide does not exceed the threshold for the duration of the time-out period, the third determination unit determines that it is safe. The threshold adopted by the first leader is greater than the threshold adopted by the second leader. The control device according to claim 16.

18. The third determination unit determines whether the distance from the subject to the leader in the video exceeded a threshold for a time-out duration. If the distance from the subject to the guide exceeds the threshold for the duration of the time-out period, the third determination unit determines that it is unsafe, and if the distance from the subject to the guide does not exceed the threshold for the duration of the time-out period, the third determination unit determines that it is safe. The time-out duration adopted by the first group leader is longer than the time-out duration adopted by the second group leader. The control device according to claim 16.

19. The system further includes a fourth determination unit that determines whether a person identified as identical to the unspecified person is a staff member of the facility by comparing the image extracted by the extraction unit with image data of the facility's staff, If the fourth determination unit determines that the person identified as the same as the aforementioned unspecified person is a staff member of the facility, the display unit will not display the image extracted by the extraction unit. The control device according to claim 1 or 2.

20. A generation unit generates a database of images of suspicious persons captured in the video by analyzing the video, A second display unit that identifies the same person as the unspecified person from among the suspicious persons by comparing the image in the database with the video, and displays the image of the suspicious person who is the same as the unspecified person, The control device according to claim 1 or 2, comprising:

Images