Management apparatus

The management device addresses the inability of existing technologies to identify hidden faces by using a system to detect and recognize individuals in real-time surveillance, improving safety in facilities by identifying malicious actors and preventing potential threats.

WO2024236933A1PCT designated stage expired Publication Date: 2024-11-21KONICA MINOLTA INC
View PDF 15 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/013951
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-16
Filing Date
2024-04-04
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Existing crime prevention technologies in facilities, such as those used in medical and commercial facilities, are ineffective in identifying malicious individuals who hide their faces to avoid detection, and they do not prevent potential kidnappings or other malicious approaches to children or vulnerable individuals.

Method used

A management device with a first determination section to detect unknown persons approaching a target, a second determination section to recognize faces, and an extraction section to compare video data with past footage to identify the person, even if their face is not visible, and display the extracted image for identification.

Benefits of technology

Enables the early identification of unspecified persons, including malicious actors, by tracking and recognizing individuals in real-time surveillance, even if they hide their faces, thereby enhancing safety measures in crowded facilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024013951_21112024_PF_FP_ABST
    Figure JP2024013951_21112024_PF_FP_ABST
Patent Text Reader

Abstract

The purpose of the present disclosure is to provide a management apparatus capable of contributing to identification of an unidentified person. This management apparatus includes a first determination unit, a second determination unit, an extraction unit, and a display unit. The first determination unit determines, in a video captured by a monitoring camera installed in a facility, whether or not there is an approach by an unidentified person to at least one subject person. The second determination unit determines whether or not the face image of the unidentified person can be recognized from the video when the first determination unit has detected the approach of the unidentified person to the subject person. When the second determination unit has determined that the face image of the unidentified person cannot be recognized, the extraction unit compares the video with moving image data captured in the past in the facility, identifies the same person as the unidentified person from at least one character appearing in a moving image of the moving image data, and extracts an image of the same person from the moving image. The display unit displays the image extracted by the extraction unit.
Need to check novelty before this filing date? Find Prior Art

Description

management device

[0001] The technology disclosed in this specification relates to a management device.

[0002] Since an unspecified number of people usually visit facilities such as medical facilities and commercial facilities, crime prevention at such facilities is an important issue for facility managers. Patent Document 1 discloses a lost child detection technology as one type of crime prevention technology for facilities. This lost child detection technology is as follows.

[0003] When a chaperone and children enter the facility, facial images of the chaperone and children are captured individually by a registered camera. When the chaperone enters their mobile phone email address, that email address is registered in association with the facial images of the chaperone and children. Multiple surveillance cameras are installed within the facility, and children are monitored by the surveillance cameras. The surveillance camera footage is transferred to a face matching server, which compares the footage with the facial images of the chaperone and children to recognize the chaperone and children in the footage. The face matching server calculates the positions of the chaperone and children based on the footage and measures the time they are separated from each other. If the chaperone and children remain separated from each other for a certain period of time, the face matching server detects this. The face matching server then sends an email to the chaperone's email address. This allows the chaperone to recognize that the child is lost.

[0004] Japanese Patent Application Laid-Open No. 2007-60528

[0005] The technology of Patent Document 1 is effective for early detection of lost children, and ultimately for early discovery of lost children. However, the technology of Patent Document 1 cannot prevent malicious individuals with the intent of kidnapping or otherwise approaching children. In particular, malicious individuals often hide their faces to avoid being detected committing crimes, and the technology of Patent Document 1 cannot identify such individuals.

[0006] Therefore, the technology disclosed in the present invention aims to provide a management device that can contribute to identifying unspecified individuals such as malicious individuals.

[0007] In order to solve at least one of the above problems, this specification discloses a management device. The management device includes a first determination unit, a second determination unit, an extraction unit, and a display unit. The first determination unit determines whether an unspecified person is approaching at least one target in video captured by a surveillance camera installed in a facility. When the first determination unit detects the unspecified person approaching the target, the second determination unit determines whether a facial image of the unspecified person can be recognized from the video. When the second determination unit determines that the facial image of the unspecified person cannot be recognized, the extraction unit compares the video with video data previously captured within the facility, identifies a person identical to the unspecified person from at least one character appearing in a video of the video data, and extracts an image of the same person from the video. The display unit displays the image extracted by the extraction unit.

[0008] Even if the face of an unspecified person is not captured in the video captured by the surveillance camera, if the same person as the unspecified person is captured in a past video of the facility, the image of that person will be extracted from the video and displayed. Thus, the technology disclosed in this specification provides a management device that contributes to the identification of unspecified persons.

[0009] FIG. 1 is a block diagram in which components of a monitoring system are represented by blocks.

[0010] One or more embodiments will be described below with reference to the drawings. Features and technical advantages of the embodiments will be more fully understood from the following detailed description and drawings. However, the scope of the present invention is not limited to the embodiments disclosed below. Because the drawings are provided for illustrative purposes only, the scope of the present invention is not limited to the illustrations in the drawings.

[0011] In the following description, a facility refers to a facility where an unspecified number of people gather, such as a medical facility, an amusement facility, a commercial facility, an exhibition facility, or an event facility. For example, a facility may be a hospital, a commercial complex, an amusement park, a zoo, or a sports viewing facility. A facility may be indoors, outdoors, or both. A target refers to a person whose safety is confirmed using the monitoring system 1 described below. A target may be, for example, a child, such as an infant, a young child, an elementary school student, or a junior high school student. A target may also be an elderly person. A target may also be a person with insufficient judgment ability. A chaperone refers to a person who leads a target, primarily a guardian responsible for protecting and supervising the target. Typically, a chaperone is a parent of the target. A chaperone may also be a person who assists, supports, or assists the target. A target group refers to a group consisting of a target and a chaperone who leads the target. The number of targets belonging to each target group may be single or multiple. For example, siblings may belong to a common target group. The number of chaperones belonging to each target group may be single or multiple. For example, parents may be chaperones and belong to a common target group.

[0012] 1 is a block diagram of a monitoring system 1. The monitoring system 1 monitors a target person who visits a facility and contributes to identifying unspecified persons who approach the target person. The monitoring system 1 includes a registration device 10, a management device 20, one or more monitoring cameras 21, a first input device 22, a first display device 23, a terminal device 24, an exit detection unit 25, and a storage device 30.

[0013] 2. Registration Device The registration device 10 is installed at the entrance of the facility. The registration device 10 has a registration terminal device 11, at least one registration camera 12, at least one second input device 13, and at least one second display device 14. The registration terminal device 11, the registration camera 12, the second input device 13, and the second display device 14 may be integrated into one device.

[0014] The registration camera 12 includes an imaging element, a lens, an image processing circuit, etc. The registration camera 12 captures an image of a scene ahead that the registration camera 12 is pointing at, and outputs the captured image to the registration terminal device 11.

[0015] The second input device 13 is installed at the entrance of the facility near the registration camera 12. The second input device 13 is an input device such as a keyboard, a mouse, a touch panel, a touch pad, a star, a pointing device, a key, or a push button. When the second input device 13 is operated, the second input device 13 outputs a signal corresponding to the operation to the registration terminal device 11.

[0016] The second display device 14 is, for example, a liquid crystal display device or an organic electroluminescence (EL) display device, and displays an image according to the image signal transferred from the registration terminal device 11.

[0017] The registration terminal device 11 is a computer having a central processing unit (CPU), a graphics processing unit (GPU), a random access memory (RAM), a storage medium, a bus, a bus controller, an interface circuit, etc. The registration terminal device 11 is installed at the entrance of the facility.

[0018] The storage medium of the registration terminal device 11 includes a semiconductor storage device or a magnetic storage device. This storage medium stores programs that can be executed by the CPU of the registration terminal device 11. When the CPU executes the programs, the registration terminal device 11 performs various functions and performs processing based on these functions.

[0019] The registration terminal device 11 generates a video signal for a screen to realize a GUI (Graphical User Interface) and outputs the video signal to the second display device 14. Then, the second display device 14 displays a screen according to the video signal.

[0020] The registration terminal device 11 transfers the image input from the registration camera 12 to the second display device 14. Then, the second display device 14 displays the image input from the registration terminal device 11.

[0021] The registration terminal device 11 is connected to a local area network. The registration terminal device 11 communicates with the management device 20 via the local area network. The registration terminal device 11 may also communicate with the management device 20 via the local area network and the Internet.

[0022] <3. First Input Device> The first input device 22 is connected to the management device 20. The first input device 22 is an input device such as a keyboard, a mouse, a touch panel, a touch pad, a star, a pointing device, a key, or a push button. When the first input device 22 is operated, the first input device 22 outputs a signal corresponding to the operation to the management device 20.

[0023] 4. Second Display Device The first display device 23 is, for example, a liquid crystal display device or an organic EL display device. The first display device 23 displays an image in accordance with a video signal transferred from the management device 20.

[0024] 5. Surveillance Camera The surveillance camera 21 has an imaging element, a lens, an image processing circuit, etc. The surveillance camera 21 is installed within a facility. When there are multiple surveillance cameras 21, the surveillance cameras 21 are installed in various locations within the facility and are distributed throughout the facility. The surveillance camera 21 is connected to the management device 20. The surveillance camera 21 may be connected to the management device 20 via a serial bus such as USB (registered trademark). The surveillance camera 21 may be connected to the management device 20 via a local area network. The surveillance camera 21 may be connected to the management device 20 via the local area network and the Internet.

[0025] The surveillance camera 21 captures images of its surroundings and instantly outputs the captured images to the management device 20. The management device 20 instantly records video data 31 based on the images input from the surveillance camera 21 in the storage device 30. The images input from the surveillance camera 21 to the management device 20 are current images, and the video of the video data 31 is from the past. Note that the surveillance camera 21 may instantly output the images to the storage device 30, thereby instantly recording the video data 31 based on the images in the storage device 30. The video may be encoded in any of the surveillance camera 21, the management device 20, or the storage device 30.

[0026] To prepare for failures such as power outages and network malfunctions, the surveillance camera 21 may have storage such as a semiconductor storage device or a magnetic storage device. In this case, before and during a failure, the surveillance camera 21 captures images of its surroundings and stores the captured images in the storage as video data. After the failure is resolved, the administrator operates the surveillance camera 21 or the surveillance camera 21 detects that the failure has been resolved, causing the surveillance camera 21 to read the video data in the storage and transfer the video data to the management device 20 or the storage device 30. The management device 20 transfers the video data received from the surveillance camera 21 to the storage device 30. The storage device 30 records the video data received from the surveillance camera 21 or the management device 20 as video data 31. As a result, the video data stored in the storage of the surveillance camera 21 is copied to the storage device 30 as video data 31. Note that the storage of the surveillance camera 21 may be detachable from the surveillance camera 21, and the administrator may manually copy the video data 31 from the storage of the surveillance camera 21 to the storage device 30.

[0027] The combination of the registered camera 12 and the registration terminal device 11 may have the same functions as the surveillance camera 21. In other words, the registered camera 12 and the registration terminal device 11 may always be in operation, and the registration terminal device 11 may instantly transfer the video images captured by the registered camera 12 to the management device 20. In this case, the management device 20 instantly records video data 31 based on the video images input from the registration terminal device 11 in the storage device 30.

[0028] 6. Storage Device The storage device 30 is a semiconductor storage device, a magnetic storage device, a NAS (Network Attached Storage), a data server, a file server, or a cloud computing system. The storage device 30 may be connected to the management device 20 via a serial bus. The storage device 30 may be connected to the management device 20 via a local area network. The storage device 30 may be connected to the management device 20 via a local area network and the Internet. The storage device 30 stores data transferred from the management device 20, reads the stored data, and transfers the data to the management device 20.

[0029] The storage device 30 stores one or more pieces of video data 31. The video data 31 is encoded data of video captured by the surveillance camera 21. The video data 31 is associated with the surveillance camera 21 that captured the video of the video data 31.

[0030] The storage device 30 stores one or more registration data 32. The registration data 32 is registered by the registration terminal device 11. The registration data 32 is data created for each target group. The registration data 32 has an identification number, a facial image of the subject, and a facial image of the chaperone. The identification number is a unique value assigned to the target group. The identification number is a value specific to the target group. For each subject belonging to the target group, a facial image of the subject is included in the registration data 32. The face of the subject belonging to the target group is reflected in the facial image of the subject. For each chaperone belonging to the target group, a facial image of the chaperone is included in the registration data 32.

[0031] The storage device 30 stores one or more pieces of facial image data 33 in advance. The facial image data 33 is a still image or video showing the face of a staff member at the facility. A different piece of facial image data 33 is prepared for each staff member. A staff member identification number is associated with the facial image data 33. This allows the facial image data 33 to be assigned to the staff member.

[0032] The storage device 30 stores one or more schedule data 34 in advance. The schedule data is time-series data of the schedules of facility staff. A staff schedule is time-series data consisting of arrival time, departure time, break time, on-duty time, off-duty time, time to perform specific tasks, and time to be in specific locations within the facility. A different schedule data 34 is prepared for each staff member. A staff member identification number is associated with the schedule data 34, and the schedule data 34 is assigned to the staff member accordingly.

[0033] 7. Management Device The management device 20 has at least one computer, and the computer has a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), RAM (Random Access Memory), a storage medium, a bus, a bus controller, an interface circuit, etc. The management device 20 may be composed of a single computer. The management device 20 may be composed of multiple computers capable of distributed processing or parallel processing. The management device 20 may be a cloud computing system. The management device 20 may be installed in a management room within a facility. The management device 20 may be installed in a data center outside the facility.

[0034] The storage medium of the management device 20 includes a semiconductor storage device or a magnetic storage device. This storage medium stores programs that can be executed by the CPU of the management device 20. When this CPU executes the programs, the management device 20 performs various functions and performs processing based on these functions.

[0035] The management device 20 is connected to a local area network. The management device 20 communicates with the registration terminal device 11 via the local area network. The management device 20 may also communicate with the registration terminal device 11 via the local area network and the Internet.

[0036] The management device 20 records data in the storage device 30. The management device 20 reads data from the storage device 30.

[0037] The management device 20 generates video signals for various screens to realize a GUI (Graphical User Interface) and outputs the video signals to the first display device 23. In response to this, the first display device 23 displays a screen in accordance with the video signals.

[0038] The management device 20 decodes the video input from the registered camera 12 as necessary and transfers the video to the first display device 23. The first display device 23 then displays the video input from the management device 20.

[0039] The management device 20 encodes the video input from the registered camera 12 as necessary, and records video data 31 based on the video in the storage device 30 .

[0040] 8. Terminal Device The terminal device 24 is composed of a general-purpose computer system. A general-purpose computer system refers to a computer system, such as a mobile phone, smartphone, tablet computer, laptop computer, or desktop computer, on which a general-purpose operating system (OS) is installed. Examples of general-purpose OSs include Windows (registered trademark), Android (registered trademark), iOS (registered trademark), macOS (registered trademark), Linux (registered trademark), and Unix (registered trademark). The terminal device 24 has a display device and an input device. The terminal device 24 is connected to the management device 20 via a local area network, the Internet, or both. The terminal device 24 stores an application program. The application program allows the terminal device 24 to access the management device 20 and function as a console for the management device 20. The terminal device 24 functions as a console for the management device 20. The terminal device 24 may be installed in a management room or the like within a facility. The terminal device 24 may be portable or stationary.

[0041] 9. Exit Detection Unit The exit detection unit 25 is installed at the exit of the facility. The exit detection unit 25 detects the exit of the subject and the chaperone and outputs a signal to that effect to the management device 20. The exit detection unit 25 may be a reader that reads RFID (Radio Frequency Identification) cards provided to the subject and the chaperone. The exit detection unit 25 may also be a camera with an image recognition function that recognizes the subject and the chaperone based on the registration data 32.

[0042] 10. Functions and Operations The functions and operations of the registration device 10, management device 20, and terminal device 24 will be described below.

[0043] <10-1. Surveillance> The surveillance camera 21 is always in operation. Therefore, the surveillance camera 21 continuously captures images of its surroundings and instantly outputs the captured images to the management device 20. The management device 20 continuously records video data 31 based on images input from the registered cameras 12 in the storage device 30. Note that, if the combination of the registered cameras 12 and the registration terminal device 11 has the same functions as the surveillance camera 21, the management device 20 continuously records video data 31 based on images captured by the registered cameras 12 input from the registration terminal device 11 in the storage device 30.

[0044] An administrator or the like operates the first input device 22 to give a command to display the surveillance video to the management device 20. In response, the management device 20 transfers the video input from the surveillance camera 21 to the display device 14, and the display device 14 displays the video input from the management device 20. The administrator or the like may also operate the terminal device 24 to give a command to display the surveillance video to the management device 20. In this case, the management device 20 transfers the video input from the surveillance camera 21 to the terminal device 24, and the terminal device 24 displays the video input from the management device 20.

[0045] <10-2. Creation of a database of images of suspicious persons> The database of images of suspicious persons is created by the creation unit of the management device 20. Specifically, this is as follows.

[0046] The management device 20 analyzes the video input from the surveillance camera 21 to identify a suspicious individual captured in the video. For example, if the management device 20 detects suspicious behavior by an individual captured in the video, it identifies the individual as a suspicious individual. Suspicious behavior includes entering a restricted area, calling out to a child or young child, entering a facility through a non-entrance entrance, or visiting a facility during closed hours, such as at night. The management device 20 calculates a suspiciousness level based on the suspicious behavior of the suspicious individual. The suspiciousness level indicates the degree of suspiciousness of the suspicious individual. The management device 20 extracts an image of the suspicious individual from the video and records the image as image data 35 in the storage device 30. Furthermore, the management device 20 associates a suspiciousness level with the image data 35 and records the suspiciousness level in the storage device 30. The image data 35 may be a video or a still image. The image data 35 may be a facial image of the suspicious individual. The image data 35 may be an image including the individual's face.

[0047] If the suspicious person's face is not shown in the image data 35, the management device 20 accesses the storage device 30 and seeks the video data 31. While seeking the video data 31, the management device 20 compares the image data 35 with the video data 31, analyzes them, identifies the person who is the suspicious person from among the characters shown in the video data 31, and extracts a facial image of that person. The management device 20 then associates the facial image data 36 with the image data 35 and stores it in the storage device 30.

[0048] As a result of the above, the management device 20 generates a database of image data 35 and face image data 36 of suspicious individuals.

[0049] <10-3. Registration> When the subject and the leader enter the facility, they are registered by the registration device 10. Specifically, this is as follows.

[0050] The registration terminal device 11 outputs a video signal of the registration screen to the display device 14, and the display device 14 displays the registration screen in accordance with the video signal. The registration screen is a screen that requests the chaperone to agree to (1) the facility manager or the like registering the subject and the chaperone in association with each other, (2) the facility manager or the like monitoring the subject and the chaperone by photographing them with a camera within the facility, and (3) the facility manager or the like storing for a certain period of time images of the subject and the chaperone obtained through monitoring. Such consent requests are made from the perspective of protecting personal information.

[0051] If the leader agrees to the items presented on the registration screen, the leader operates the second input device 13 to indicate consent. The registration terminal device 11 then recognizes the agreement and activates the registration camera 12. The registration camera 12 captures an image of the area in front of the registration camera 12 and outputs the captured image as a video signal to the registration terminal device 11. The registration terminal device 11 transfers the video signal input from the registration camera 12 to the display device 14. The display device 14 instantly displays an image in accordance with the video signal input from the registration terminal device 11. When the leader turns his / her face toward the registration camera 12, the registration terminal device 11 detects the leader's facial image from the video captured by the registration camera 12 and extracts the leader's facial image from the video. When the subject subsequently turns their face toward the registration camera 12, the registration terminal device 11 detects the subject's facial image from the video captured by the registration camera 12 and extracts the subject's facial image from the video. Thereafter, the registration terminal device 11 associates the facial images of the leader and the subject with each other and transmits the facial images of the leader and the subject to the management device 20. The management device 20 receives the facial images of the leader and the subject. The management device 20 assigns identification numbers to the facial images of the leader and the subject, associates the facial images of the leader and the subject with the identification numbers, and records them as registration data 32 in the storage device 30.

[0052] <10-3-1. Correspondence with Past Facial Images of Chaperones> Before recording facial images of the chaperone and the subject in the storage device 30, the management device 20 may refer to the registration data 32 already registered in the storage device 30. Specifically, the management device 20 refers to the registration data 32 in the storage device 30, compares the subject's latest facial image with the registration data 32, and searches the storage device 30 for registration data 32 containing the same facial image as the subject's latest facial image. The subject's latest facial image is extracted from video captured by the registration camera 12 when the subject enters the facility. The same facial image as the subject's latest facial image is extracted from video captured by the registration camera 12 when the subject previously entered the facility. If registration data 32 containing the same facial image as the subject's latest facial image exists, the management device 20 correlates the subject's latest facial image with the facial image of the current chaperone and the facial image of the previous chaperone included in the registration data 32, and updates and records the registration data 32. Therefore, the updated and recorded registration data 32 contains two or more face images of chaperones, and these face images may show the faces of the same or different chaperones. For example, the face image of the current chaperone is the face image of the subject's mother, and the face image of the past chaperone is the face image of the subject's father. In the updated and recorded registration data 32, information indicating the current chaperone is associated with the face image of the current chaperone, and information indicating the past is associated with the face image of the past chaperone.

[0053] The relationship between the past chaperone and the subject and the current chaperone may be dissolved. The past chaperone may also lose his or her capacity to be held responsible. In such cases, the current chaperone notifies the administrator upon entering the facility. The administrator can then operate the first input device 22 or the terminal device 24 to specify registration data 32 containing the subject's facial image, the current chaperone's facial image, and the past chaperone's facial image, and then instruct the management device 20 to cancel the association of the past chaperone's facial image. In this case, the management device 20 separates the registration data 32 from the past chaperone's facial image and updates and records the registration data 32. This cancels the association of the past chaperone's facial image with the registration data 32.

[0054] <10-4. Safety Determination> After registration, the management device 20 extracts a person's facial image from the video by performing image recognition processing on the video input from each monitoring camera 21. Furthermore, the management device 20 compares the person's facial image with the target's facial image to instantly recognize and track the target in the video input from each monitoring camera 21. Similarly, the management device 20 compares the person's facial image in the video input from each monitoring camera 21 with the leader's facial image to instantly recognize and track the leader in the video input from each monitoring camera 21. Note that if the target is not captured by any of the monitoring cameras 21, the management device 20 does not recognize the target. Similarly, if the leader is not captured by any of the monitoring cameras 21, the management device 20 does not recognize the leader.

[0055] The management device 20 determines whether the chaperone being tracked is leading the subject being tracked in the video from each monitoring camera 21. Specifically, the management device 20 determines whether the distance from the subject being tracked to the chaperone being tracked exceeds a first threshold for a certain period of time in the video from each monitoring camera 21. In other words, the management device 20 determines whether the chaperone has been outside a detection area centered on the subject for a certain period of time. This certain period of time is hereinafter referred to as a timeout duration. The detection area refers to a circular area centered on the subject, with a radius from the subject equal to the first threshold.

[0056] The management device 20 performs the above-described determination process for each subject. When there are two or more chaperones, the management device 20 may determine whether the distance to the subject exceeds the first threshold for all chaperones for the timeout duration. When there are two or more chaperones, the management device 20 may determine whether the distance to the subject exceeds the first threshold for any one chaperone for the timeout duration. In this case, the first threshold, the timeout duration, or both may be different for each chaperone. For example, if the first threshold used by the current chaperone is greater than the first threshold used by past chaperones, the subject is more likely to be determined to be in danger by past chaperones than by the current chaperone. For example, if the timeout duration used by the current chaperone is longer than the timeout duration used by past chaperones, the subject is more likely to be determined to be in danger by past chaperones than by the current chaperone. The first threshold or timeout duration is set as described above because the reliability of the past registration data 32 is low.

[0057] If the determination in the determination process is positive, that is, if the distance from the subject being tracked to the leader being tracked in the video of each monitoring camera 21 exceeds the first threshold and continues for the timeout duration, the management device 20 determines that the subject is in danger because the leader is not leading the subject. If the determination in the determination process is negative, that is, if the distance from the subject to the leader in the video of each monitoring camera 21 does not exceed the first threshold and continues for the timeout duration, the management device 20 determines that the subject is safe because the leader is leading the subject. Note that the distance from the subject to the leader in the video of each monitoring camera 21 exceeding the first threshold corresponds to the leader leaving a detection area with the first threshold as its radius and centered on the subject.

[0058] The determination that the subject is in danger may be made, for example, if the subject gets lost or if the subject becomes separated from the leader while the leader is away from the subject.

[0059] A specific example of the process for determining the safety of a subject is as follows. First, the management device 20 instantly calculates the distance from the subject to the leader in the video from each monitoring camera 21 and instantly compares this distance with a first threshold. If the distance from the subject to the leader exceeds the first threshold, the management device 20 starts timing. If the distance from the subject to the leader falls below the first threshold before the timeout duration is reached, the management device 20 stops timing and resets the timeout. Thereafter, the management device 20 performs the same process. If the timeout duration is reached without the distance from the subject to the leader falling below the first threshold, the management device 20 determines that the subject is in danger. Note that the management device 20 determines that the subject is safe until it determines that the subject is in danger.

[0060] The first threshold may be a constant or a variable that changes depending on some factor. When the first threshold is a variable and there are multiple subjects belonging to the target group, the management device 20 calculates the distance between each pair of subjects in the target group. The management device 20 then determines the first threshold based on the distance between the farthest pair of subjects. The longer the distance, the smaller the first threshold. Therefore, the longer the distance, the more likely the subject is to be determined to be at risk. When the first threshold is a variable and there are multiple subjects belonging to the target group, the management device 20 estimates the age of each subject in the target group based on the footage from each surveillance camera 21. The management device 20 determines the first threshold based on the largest age among the ages of these subjects. The lower the largest age, the smaller the first threshold. Therefore, the lower the largest age, the more likely the subject is to be determined to be at risk.

[0061] The timeout duration may be a constant or a variable that changes depending on some factor. When the timeout duration is a variable and there are multiple subjects in the target group, the management device 20 calculates the distance between each pair of subjects in the target group. The management device 20 then determines the timeout duration based on the distance between the farthest pair of subjects. The longer the distance, the shorter the timeout duration. Therefore, the longer the distance, the more likely the subjects are to be determined to be in danger. When the timeout duration is a variable and there are multiple subjects in the target group, the management device 20 estimates the age of each subject in the target group based on the footage from each surveillance camera 21. The management device 20 determines the timeout duration based on the largest age among the subjects. The lower the largest age, the shorter the timeout duration. Therefore, the lower the largest age, the more likely the subject is to be determined to be in danger. When the timeout duration is a variable and there are multiple chaperones in the target group, the management device 20 may determine the timeout duration based on the number of chaperones whose distance to the subject is equal to or less than a first threshold. The fewer such chaperones there are, the shorter the timeout duration will be, and therefore the more likely the subject will be judged as dangerous.

[0062] The management device 20 continues to execute the process of determining the safety of the target person while recognizing and tracking the target person in the images input from each monitoring camera 21. If the management device 20 determines that the target person is in danger, the management device 20 executes the following process.

[0063] <10-5. Detection of approach of unspecified persons> While the management device 20 recognizes and tracks a target person in the video input from each monitoring camera 21, the management device 20 executes image recognition processing on the video input from each monitoring camera 21, thereby instantly recognizing and tracking unspecified persons in the video.

[0064] The management device 20 determines whether or not an unspecified person being tracked is approaching the target being tracked in the video of each monitoring camera 21. Specifically, the management device 20 determines whether or not the distance from the target being tracked to the unspecified person being tracked in the video of each monitoring camera 21 is equal to or less than a second threshold. In other words, the management device 20 determines whether or not the unspecified person has entered a second detection area having a radius equal to the second threshold and centered on the target.

[0065] If the determination is positive, that is, if the distance from the target being tracked to the unspecified person being tracked in the video of each monitoring camera 21 is equal to or less than the second threshold, the management device 20 detects the approach of the unspecified person to the target. If the determination is negative, that is, if the distance from the target being tracked to the unspecified person being tracked in the video of each monitoring camera 21 exceeds the second threshold, the management device 20 does not detect the approach of the unspecified person to the target.

[0066] The second threshold may be a constant or a variable that changes depending on some factor. When the second threshold is a variable, the management device 20 instantly calculates the distance from the subject to the leader in the video of each monitoring camera 21. The management device 20 then sets a larger second threshold as the distance increases. Therefore, as the leader moves away from the subject, it becomes easier to detect the approach of an unspecified person to the subject.

[0067] The management device 20 continuously executes a process of determining whether or not an unspecified person is approaching while the management device 20 recognizes and tracks the target person in the images input from each monitoring camera 21. When the management device 20 detects that an unspecified person is approaching the target person, the management device 20 executes the following process.

[0068] <10-6. Recognition of the face of an unspecified person> The management device 20 performs image recognition processing on the video input from each monitoring camera 21. As a result, the management device 20 determines whether or not it can recognize the facial image of an unspecified person from the video. If the face of an unspecified person is captured in the video, the management device 20 recognizes the facial image of the unspecified person. If the face of an unspecified person is not captured in the video, or if the face of an unspecified person is captured unclearly, the management device 20 cannot recognize the facial image of the unspecified person.

[0069] <10-7. Video Analysis> If the management device 20 cannot recognize the facial image of an unspecified person, the management device 20 accesses the storage device 30 and seeks the video data 31. While seeking the video data 31, the management device 20 compares and analyzes the video images from each surveillance camera 21 with the video data 31, identifies the same person as the unspecified person from among the characters appearing in the video of the video data 31, and extracts an image of the same person. The image of the same person may be a video or a still image. There may be one image of the same person, or multiple images. The image of the same person may be a facial image of that person. The image of the same person may be an image including the face of that person.

[0070] Specifically, the management device 20 identifies the same person as the unspecified person from among the characters appearing in the video of the video data 31 as follows.

[0071] The management device 20 extracts features of unspecified persons from the video of each surveillance camera 21 using skeletal detection technology or the like. The features of unspecified persons are, for example, features of the entire or partial body of the unspecified person captured in the video, or features of the movements of the entire or partial body of the unspecified person. Features of the entire or partial body of the unspecified person refer to elements other than the face, such as the height, torso length, leg length, or hand length of the unspecified person. The entire or partial body of the unspecified person may include clothing worn by the unspecified person. The entire or partial body of the unspecified person may also include accessories worn by the unspecified person. Accessories include, for example, hats, face masks, helmets, scarves, necklaces, or wristbands. The entire or partial body of the unspecified person may also include luggage carried by the unspecified person. The movements of the entire or partial body of the unspecified person refer to walking, such as raising and lowering legs, waving hands, shaking the upper body, and shaking the head. Examples of movement features include stride length, walking speed, foot height, foot lift speed, foot lower speed, hand swing amplitude, hand swing speed, upper body sway amplitude, upper body sway speed, and head sway amplitude or head sway speed. The management device 20 extracts features of at least one character appearing in the video from the video data 31 using skeletal detection technology or the like. The character features are, for example, body features of the entire or a portion of the character appearing in the video, or features of the entire or a portion of the character's body movements. Furthermore, the management device 20 extracts images of the characters appearing in the video from the video data 31 for each character. For each character whose image has been successfully extracted, the management device 20 associates the extracted images with the features. The management device 20 then compares the features of unspecified individuals extracted from the video of each surveillance camera 21 with the features of each character extracted from the video data 31 to identify the unspecified individual from among the characters. For example, the management device 20 may calculate the deviation between the feature amount of the unspecified person and the feature amount of each character, and identify the character with the smallest deviation as the same person as the unspecified person. Here, "top" refers to first, second, third, or the like.The management device 20 may calculate the deviation between the feature amount of the unspecified person and the feature amount of each character, and identify a character having a feature amount whose deviation is equal to or greater than a predetermined value as the same person as the unspecified person. Note that the identified same person is not limited to one person, and there may be two or more. In this way, a person who is the same as the unspecified person is identified from among the characters appearing in the video of the video data 31, and an image of that same person is extracted and identified.

[0072] <10-7-1. Cancellation of Identification> When a character identified as the same as an unspecified person is a staff member of a facility, the character's identification as an unspecified person may be cancelled as follows.

[0073] The management device 20 compares the extracted image of a character identified as the same as an unspecified person with the facial image data 33, and determines whether the character is a facility staff member. Specifically, the management device 20 extracts feature amounts of the character and the staff member from the extracted image and the image data, respectively, and determines whether the character is a facility staff member based on these feature amounts. If the determination is positive, that is, if the character is a facility staff member, the management device 20 cancels the identification of the character identified as the same as an unspecified person. If the determination is negative, that is, if the character is not a facility staff member, the management device 20 maintains the identification of the character identified as the same as an unspecified person.

[0074] Note that even if a character identified as the same as an unspecified person is a facility staff member, that character may have a weak relationship with the target person. Therefore, the management device 20 may determine whether to cancel the identification of the character as an unspecified person based on the schedule data 34. For example, if a staff member is off duty or performing work unrelated to the target person, the management device 20 can recognize this from the schedule data 34, and therefore the management device 20 maintains the identification of the character identified as the same as an unspecified person. For example, if a staff member is on duty or performing work related to the target person, the management device 20 can recognize this from the schedule data 34, and therefore the management device 20 cancels the identification of the character identified as the same as an unspecified person.

[0075] <10-8. Display of Extracted Images of Characters Identical to Unspecified Persons> After identifying a character identical to an unspecified person, the management device 20 displays an extracted image associated with the character on the first display device 23. For example, the management device 20 may display an extracted image of the character identical to the unspecified person on the first display device 23 in parallel with instantaneous display of the video images input from each monitoring camera 21 on the first display device 23. The management device 20 may composite the extracted image of the character with the video images input from each monitoring camera 21 and instantaneously display the composite video images on the first display device 23. The management device 20 may composite the extracted image of the character near the unspecified person in the video images input from each monitoring camera 21 and instantaneously display the composite video images on the first display device 23. When there are multiple extracted images, the management device 20 may display these extracted images on the first display device 23 in order of clarity. If there are multiple extracted images, the management device 20 may sequentially display these extracted images on the first display device 23 in order of clarity. If there are multiple extracted images, the management device 20 may arrange these extracted images in ascending or descending order of the recording time of the extracted images and display them on the first display device 23. If there are multiple extracted images, the management device 20 may sequentially display these extracted images on the first display device 23 in ascending or descending order of the recording time of the extracted images. If there are multiple extracted images, the management device 20 may classify the extracted images by, for example, the pattern of the clothing of the characters and display the extracted images for each classification on the first display device 23. The management device 20 may display the extracted images on the first display device 23 together with information identifying the location of the surveillance camera 21 that acquired the video data 31 from which the extracted images were extracted. The information identifying the location of the surveillance camera 21 may be, for example, a map of the facility. An administrator or the like may operate the terminal device 24, and the terminal device 24 may communicate with the management device 20, causing the terminal device 24 to display extracted images of characters that are the same as unspecified persons.

[0076] It should be noted that if the identification of a character that has been identified as the same as an unspecified person is cancelled, the extracted image of that character will not be displayed.

[0077] By displaying an image of a character identical to the unspecified person on the first display device 23 or the terminal device 24, facility managers and guides can view the image and identify the unspecified person.

[0078] 10-9. Displaying Images of Suspicious Individuals Identical to Unspecified Individuals The management device 20 identifies the same person as the unspecified individual from among the suspicious individuals by comparing the video from each monitoring camera 21 with each image data 35 in the database. For example, the identification method is as follows. First, the management device 20 extracts feature amounts of the unspecified individual from the video from each monitoring camera 21 using skeletal structure detection technology or the like. The management device 20 also performs a feature extraction process on each image data 35 to extract feature amounts of the suspicious individual for each image data 35 in the database. The management device 20 compares the feature amounts of the unspecified individual extracted from the video from each monitoring camera 21 with the feature amounts of each suspicious individual extracted from the image data 35, respectively, to identify the same person as the unspecified individual from among the suspicious individuals. In this case, the management device 20 performs the feature extraction process and the comparison process on the image data 35 in descending order of the suspiciousness level associated with the image data 35.

[0079] After identifying the suspicious person who is the same as the unspecified person, the management device 20 displays an image on the first display device 23 according to the image data 35 and face image data 36 of the suspicious person who is the same as the unspecified person. The image may be displayed together with an extracted image of a character who is the same as the unspecified person.

[0080] <10-10. Deleting Registration> When a certain period of time has passed since the registration of the registration data 32, the management device 20 may delete the registration data 32 from the storage device 30. The certain period of time may be, for example, six months, one year, or two years.

[0081] When the subject and the chaperone leave the facility through the exit, the exit detection unit 25 detects the subject and the chaperone's exit and outputs a signal to that effect to the management device 20. When the management device 20 receives the signal, the registration data 32 may be deleted from the storage device 30.

[0082] The administrator may use the first input device 22 or the terminal device 24 to issue a command to delete the registered data 32 to the management device 20, which may cause the management device 20 to delete the registered data 32 from the storage device 30.

[0083] In this way, the registration data 32 is deleted from the viewpoint of protecting personal information.

[0084] 11. Technically Advantageous Effects Facial images of the subject and the chaperone are registered as registration data 32. The registration of the facial images of the subject and the chaperone contributes to enabling management device 20 to recognize and track the subject and the chaperone from among people appearing in video captured by surveillance camera 21. Consequently, the registration of facial images of the subject and the chaperone contributes to enabling management device 20 to determine the safety of the subject.

[0085] If the chaperone continues to leave a detection area with a radius of a first threshold value centered on the target for a certain period of time, the management device 20 recognizes and tracks an unspecified person in the video captured by each surveillance camera 21. The management device 20 determines whether or not an unspecified person is approaching the target in the video. If the management device 20 detects an unspecified person approaching the target and performs image recognition processing on the video, and is unable to recognize the face of the unspecified person, the management device 20 compares the video of the captured video data 31 with the video of the surveillance camera 21, analyzes them, and identifies the same person as the unspecified person from among the characters appearing in the video data 31. The management device 20 extracts an image of the identified same person from the video data 31. The image is displayed on the first display device 23. The facility manager, chaperone, or the like can view the image and identify the unspecified person.

[0086] <12.> One or more embodiments have been described and illustrated in detail above. The above-disclosed embodiments have been made for purposes of illustration and example only and are not intended to limit the scope of the present invention, which should be interpreted by the terms of the claims.

[0087] REFERENCE SIGNS LIST 1 Surveillance system 12 Registration camera 10 Registration device 11 Registration terminal device 20 Management device 21 Surveillance camera 22 First input device 23 First display device 30 Storage device 31 Video data 32 Registration data 33 Face image data 34 Schedule data 35 Image data 36 Face image data

Claims

1. A management device comprising: a first determination unit that determines whether or not an unspecified person is approaching at least one target person in a video captured by a surveillance camera installed within a facility; a second determination unit that, when the first determination unit detects the approach of the unspecified person to the target person, determines whether or not a facial image of the unspecified person can be recognized from the video; an extraction unit that, when the second determination unit determines that the facial image of the unspecified person cannot be recognized, compares the video with video data previously captured within the facility, identifies a person identical to the unspecified person from at least one character appearing in a video of the video data, and extracts an image of the same person from the video; and a display unit that displays the image extracted by the extraction unit.

2. The management device of claim 1, wherein the extraction unit extracts features of the unspecified person from the video, extracts features of the characters from the video data, identifies the same person from among the characters by comparing the features of the unspecified person with the features of the characters, and extracts an image of the same person from the video data.

3. The management device according to claim 1 or 2, wherein the images of the same person are images including the face of the same person.

4. The management device of claim 1 or 2, further comprising a third judgment unit that uses a facial image of the subject and a facial image of at least one leader to recognize and track the subject and the leader in the video, and judges whether it is safe or not based on the fact that the leader is leading the subject in the video, and when the third judgment unit judges it is not safe, the first judgment unit judges whether or not the subject is being approached by an unspecified person.

5. The management device of claim 4, wherein the third judgment unit judges whether the state in which the distance from the subject to the chaperone exceeds a threshold in the video continues for a timeout duration, and if the state in which the distance from the subject to the chaperone exceeds the threshold continues for the timeout duration, the third judgment unit judges that it is unsafe, and if the state in which the distance from the subject to the chaperone exceeds the threshold does not continue for the timeout duration, the third judgment unit judges that it is safe.

6. The management device according to claim 5, wherein the number of subjects is more than one, and the third determination unit sets the threshold to a smaller value as the distance between the subjects increases.

7. The management device according to claim 6, wherein the number of subjects is multiple, and the third judgment unit sets the threshold value smaller as the maximum age of the subjects becomes lower.

8. The management device according to claim 5, wherein the number of the subjects is more than one, and the third determination unit determines the timeout duration to be shorter as the distance between the subjects increases.

9. The management device according to claim 5, wherein the number of subjects is more than one, and the third determination unit sets the timeout duration to be shorter as the maximum age of the subjects is lower.

10. The management device according to claim 5, wherein the number of chaperones is multiple, and the third judgment unit sets the timeout duration to be shorter as the number of chaperones whose distance to the subject is less than or equal to the threshold is fewer.

11. The management device of claim 5, wherein the number of chaperones is multiple, and the third judgment unit judges whether the state in which the distance to the subject of any of the chaperones exceeds the threshold continues for the timeout duration.

12. The management device of claim 5, wherein the number of chaperones is multiple, and the third judgment unit judges whether the state in which the distance to the subject for any one of the chaperones exceeds the threshold continues for the timeout duration.

13. The management device according to claim 12, wherein the threshold or the timeout duration, or both, are different for each leader.

14. The management device according to claim 4, wherein the facial image of the subject and the facial image of the leader are registered by being photographed by a registration camera installed at an entrance of the facility.

15. The management device of claim 14, wherein the registration terminal device presents a registration screen to the chaperone requesting consent, and when the chaperone inputs consent into the registration terminal device, the registration terminal device causes the registration camera to take a facial image of the subject and a facial image of the chaperone.

16. The management device of claim 4, wherein the number of chaperones is multiple, the facial image of the subject and the facial image of any first of the chaperones are registered by being photographed by a registered camera installed at the entrance of the facility, and the facial image of any other second of the chaperones is a facial image that has previously been associated with the same facial image as the facial image of the subject and registered.

17. The management device described in claim 16, wherein the third judgment unit judges whether the state in which the distance from the subject to the chaperone exceeds a threshold in the video continues for a timeout duration, the third judgment unit judges that it is unsafe if the state in which the distance from the subject to the chaperone exceeds the threshold continues for the timeout duration, and the third judgment unit judges that it is safe if the state in which the distance from the subject to the chaperone exceeds the threshold does not continue for the timeout duration, and the threshold adopted for the first chaperone is greater than the threshold adopted for the second chaperone.

18. The management device described in claim 16, wherein the third judgment unit judges whether the state in which the distance from the subject to the chaperone exceeds a threshold in the video continues for the timeout duration, the third judgment unit judges that it is unsafe if the state in which the distance from the subject to the chaperone exceeds the threshold continues for the timeout duration, and the third judgment unit judges that it is safe if the state in which the distance from the subject to the chaperone exceeds the threshold does not continue for the timeout duration, and the timeout duration adopted by the first chaperone is longer than the timeout duration adopted by the second chaperone.

19. The management device of claim 1 or 2, further comprising a fourth judgment unit that judges whether the person identified as the same as the unspecified person is a staff member of the facility by comparing the image extracted by the extraction unit with image data of a staff member of the facility, and when the fourth judgment unit judges that the person identified as the same as the unspecified person is a staff member of the facility, the display unit does not display the image extracted by the extraction unit.

20. A management device as described in claim 1 or 2, comprising: a generation unit that generates a database of images of suspicious persons captured in the video by analyzing the video; and a second display unit that identifies a person identical to the unspecified person from among the suspicious persons by comparing the images in the database with the video, and displays an image of the suspicious person identical to the unspecified person.

Citation Information

Patent Citations

  • Facility user management system and user management method

    JP2007060528A

  • Monitor, method, and program

    JP2007300363A

  • System and method for finding lost child

    JP2013069128A

  • System for store, recording medium processing apparatus, and imaging information processing apparatus

    JP2013117993A

  • Image processing system, control method thereof, and program

    JP2019061665A