Facility management device, facility management system, and facility management method

Abstract

To provide a facility management system capable of allowing a user to immediately check the status of an event by detecting an event that has occurred in a monitored area in real-time, and further allowing the user to easily and properly grasp the situation of the event and by checking the previous situation of the moving body related to the event.SOLUTION: The facility management system includes: a plurality of cameras 1 for picking up images in a monitoring area; and a facility management server 2 that detects an event that has occurred in the monitoring area based on images picked up by the cameras. Facility management A server processor 13 of the facility management server is configured to perform the steps of: acquiring images picked up at each time; detecting moving bodies containing a person as a monitoring target from the picked-up images (moving body detection processing); detecting partial changes in the moving bodies by comparing the moving body detection results; detecting events that have occurred in the monitoring area based on the situation of the partial changes of the moving body (event detection processing); and extracting and outputting a first image included in the moving body from the picked-up images when the event has been detected and extracting and outputting a second image including the moving body from picked-up images before the event has been detected (output control processing).SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to a facility management device, a facility management system, and a facility management method for detecting a predetermined event that has occurred in a monitoring area based on captured images from a plurality of cameras that capture the monitoring area within a facility.

Background Art

[0002] A plurality of cameras for capturing a monitoring area within a facility are installed, and a system for detecting an abnormal event that has occurred in the monitoring area, such as detecting an unattended package as a suspicious object, based on the captured images from the cameras is widely used.

[0003] As such a system for detecting an abnormal event, conventionally, in a toilet, a camera for capturing a user and a sensor for detecting an object within a toilet booth are provided. When the detection result of the sensor is different before and after the user enters and exits, the toilet booth is determined to be abnormal, and a technique has been known that allows a monitor to confirm the user in the captured image when the user of the toilet booth enters or exits (see Patent Document 1).

[0004] In addition, in order to detect an abnormal event, particularly an unattended package as a suspicious object, it is necessary to grasp the ownership relationship between a person and a package. Therefore, a technique is known in which a person is detected and tracked, and a package is detected and tracked, and the degree of association between the person and the package is determined based on the distance between the person and the package, and the ownership relationship between the person and the package is comprehensively determined (see Patent Document 2).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, the technology described in Patent Document 1 has a problem in that if the sensor fails to detect an object inside the toilet booth, abnormal events, such as abandoned luggage, will not be detected as suspicious objects, and abnormal events may be missed. Furthermore, the technology described in Patent Document 2 is a technology that searches for luggage or people linked to a person or luggage, and is unsuitable for applications that detect predetermined events occurring in a monitoring area in real time, allowing users to immediately check various events that occurred in the monitoring area. Moreover, if users can also check the previous situation of moving objects related to the detected event, they can understand the detected event more easily and appropriately.

[0007] Therefore, the main objective of the present invention is to provide a facility management device, a facility management system, and a facility management method that can detect predetermined events occurring in a monitoring area in real time, allowing users to immediately confirm the status of events occurring in the monitoring area, and furthermore, allowing users to confirm the previous status of moving objects related to the events, thereby enabling them to easily and appropriately grasp the status of events. [Means for solving the problem]

[0008] The facility management device of the present invention is a facility management device in which a processor performs a process to detect a predetermined event occurring in a monitoring area based on images captured by a plurality of cameras that photograph the monitoring area, the processor acquires the captured images at each time, detects moving objects including people as the target of monitoring from the captured images, compares the detection results of the moving objects to detect partial changes in the moving objects, detects an event that occurred in the monitoring area based on the state of partial changes in the moving objects, and from the captured images at the time the event was detected The first image including the extracted moving object and From the aforementioned captured image taken before the aforementioned event was detected The system outputs an event notification screen that includes a second image containing the extracted moving object, and further outputs a timeline confirmation screen in which, upon user operation to instruct the user to confirm the event on the event notification screen, the captured images containing the moving object taken at the time of the event and before and after the event are arranged for each camera in chronological order of capture time. This will form the structure.

[0009] Furthermore, the facility management system of the present invention is a facility management system comprising: a plurality of cameras that photograph a monitoring area; and a facility management device that performs processing to detect predetermined events occurring in the monitoring area based on images captured by the plurality of cameras, wherein the facility management device acquires the captured images at each time, detects moving objects including people as subjects of monitoring from the captured images, compares the detection results of the moving objects to detect partial changes in the moving objects, detects events occurring in the monitoring area based on the state of partial changes in the moving objects, and from the captured images at the time the event was detected The first image including the extracted moving object and From the aforementioned captured image taken before the aforementioned event was detected The system outputs an event notification screen that includes a second image containing the extracted moving object, and further outputs a timeline confirmation screen in which, upon user operation to instruct the user to confirm the event on the event notification screen, the captured images containing the moving object taken at the time of the event and before and after the event are arranged for each camera in chronological order of capture time. This will form the structure.

[0010] Furthermore, the facility management method of the present invention is a facility management method in which an information processing device performs a process to detect a predetermined event occurring in a monitoring area based on images captured by a plurality of cameras that photograph the monitoring area, the method acquires the captured images at each time, detects moving objects including people as the target of monitoring from the captured images, compares the detection results of the moving objects to detect partial changes in the moving objects, detects an event that occurred in the monitoring area based on the state of partial changes in the moving objects, and from the captured images at the time the event was detected The first image including the extracted moving object and From the aforementioned captured image taken before the aforementioned event was detected The system outputs an event notification screen that includes a second image containing the extracted moving object, and further outputs a timeline confirmation screen in which, upon user operation to instruct the user to confirm the event on the event notification screen, the captured images containing the moving object taken at the time of the event and before and after the event are arranged for each camera in chronological order of capture time. This will form the structure. [Effects of the Invention]

[0011] According to the present invention, the monitoring area When an event is detected, the event notification screen displays a timeline screen in which, based on the user's actions to instruct them to confirm the event, images including moving objects taken at the time of the event and before and after the event are arranged by camera in chronological order of capture time. Therefore, users can immediately check the status of events that occur in the monitoring area, and furthermore, related to those events. Status of moving objects in each camera Being able to check this information as well allows for an easy and appropriate understanding of the situation. [Brief explanation of the drawing]

[0012] [Figure 1] Overall configuration diagram of the facility management system according to this embodiment [Figure 2] This diagram illustrates the first type of event that is subject to event detection performed by the facility management server. [Figure 3] Explanatory diagram showing the second and third types of events to be detected by the facility management server [Figure 4] Explanatory diagram showing the outline of event detection performed by the facility management server [Figure 5] Explanatory diagram showing the outline of work measurement performed by the facility management server [Figure 6] Explanatory diagram showing the registration content of the detection result database managed by the facility management server [Figure 7] Block diagram showing the schematic configuration of the facility management server [Figure 8] Explanatory diagram showing the overall display screen displayed on the monitoring terminal [Figure 9] Explanatory diagram showing the target camera setting screen displayed on the monitoring terminal [Figure 10] Explanatory diagram showing the event occurrence notification screen related to the first type of event displayed on the monitoring terminal [Figure 11] Explanatory diagram showing the timeline confirmation screen related to the first type of event displayed on the monitoring terminal [Figure 12] Explanatory diagram showing the timeline confirmation screen related to the first type of event displayed on the monitoring terminal [Figure 13] Explanatory diagram showing the change location confirmation screen related to the first type of event displayed on the monitoring terminal [Figure 14] Explanatory diagram showing the map confirmation screen related to the first type of event displayed on the monitoring terminal [Figure 15] Explanatory diagram showing the measurement target confirmation screen related to the work measurement displayed on the monitoring terminal [Figure 16] Explanatory diagram showing the event occurrence notification screen related to the second type of event displayed on the monitoring terminal [Figure 17] Explanatory diagram showing the timeline confirmation screen related to the second type of event displayed on the monitoring terminal [Figure 18] Explanatory diagram showing the timeline confirmation screen related to the second type of event displayed on the monitoring terminal [Figure 19] Explanatory diagram showing the change location confirmation screen related to the second type of event displayed on the monitoring terminal [Figure 20]An explanatory diagram showing the event occurrence notification screen for the third type of event displayed on the monitoring terminal. [Figure 21] An explanatory diagram showing the timeline confirmation screen for the third type of event displayed on the monitoring terminal. [Figure 22] An explanatory diagram showing the timeline confirmation screen for the third type of event displayed on the monitoring terminal. [Figure 23] An explanatory diagram showing the screen for confirming changes related to the third type of event displayed on the monitoring terminal. [Figure 24] This diagram illustrates the fourth type of event that is subject to event detection performed by the facility management server. [Figure 25] An explanatory diagram showing the registered contents of the detection result database related to the fourth type of event displayed on the monitoring terminal. [Figure 26] An explanatory diagram showing the event occurrence notification screen for the fourth type of event displayed on the monitoring terminal. [Figure 27] An explanatory diagram showing the timeline confirmation screen for the fourth type of event displayed on the monitoring terminal. [Figure 28] An explanatory diagram showing the timeline confirmation screen for the fourth type of event displayed on the monitoring terminal. [Figure 29] An explanatory diagram showing the map confirmation screen related to the fourth type of event displayed on the monitoring terminal. [Modes for carrying out the invention]

[0013] The first invention made to solve the aforementioned problems is a facility management device in which a processor performs a process to detect a predetermined event occurring in a monitoring area based on images captured by a plurality of cameras that photograph the monitoring area, wherein the processor acquires the captured images at each time, detects moving objects including people as the target of monitoring from the captured images, compares the detection results of the moving objects to detect partial changes in the moving objects, detects an event occurring in the monitoring area based on the state of partial changes in the moving objects, and from the captured images at the time the event was detected The first image including the extracted moving object and From the aforementioned captured image taken before the aforementioned event was detected The system outputs an event notification screen that includes a second image containing the extracted moving object, and further outputs a timeline confirmation screen in which, upon user operation to instruct the user to confirm the event on the event notification screen, the captured images containing the moving object taken at the time of the event and before and after the event are arranged for each camera in chronological order of capture time. This will be the structure.

[0014] According to this, the surveillance area When an event is detected, the event notification screen displays a timeline screen in which, based on the user's actions to instruct them to confirm the event, images including moving objects taken at the time of the event and before and after the event are arranged by camera in chronological order of capture time. Therefore, users can immediately check the status of events that occur in the monitoring area, and furthermore, related to those events. Status of moving objects in each camera Being able to check this information as well allows for an easy and appropriate understanding of the situation.

[0015] Furthermore, the second invention is configured such that the processor detects the moving body, including a person and their belongings, and detects a change in the moving body that involves either an increase or decrease in the belongings.

[0016] According to this, it is possible to detect changes such as an increase or decrease in the amount of belongings carried.

[0017] Furthermore, the third invention is configured such that the processor detects the moving body, including a person and their belongings, and detects a change in the moving body that involves either the appearance or disappearance of the belongings.

[0018] According to this, it is possible to detect changes such as the appearance or disappearance of carried items.

[0019] Furthermore, the fourth invention is configured such that the processor detects a change in appearance due to the removal or attachment of clothing worn by a person as a partial change in the moving body.

[0020] According to this, it is possible to detect changes in a person's appearance when they put on or take off clothing (clothes, shoes, hats, etc.).

[0021] Furthermore, the fifth invention is configured such that the processor extracts and outputs the second image from images taken at multiple time points prior to the detection of the event.

[0022] According to this, users can visually examine images taken at multiple times prior to the event being detected, allowing them to confirm the details of the moving object.

[0025] Also, The sixth The invention further includes a configuration in which the processor superimposes an image on the captured image that shows the region in which a partial change of the moving object appears in the captured image.

[0026] According to this, users can easily identify areas in the captured image where partial changes in motion have occurred. In this case, for example, a frame image surrounding the area where partial changes in motion have occurred may be superimposed on the captured image.

[0027] Also, 7th The invention relates to a facility management system comprising: a plurality of cameras that photograph a monitoring area; and a facility management device that performs processing to detect predetermined events occurring in the monitoring area based on images captured by the plurality of cameras, wherein the facility management device acquires the captured images at each time, detects moving objects including people as subjects of monitoring from the captured images, compares the detection results of the moving objects to detect partial changes in the moving objects, detects events occurring in the monitoring area based on the state of partial changes in the moving objects, and from the captured images at the time the event was detected The first image including the extracted moving object and From the aforementioned captured image taken before the aforementioned event was detected The system outputs an event notification screen that includes a second image containing the extracted moving object, and further outputs a timeline confirmation screen in which, upon user operation to instruct the user to confirm the event on the event notification screen, the captured images containing the moving object taken at the time of the event and before and after the event are arranged for each camera in chronological order of capture time. This will be the structure.

[0028] According to this, similar to the first invention, The output of the timeline confirmation screen for the monitoring area indicates that Users can immediately check the status of events that occur in the monitoring area, and also check related information. Status of moving objects in each camera Being able to check this information as well allows for an easy and appropriate understanding of the situation.

[0029] Also, 8th The invention relates to a facility management method in which an information processing device performs a process to detect predetermined events occurring in a monitoring area based on images captured by multiple cameras that photograph the monitoring area, the method involves acquiring the captured images at each time, detecting moving objects including people as the target of monitoring from the captured images, comparing the detection results of the moving objects to detect partial changes in the moving objects, detecting events occurring in the monitoring area based on the state of partial changes in the moving objects, and using the captured images at the time the event was detected. The first image including the extracted moving object and From the aforementioned captured image taken before the aforementioned event was detected The system outputs an event notification screen that includes a second image containing the extracted moving object, and further outputs a timeline confirmation screen in which, upon user operation to instruct the user to confirm the event on the event notification screen, the captured images containing the moving object taken at the time of the event and before and after the event are arranged for each camera in chronological order of capture time. This will be the structure.

[0030] According to this, similar to the first invention, The output of the timeline confirmation screen for the monitoring area indicates that Users can immediately check the status of events that occur in the monitoring area, and also check related information. Status of moving objects in each camera Being able to check this information as well allows for an easy and appropriate understanding of the situation.

[0031] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0032] Figure 1 is an overall configuration diagram of the facility management system according to this embodiment.

[0033] This system performs event detection, which detects specified events such as abnormal occurrences, and work measurement, which collects information to understand the work content of individuals (workers), targeting monitoring areas set up within the facility. The system consists of a camera 1, a facility management server 2 (facility management device), and a monitoring terminal 3 (terminal device). Camera 1 and the facility management server 2 are connected via a network. Monitoring terminal 3 is connected to the facility management server 2.

[0034] Camera 1 is installed inside the facility. Camera 1 photographs the designated monitoring area within the facility. Camera 1 transmits the captured images of the monitoring area to the facility management server 2.

[0035] The facility management server 2 performs image analysis on the images received from camera 1, thereby executing processes related to event detection and work measurement.

[0036] The monitoring terminal 3 (management terminal) is a tablet device or a PC. Based on information transmitted from the facility management server 2, the monitoring terminal 3 displays monitoring screens and other information for the user (monitor, administrator, etc.). When a predetermined event is detected by the facility management server 2, the monitoring terminal 3 notifies the user of the occurrence of the predetermined event. This allows for appropriate action to be taken in response to the predetermined event (on-site response, situation observation, etc.). The monitoring terminal 3 also presents the work measurement results to the user. The monitoring terminal 3 may be connected to the facility management server 2 via a wireless network.

[0037] In the case of event detection, the facilities where the monitoring area is set are, for example, places where a large number of people gather, such as train stations, airports, and commercial facilities. Alternatively, specific event detection may be performed in schools, kindergartens, daycare centers, and nursing homes. On the other hand, in the case of work measurement, the facilities where the monitoring area is set are, for example, factories and warehouses.

[0038] If only one facility is targeted, the facility management server 2 and monitoring terminal 3 may be installed within the facility together with the camera 1, and the camera 1, facility management server 2, and monitoring terminal 3 may be connected via a closed network. On the other hand, if multiple facilities are targeted, a camera 1 may be installed in each of the multiple facilities. In addition, the facility management server 2 may be configured as a cloud server, and the camera 1 and facility management server 2 may be connected via the internet. Furthermore, in addition to being connected to the facility management server 2, the monitoring terminal 3 may also be connected to the facility management server 2 via the internet.

[0039] Next, we will explain the first, second, and third types of events that are subject to event detection performed by the facility management server 2. Figure 2 is an explanatory diagram showing the first type of event. Figure 3 is an explanatory diagram showing the second and third types of events.

[0040] In this embodiment, changes involving an increase or decrease in carried items are detected as a first type of event. A first type of event is, for example, when a person leaves their belongings (such as a bag) behind in a restroom, as shown in Figure 2(A). In this case, the belongings are detected as carried items, and when the person leaves the restroom, the belongings are gone, indicating a change in the carried items, such as their disappearance or decrease, and therefore it is detected as a first type of event.

[0041] Conversely, there are also cases where changes occur in which personal belongings appear or increase. For example, when a person takes toilet paper or other supplies from a toilet, this falls under the first type of event. In this case, the supplies taken from the toilet are detected as personal belongings, and since a change has occurred in which personal belongings appear or increase, it is detected as the first type of event.

[0042] If a person leaves their belongings behind in this manner, it may be determined that a suspicious object, such as a dangerous item, has been placed in the toilet, and security personnel may be notified accordingly. Similarly, if a person takes any toilet equipment, security personnel may also be notified accordingly.

[0043] Furthermore, when a person (cleaner) brings toilet supplies into the toilet, the supplies are detected as personal belongings, and when the person leaves the toilet, the supplies are no longer present, so it is detected as an event of the first type. In this case, it would be advisable to exclude the person from notification by using methods such as identifying the cleaner by their work uniform. Additionally, work tracking may be implemented to measure the number of times the toilet is cleaned based on the entry and exit of people (cleaners).

[0044] Furthermore, the items carried are not limited to those held in a person's hands. For example, carried items may include strollers, carry-on bags, or carts that a person pushes or pulls.

[0045] The example shown in Figure 2(B) is for work measurement. In this example, goods are loaded onto a cart and transported to a truck or warehouse, or to a truck or warehouse (freezer). In this case, the cart is detected as an item carried because a person is moving the cart. Furthermore, if there are goods loaded on the cart, those goods are also detected as an item carried along with the cart. When goods are loaded onto a truck or warehouse, the goods on the cart are removed, resulting in a decrease (disappearance) in the number of items carried. Conversely, when goods are taken out of a truck or warehouse, goods are loaded onto the cart, resulting in an increase (appearance) in the number of items carried. This allows for the determination of the person's work content.

[0046] In this case, by counting the number of items loaded on the cart, work efficiency can be measured and overloading can be detected. Furthermore, by using person detection processing to determine whether the person transporting the goods is one of the pre-registered workers, the work content of each worker can be evaluated. In addition, by using person detection processing to determine whether the person transporting the goods is a pre-registered worker or not, unauthorized removal of goods can be detected.

[0047] Furthermore, when loading goods onto trolleys while unloading them from trucks or warehouses, the loading process takes time. Therefore, if the entrance or exit of the truck or warehouse becomes part of the monitoring area (camera 1's shooting area), the time spent within the monitoring area will be longer. This length of time spent within the area may also be taken into consideration when determining the nature of the work.

[0048] Furthermore, in this embodiment, changes involving the movement of carried items are detected as a second type of event. The second type of event is, for example, when a person puts on or takes off clothing (clothes, shoes, hat, etc.), as shown in Figure 3(A). In this case, the person's appearance changes between the state with and without the clothing, and at the same time, the clothing changes into a carried item when the person holds the removed clothing in their hand, and the carried item disappears when the person puts the clothing, now a carried item, back on their body. Therefore, this is detected as a second type of event.

[0049] In this example, the person changes from wearing a coat to taking it off and holding the removed coat in their hand. Similarly, the person changes from holding the coat in their hand to wearing the coat again. As the coat is removed and put on, the appearance of various parts of the person (torso, legs, arms) changes, and the coat held in the person's hand becomes an item they are carrying (luggage), resulting in an increase or decrease in the amount of luggage they are carrying.

[0050] In this case, during motion tracking, the person is identified (determined to be the same person) based on the parts of the person's appearance (clothing) that do not change. In this example, the person is identified based on attributes (feature information) of parts not hidden by the coat, such as shoes, hat, and both legs (in the case of a half coat).

[0051] Furthermore, in this embodiment, an overall change in a person's appearance is detected as a third type of event. A third type of event is, for example, as shown in Figure 3(B), when a person enters a fitting room in a store, changes into the merchandise, and then comes out. In this case, the person's clothing changes significantly before and after entering the fitting room, and is therefore detected as a third type of event.

[0052] In this case, since changing clothes takes time, if the area in front of the fitting room becomes the surveillance area (camera 1's shooting area), the time spent in the surveillance area will be longer. Therefore, the length of this time spent may also be taken into consideration when determining whether or not the event falls under the third type. Note that since motion tracking is interrupted while the person is inside the fitting room, the person will be re-identified once they come out of the fitting room.

[0053] Furthermore, in this case, if a person leaves the store while still trying on the merchandise, it may be considered theft of the goods, and the security guard may be notified accordingly.

[0054] While this example concerns a fitting room, a general change in a person's appearance (the third type of event) can also occur in front of a toilet. Similar to the fitting room, if a person changes clothes inside a toilet, their overall appearance (clothing) changes, and this is detected as the third type of event.

[0055] Incidentally, as shown in the example in Figure 2(A), changes involving an increase or decrease in the amount of belongings carried (the first type of event) can also occur on public transportation such as trains and buses. For example, if a person boards car number 2 on platform 1 at station A, and then alights from car number 2 on platform 2 at station B, and the person does not have the luggage they were carrying when boarding when they alight, it can be assumed that the person left their luggage behind on the train.

[0056] The facility management server 2 performs event detection to detect predetermined events, but the events targeted by this event detection are not limited to abnormal events such as unattended luggage. For example, event detection may also be performed for purposes such as searching for lost or found items, or for understanding the usage status (changing clothes) of restrooms.

[0057] Next, we will explain the event detection performed by the facility management server 2. Figure 4 is an explanatory diagram showing an overview of event detection.

[0058] Facility management server 2 performs event detection to detect predetermined events such as abnormal events that occur within the facility, specifically, changes involving an increase or decrease in the amount of personal belongings carried (Type 1 event), changes involving the movement of personal belongings (Type 2 event), overall changes in a person's appearance (Type 3 event), and a state in which a person's appearance differs from that of other people (Type 4 event).

[0059] In facility management server 2, for event detection, motion detection is first performed by analyzing the images captured by camera 1 to detect moving objects that appear within the monitoring area. Motion detection includes people and their belongings. In event detection, based on the motion detection results, partial changes in the moving object specific to each type of event to be detected are detected, and based on the state of these partial changes in the moving object, it is determined which type of event has occurred.

[0060] Multiple cameras (Camera 1) are installed within the facility. Each camera (Camera 1) captures a designated monitoring area within the facility. In this example, the areas in front of the toilets and the corridors within the facility are the monitoring areas. Camera 1 (#1) captures the area in front of the toilets, while cameras (#2, #3, #4, and #5) capture the corridors. Note that "in front of the toilets" refers to the entrance to the toilet area, but it may also refer to the area in front of the doors of the multiple toilet stalls installed within the toilet area.

[0061] In this embodiment, the camera 1 that captures the detected event is referred to as the detection camera. Additionally, the cameras 1 that capture moving objects related to the detected event before or after the event occurs are referred to as peripheral cameras. In this example, camera #1 is the detection camera, and cameras #2, #3, #4, and #5 are peripheral cameras.

[0062] Furthermore, the facility management server 2 displays images captured by multiple cameras 1 (detection camera and surrounding camera), as well as motion images extracted from those images, on the monitoring terminal 3. Users can visually inspect the images captured by multiple cameras 1 and the motion images on the monitoring terminal 3 to confirm in detail the detected events and the status of the motion related to those events.

[0063] Next, we will explain the work measurement performed on the facility management server 2. Figure 5 is an explanatory diagram showing an overview of the work measurement.

[0064] Facility management server 2 performs work measurement to collect information to understand the work being done by individuals (workers). In work measurement, motion detection is first performed by analyzing the images captured by camera 1 to detect moving objects, including individuals and their belongings. In work measurement, based on the motion detection results, partial changes in the moving objects specific to the work being measured are detected, and the work being done by the person is identified based on the state of these partial changes in the moving objects.

[0065] This example involves a person loading goods onto a cart and transporting them to or from a truck or warehouse. In this case, the person and the cart itself remain unchanged, but the goods loaded onto the cart change, and these goods become the subject of work measurement. In other words, in work measurement, changes in the goods are detected as partial changes in a moving object, and information representing the person's workload is collected based on these changes in the goods. Specifically, the number of items loaded onto the cart (load capacity) and the utilization rate (loading rate), which is the ratio of the actual load capacity to the cart's maximum load capacity, are collected.

[0066] Next, we will explain the detection result database managed by the facility management server 2. Figure 6 is an explanatory diagram showing the contents registered in the detection result database.

[0067] Facility management server 2 registers and manages the detection results of motion detection processing in the detection results database.

[0068] The detection results database registers the serial number, detection date and time (year, month, day, hour, minute, second, millisecond), camera information, overall image, enlarged image, comparison camera information, and the number of changed attributes.

[0069] Here, camera information includes the IP address, information about the installation location of camera 1, the name of camera 1, and information about whether it is a detection camera or a surrounding camera. The overall image is the image captured by camera 1. The enlarged image is a motion image (an image extracted from the captured image showing the region of motion). The comparison camera information is information about camera 1 that is used as a comparison when calculating the number of changed attributes. The number of changed attributes is the total number of parts where attributes have changed in motion, including people and their belongings.

[0070] The detection results database also registers information about the attributes (characteristic information) of the detected person and their belongings.

[0071] Next, we will describe the general configuration of facility management server 2. Figure 7 is a block diagram showing the general configuration of facility management server 2.

[0072] The facility management server 2 comprises a communication unit 11, a storage unit 12, and a processor 13.

[0073] The communication unit 11 communicates with the camera 1 and the monitoring terminal 3.

[0074] The memory unit 12 stores programs executed by the processor 13. The memory unit 12 also stores motion detection results, including captured images received from the camera 1, motion images acquired by the processor 13 (images in which the region of the moving object is extracted from the captured image), and attributes (feature information) of the moving object.

[0075] The processor 13 performs various processes by executing programs stored in the memory unit 12. In this embodiment, the processor 13 performs motion detection processing, event detection processing, change attribute count acquisition processing, work measurement processing, and output control processing.

[0076] In the camera image acquisition process, the processor 13 acquires the captured image received from the camera 1 by the communication unit 11.

[0077] In the motion detection process, the processor 13 detects moving objects appearing in the image captured by the camera 1. Moving objects include people and their belongings. The motion detection process includes person detection, belongings detection, and motion tracking. The detection results of the motion detection process are registered in the detection result database (see Figure 6).

[0078] In the person detection process, the processor 13 detects a person from the image captured by the camera 1 and obtains the person's attributes (feature information). At this time, information about the characteristics (e.g., color, shape, etc.) of each part of the body (e.g., head, torso, legs, arms, hat, shoes, etc.) is obtained.

[0079] In the object detection process, the processor 13 detects objects from the image captured by the camera 1 and obtains the attributes (feature information) of those objects. At this time, objects that move accompanying a person detected in the person detection process are detected as objects. Specifically, bags and coats held in a person's hands are detected as objects. Strollers, carry-on bags, and carts that a person pushes or pulls are also detected as objects.

[0080] In the motion tracking process, the processor 13 tracks moving objects, including people and their belongings. Specifically, it identifies people and their belongings (determines if they are the same person) based on the similarity of each part of the person and their belongings detected from the images captured by the camera 1 at each time point. For person identification, it determines if they are the same person based on the attributes of each part: head, torso, arms, legs, hat, and shoes. If the person is temporarily not detected and the tracking of the person is interrupted, the person is re-identified. When identifying a person through re-identification, the determination of whether detected belongings (luggage, trolley, stroller, etc.) belong to the same person may also be made.

[0081] Furthermore, for motion detection processing (such as person detection and object detection), an image recognition engine (machine learning model) built using machine learning such as deep learning may be used.

[0082] In the event detection process, the processor 13 detects a predetermined event based on the detection results of the motion detection process. Specifically, based on the detection results of the motion detection process, partial changes in moving objects (people and objects) specific to each type of event to be detected are detected, and based on the state of the partial changes in the moving objects, it is determined which type of event has occurred. In this embodiment, changes involving an increase or decrease in objects (first type of event), changes involving the movement of objects (second type of event), and overall changes in the appearance of a person (third type of event) are detected. Furthermore, in this embodiment, a state in which a person's appearance differs from that of other people (fourth type of event) is detected.

[0083] In the process of acquiring the number of changed attributes, the processor 13 counts the number of changed attributes, that is, the total number of parts of the attributes (feature information) that have changed in the moving object, including the person and their belongings, based on the detection results of the motion detection process.

[0084] In the work measurement process, the processor 13 collects information to understand the work being done by a person (worker) based on the detection results of the motion detection process. At this time, based on the detection results of the motion detection process, partial changes in the moving body specific to the work being measured are detected, and the work being done by the person is identified based on the state of these partial changes in the moving body. For example, in the work of a person loading and carrying goods onto a cart, information such as the number of goods loaded onto the cart (load capacity) and the utilization rate (loading rate), which is the ratio of the actual load capacity to the maximum load capacity of the cart, is collected.

[0085] In the output control process, the processor 13 generates various screens (see Figure 8, etc.) and displays them on the monitoring terminal 3. In addition, the processor 13 outputs a notification to the department in charge of on-site response, requesting on-site response.

[0086] Next, we will explain the overall display screen 101 (monitoring screen) that is displayed on the monitoring terminal 3. Figure 8 is an explanatory diagram showing the overall display screen 101.

[0087] On monitoring terminal 3, the overall display screen 101 is displayed during normal operation.

[0088] The overall display screen 101 includes a camera image display unit 102 and a target area selection unit 103. The target area selection unit 103 allows the user to select a target area. In this example, the target facility is a train station, and the target area can be selected from the area around the toilets, the concourse, the platform, or the ticket gate. The camera image display unit 102 displays images 21 captured by camera 1 installed in the selected target area. The images 21 displayed here are live images acquired in real time from camera 1.

[0089] On the overall display screen 101, a selection menu appears when the user right-clicks on the display area of ​​the target area (toilet, concourse, platform, and ticket gate) shown in the target area selection section 103. Selecting the settings screen from this menu transitions to the target camera settings screen 111 (see Figure 9).

[0090] Furthermore, on the overall display screen 101, when the user performs an operation to select camera 1 on the camera image display unit 102 (for example, by operating the title field of camera 1), a detailed display screen (not shown) is displayed. On the detailed display screen, only the captured image 21 (live image) of the selected camera 1 is displayed in an enlarged view.

[0091] When the facility management server 2 detects a predetermined event, namely a change involving an increase or decrease in carried items (first type of event), a change involving the movement of carried items (second type of event), or an overall change in a person's appearance (third type of event), an event occurrence notification screen 121 (see Figure 10) is displayed on the overall display screen 101.

[0092] Next, we will explain the target camera settings screen 111 displayed on the monitoring terminal 3. Figure 9 is an explanatory diagram showing the target camera settings screen 111.

[0093] The target camera settings screen 111 is provided with a camera image display unit 112. The camera image display unit 112 displays all the captured images 21 from each camera 1 installed in the target area in a row. The captured images 21 displayed here are live images acquired in real time from camera 1.

[0094] On the target camera settings screen 111, the user can specify whether or not each camera 1 should be included in event detection, that is, which camera 1's captured image 21 should be displayed on the overall display screen 101 (see Figure 8). Specifically, the user right-clicks on the name display field for each camera 1 displayed on the camera image display unit 112, which displays a selection menu where the user can choose either to include or exclude a camera from detection. This sets the camera 1 to be included in event detection, i.e., the camera 1 whose captured image 21 should be displayed on the overall display screen 101 (see Figure 8).

[0095] Additionally, the target camera settings screen 111 is provided with a "back" button 115. When the user operates the "back" button 115, they return to the overall display screen 101 (see Figure 8).

[0096] Next, we will explain the screen displayed on the monitoring terminal 3 when a change involving an increase or decrease in the amount of items carried (the first type of event) is detected.

[0097] First, we will explain the event occurrence notification screen 121 related to the first type of event displayed on the monitoring terminal 3. Figure 10 is an explanatory diagram showing the event occurrence notification screen 121.

[0098] When the monitoring terminal 3 detects a change involving an increase or decrease in the number of items carried (the first type of event) in the facility management server 2, the event occurrence notification screen 121 is displayed as a pop-up on the overall display screen 101 (see Figure 8).

[0099] The event notification screen 121 is equipped with an event details display unit 122. The event details display unit 122 displays that the event to be notified has occurred, the location where the event occurred (in this example, the toilet), and the camera 1 that captured the event (in this example, camera #1).

[0100] Furthermore, the event occurrence notification screen 121 is provided with a detection image display unit 123. The detection image display unit 123 displays a motion image 22 of the body after the event has been detected. The motion image 22 is extracted from the image captured by camera 1, including the area of ​​the moving body, which includes the person and the carried object. In this example, the motion image 22 extracted from the image captured by camera 1 of #1 is displayed, and the motion image 22 shows a person who is not carrying any luggage as a carried object.

[0101] Furthermore, the event notification screen 121 is provided with a pre-detection image display unit 124. The pre-detection image display unit 124 displays the motion image 22 before the change when the event is detected, and the motion image 22 prior to that. If a person is detected by multiple cameras 1, multiple motion images 22 are displayed. In this example, motion images 22 extracted from the images captured by cameras 1 #1, #2, and #3 are displayed.

[0102] Here, the pre-detection image display unit 124 shows a person holding luggage as an object in the same motion image 22 taken by camera #1 1 when the event was detected. This allows the user to recognize that the person leaving luggage behind has been detected as an event of the first type. Also, the motion image 22 taken by camera #2 1 shows a person holding luggage as an object, similar to the motion image 22 taken by camera #1 1. On the other hand, the motion image 22 taken by camera #3 1 also shows a person holding luggage as an object, similar to the motion image 22 taken by camera #2 1, but the person's appearance (clothing) is different compared to the motion image 22 taken by camera #2 1, suggesting that the person changed clothes before entering the shooting area of ​​camera #2 1.

[0103] Furthermore, the event notification screen 121 is equipped with a "No Problem" button 127 and a "Confirm" button 128. The user visually inspects the motion image 22 at the time of event detection and the motion image 22 prior to the event detection and confirms that there is no problem, then operates the "No Problem" button 127. This returns the user to the overall display screen 101 (see Figure 8). On the other hand, the user visually inspects the motion image 22 at the time of event detection and the motion image 22 prior to the event detection and confirms that there is a problem, then operates the "Confirm" button 128. This transitions the user to the timeline confirmation screen 131 (see Figure 11).

[0104] In this way, the event notification screen 121 allows the user to visually check the details of the situation when the event occurred by viewing the motion image 22 at the time of event detection and the motion image 22 prior to the event detection.

[0105] Next, we will explain the timeline confirmation screen 131 related to the first type of event displayed on the monitoring terminal 3. Figures 11 and 12 are explanatory diagrams showing the timeline confirmation screen 131. Figure 11 is the timeline confirmation screen 131 immediately after the event occurs. Figure 12 is the timeline confirmation screen 131 after a certain amount of time has passed since the event occurred.

[0106] The timeline confirmation screen 131 is provided with a timeline display unit 132. The timeline display unit 132 is provided with an event occurrence display unit 141, a pre-event display unit 142, and a post-event display unit 143.

[0107] As shown in Figures 11 and 12, the event occurrence display unit 141 displays the motion image 22 (second image) before the change and the motion image 22 (first image) after the change, both taken by the same camera 1 (detection camera) when the event occurred. In this example, an event (increase or decrease in carried items) is detected in the image taken by camera 1 #1, and two motion images 22 are displayed showing the partial change in the motion related to the detected event. In the two motion images 22 taken by camera 1 #1, the person changes from a state where they are carrying luggage to a state where they are not carrying luggage.

[0108] The pre-event display unit 142 displays multiple motion images 22 (second images) taken before the event occurred. These motion images 22 were taken by a different camera 1 (surrounding camera) than the camera 1 (detection camera) that captured the detected event. In this example, motion images 22 from cameras #2 and #3 are displayed. Motion image 22 from camera #2 shows a person holding luggage as personal belongings. On the other hand, motion image 22 from camera #3 also shows a person holding luggage as personal belongings, similar to motion image 22 from camera #2, but the person's appearance (clothing) is different compared to motion image 22 from camera #2, suggesting that the person changed clothes before entering the shooting area of ​​camera #2.

[0109] Furthermore, as shown in Figure 12, the post-event display unit 143 displays multiple motion images 22 (third images) taken after the event occurred. The post-event display unit 143 displays motion images 22 taken by cameras 1 (surrounding cameras) other than the camera 1 (detection camera) that captured the detected event. In this example, motion images 22 taken by cameras #4 and #5 are displayed. In the motion image 22 taken by camera #4, the person is captured in the same state as in the motion image 22 taken by camera #1 after the change. In the motion image 22 taken by camera #5, the appearance (clothing) of the person is different compared to the motion image 22 taken by camera #4, so it is assumed that the person changed clothes before entering the shooting area of ​​camera #5.

[0110] Furthermore, the time is displayed in the item field 144 of the timeline display unit 132. This allows the user to check the time the moving image 22 was taken. The item field 144 is also provided with a time scale switching unit 145. The time scale switching unit 145 allows the user to change the time scale (time interval) to a predetermined value (for example, 10 seconds, 30 seconds, 1 minute, 5 minutes, 10 minutes) by operating a pull-down menu. When the time scale is shortened, the number of moving images 22 displayed increases, and when the time scale is lengthened, the number of moving images 22 displayed decreases. The time scale switching unit 145 is also provided in both the pre-event display unit 142 and the post-event display unit 143. This allows the user to set different time scales for the pre-event display unit 142 and the post-event display unit 143.

[0111] The item column 144 is displayed in a different predetermined color in each of the event occurrence display unit 141, the pre-event display unit 142, and the post-event display unit 143. For example, the item column 144 is displayed in pink in the event occurrence display unit 141, in light blue in the pre-event display unit 142, and in green in the post-event display unit 143.

[0112] Furthermore, the timeline display unit 132 is provided with a display field 146 for the number of change attributes corresponding to each of the displayed moving images 22. The display field 146 displays the number of change attributes, that is, the total number of changes that have appeared in the moving image, including the person and their belongings. The event occurrence display unit 141 displays the number of change attributes for the moving image 22 after the change, based on the moving image 22 before the change. The pre-event display unit 142 displays the number of change attributes based on the moving image 22 before the change shown in the event occurrence display unit 141. The post-event display unit 143 displays the number of change attributes based on the moving image 22 after the change shown in the event occurrence display unit 141.

[0113] Furthermore, in the timeline display unit 132, if there are many cameras 1 (for example, three or more cameras 1) and some of the moving images 22 from some of the cameras 1 do not fit within the display area, a scroll unit (not shown) will be displayed, and the user can operate the scroll unit to display the moving images 22 that are outside the display area.

[0114] Furthermore, the timeline confirmation screen 131 is equipped with a video playback unit 133. In the video playback unit 133, when the user selects a moving image 22 on the timeline display unit 132, the image 21 captured by camera 1 corresponding to the selected moving image 22 is displayed as a video. In this example, the moving image 22 after the change in the event occurrence display unit 141 is selected.

[0115] Furthermore, the timeline confirmation screen 131 is provided with an image detail display section 134, a "Before Change Confirmation" button 135, and a "After Change Confirmation" button 136. The image detail display section 134 displays enlarged versions of the moving image 22 before the change (second image) and the moving image 22 after the change (first image) when the event occurred. If the user wants to check the carried object that appeared in the moving image 22 before the change, they operate the "Before Change Confirmation" button 135. This transitions to the change confirmation screen 151 (see Figure 13) regarding the carried object that appeared in the moving image 22 before the change. If the user wants to check the carried object that appeared in the moving image 22 after the change, they operate the "After Change Confirmation" button 136. This transitions to the change confirmation screen 151 (see Figure 13) regarding the carried object that appeared in the moving image 22 after the change.

[0116] Furthermore, in the image detail display unit 134, a detection frame 147 (frame image) surrounding the area where a partial change in the moving object has occurred is superimposed on the moving object image 22 before the change or the moving object image 22 after the change. In this example, the detection frame 147 is displayed in the area of ​​the carried object (luggage) that disappears in the moving object image 22 after the change in the moving object image 22 before the change. This detection frame 147 represents the area of ​​the carried object detected from the image captured by camera 1 during the carried object detection process.

[0117] Furthermore, the timeline confirmation screen 131 is equipped with a "On-site Response" button 137 and a "Map Confirmation" button 138. If the user visually inspects the captured images 21 or motion images 22 and determines that on-site response is necessary, they operate the "On-site Response" button 137. This sends a notification requesting on-site response to the department in charge. Alternatively, if the user visually inspects the captured images 21 or motion images 22 and wants to check the positional relationship on a map, they operate the "Map Confirmation" button 138. This transitions to the map confirmation screen 161 (see Figure 14).

[0118] In this way, on the timeline confirmation screen 131, the user can visually check the situation of the moving object (person and object) at the time the event occurred, as well as the situation of the moving object before and after the event occurred, by visually checking the moving object images 22 at each time point displayed on the timeline display unit 132. In addition, the user can visually check the situation of partial changes in the moving object by visually checking the video of the moving object images 22 displayed on the video playback unit 133. Furthermore, the user can visually check the situation of partial changes in the moving object in detail by visually checking the enlarged moving object images 22 displayed on the image detail display unit 134.

[0119] Next, we will explain the change location confirmation screen 151 related to the first type of event displayed on the monitoring terminal 3. Figure 13 is an explanatory diagram showing the change location confirmation screen 151.

[0120] The change confirmation screen 151 is equipped with an image display unit 152. When the "Confirm Before Change" button 135 is pressed on the timeline confirmation screen 131 (see Figures 11 and 12), the motion image 22 before the change occurred is displayed on the image display unit 152. When the "Confirm After Change" button 136 is pressed on the timeline confirmation screen 131, the motion image 22 after the change occurred is displayed on the image display unit 152. In this example, the motion image 22 before the change is displayed on the image display unit 152. In addition, the image display unit 152 displays a detection frame 147 (frame image) of the carried item (bag) related to the detected event (in this example, an increase or decrease in carried items) superimposed on the motion image 22.

[0121] Furthermore, the change confirmation screen 151 is equipped with a "Change Search" button 153, a "Change Registration" button 154, and a timeline display section 155.

[0122] When a user operates the "Register Change" button 154, the facility management server 2 performs a process in the image display unit 152 to register the portable object that appeared in the moving image 22 into the database.

[0123] When a user operates the "Change Search" button 153, the facility management server 2 performs a search for the items carried by the user, and the search results are displayed on the timeline display unit 155. At this time, the facility management server 2 searches for items similar to the items carried that are related to the detected event (in this example, an increase or decrease in items carried by the user) from among the items that have been previously detected and registered in the database. Specifically, similarity is determined based on the characteristic information of the items carried by the user obtained in the item detection process, and items that fall within a predetermined similarity range are extracted.

[0124] The timeline display unit 155 displays the carried item image 23 as a search result. The carried item image 23 is extracted from the image taken by camera 1, showing the area of ​​the carried item. In this example, the carried item image 23 showing a bag as the carried item is displayed. The timeline display unit 155 is provided with cells partitioned by camera 1 and time period, and the carried item image 23 is displayed in the corresponding cell for the camera 1 that took the picture of the carried item and the time the carried item appeared, that is, the time the image in which the carried item appeared was taken. In this example, multiple similar carried items were found, but no identical carried items exist.

[0125] Furthermore, the timeline display unit 155 is provided with a period change unit 157 and a time zone change unit 158. When the user operates the period change unit 157, a period selection menu is displayed, where the user can change the period to be searched. When the user operates the time zone change unit 158, a time zone selection menu is displayed, where the user can change the range of time zones to be searched.

[0126] Additionally, the change confirmation screen 151 is equipped with a "back" button 115. When the user operates the "back" button 115, they return to the timeline confirmation screen 131 (see Figures 11 and 12).

[0127] In this way, the change confirmation screen 151 allows the user to easily check whether similar items to the detected event (in this example, an increase or decrease in the number of items carried) have been detected in the past. In particular, by visually inspecting the item images 23 displayed on the timeline display unit 155, the user can check what kind of items are similar to the items related to the detected event, and at what time period in which images taken by which camera 1 those items appeared in the past.

[0128] Next, we will explain the map confirmation screen 161 related to the first type of event displayed on the monitoring terminal 3. Figure 14 is an explanatory diagram showing the map confirmation screen 161.

[0129] The map confirmation screen 161 displays a map image 162 showing the layout of the monitoring area. Icons 163 indicating the installation locations of cameras 1 installed in the monitoring area are superimposed on the map image 162. In this example, a floor in a commercial facility is the target of monitoring. The floor has multiple shops and restrooms. Multiple cameras 1 are also installed on the floor. Cameras 1 photograph corridors and entrances to the restrooms.

[0130] Furthermore, the map confirmation screen 161 is equipped with a button 165 labeled "Before Event Occurrence" and a button 166 labeled "After Event Occurrence". When the user operates the "Before Event Occurrence" button 165, the map confirmation screen 161 in the state before the event occurred is displayed. When the user operates the "After Event Occurrence" button 166, the map confirmation screen 161 in the state after the event occurred is displayed. In this example, the map confirmation screen 161 is in the state after the event occurred.

[0131] Furthermore, on the map confirmation screen 161, the camera 1 that displayed the motion image 22 on the timeline confirmation screen 131 (see Figures 11 and 12), that is, the camera 1 that captured the detected event (increase or decrease in carried items) (detection camera), and the camera 1 that captured motion (people) related to the event before or after the detected event occurred (surrounding cameras) are highlighted. In this example, the map confirmation screen 161 is shown in the state after the event has occurred, and the camera 1 of #1 that captured the event, and the cameras 1 of #4 and #5 that captured motion related to the event after the event occurred are highlighted. Specifically, the icons 163 of cameras 1 of #1, #4, and #5 are drawn in a predetermined color (for example, red). Note that on the map confirmation screen 161 in the state before the event has occurred, the icons 163 of cameras 1 of #1, #2, and #3 are highlighted. In addition, the camera 1 that captured the detected event (detection camera) is highlighted as being of particularly high importance. In this example, the outer perimeter of camera 1 of #1 is drawn with a thick line.

[0132] Furthermore, on the map confirmation screen 161, when the user selects camera 1, specifically by operating the camera 1 icon 163, the screen transitions to a screen (not shown) displaying the real-time image (live image) captured by the selected camera 1.

[0133] Additionally, the map confirmation screen 161 is equipped with a "back" button 115. When the user operates the "back" button 115, they return to the timeline confirmation screen 131 (see Figures 11 and 12).

[0134] In this way, the map confirmation screen 161 allows the user to check the installation status of cameras 1 in the monitoring area, in particular the installation status of the camera 1 that was the source of the motion image 22 displayed on the timeline confirmation screen 131 (see Figures 11 and 12), that is, the camera 1 that captured the detected event (detection camera) and the camera 1 that captured motion related to the event before or after the detected event occurred (surrounding camera), specifically the relative positions and shooting ranges of each camera 1.

[0135] Next, we will explain the measurement target confirmation screen 171 related to work measurement, which is displayed on the monitoring terminal 3. Figure 15 is an explanatory diagram showing the measurement target confirmation screen 171.

[0136] In this embodiment, work measurement is performed on the facility management server 2 based on images captured by camera 1. In this example, the work to be measured is that of a person (worker) using a cart to carry luggage. On the facility management server 2, motion detection processing detects the cart moved by the person as a carried item, and event detection processing detects an increase or decrease in the amount of luggage loaded on the cart as an increase or decrease in carried items. In addition, the facility management server 2 has characteristic information of each person (worker) registered in a database in advance, and in the person detection processing, the person performing the work is identified based on the characteristic information of each person.

[0137] The measurement target confirmation screen 171 is provided with a timeline display unit 172. The timeline display unit 172 displays an image 23 of a cart loaded with luggage as the work measurement result. The timeline display unit 172 is provided with cells partitioned by items for people and time periods, and the image 23 of the luggage is displayed in the corresponding cell with respect to the person performing the work and the time the luggage (cart) appeared, that is, the time the photograph was taken in which the luggage appeared.

[0138] Furthermore, the timeline display unit 172 is provided with a display area 175 for work measurement results corresponding to each of the displayed images of the carried items 23. In the display area 175, information regarding work efficiency, specifically the number of items loaded on the cart (loading capacity) and the utilization rate (loading rate), which is the ratio of the actual loading capacity to the maximum loading capacity of the cart, are displayed as work measurement results corresponding to the images of the carried items 23.

[0139] Furthermore, on the measurement target confirmation screen 171, if an abnormal condition is detected where the cart is loaded with more luggage than its load limit (maximum load capacity), an alert is displayed. In this example, for person C, at the 17:00 time slot, the measurement result display field 175 is highlighted in a different color (for example, red) than when it is normal, as an alert is displayed. In this example, the cart's load limit was 10 items, but 12 items were loaded onto the cart.

[0140] Next, we will explain the screen displayed on the monitoring terminal 3 when a change involving the movement of carried items (the second type of event) is detected.

[0141] First, we will explain the event occurrence notification screen 121 related to the second type of event displayed on the monitoring terminal 3. Figure 16 is an explanatory diagram showing the event occurrence notification screen 121.

[0142] In the event notification screen 121 shown in Figure 16, the event content display unit 122 displays that the event to be notified has occurred, the location where the event occurred (in this example, the toilet), and the camera 1 that captured the event (in this example, camera #1). The detection image display unit 123 displays the motion image 22 after the event was detected. The motion image 22 shows a person wearing a coat. The pre-detection image display unit 124 displays the motion image 22 before the event was detected and the motion image 22 from before that.

[0143] Here, the pre-detection image display unit 124 shows a person holding a coat as an object in the same motion image 22 taken by camera #1 1 when the event was detected. This allows the user to recognize that the person putting on the coat they were holding was detected as a second type of event. Also, the motion image 22 taken by camera #2 1 shows a person holding a coat as an object, similar to the motion image 22 taken by camera #1 1. On the other hand, the motion image 22 taken by camera #3 1 shows a person wearing the coat, suggesting that the person took off the coat before entering the shooting area of ​​camera #2 1.

[0144] Next, we will explain the timeline confirmation screen 131 related to the second type of event displayed on the monitoring terminal 3. Figures 17 and 18 are explanatory diagrams showing the timeline confirmation screen 131. Figure 17 is the timeline confirmation screen 131 immediately after the event occurs. Figure 18 is the timeline confirmation screen 131 after a certain amount of time has passed since the event occurred.

[0145] In the timeline confirmation screen 131 shown in Figures 17 and 18, the event occurrence display unit 141 displays the motion image 22 (second image) before the change and the motion image 22 (first image) after the change, both taken by the same camera 1 (detection camera) when the event occurred. In this example, the event (movement of an object) is detected in the image taken by camera 1 #1, and two motion images 22 are displayed showing the partial change of the motion related to the detected event. In the two motion images 22, the person changes from holding a coat as an object to wearing the coat.

[0146] Furthermore, the pre-event display unit 142 displays multiple motion images 22 taken before the event occurred. In this example, motion images 22 taken by cameras #2 and #3 are displayed. The motion image 22 taken by camera #2 shows a person holding a coat as an item. On the other hand, the motion image 22 taken by camera #3 shows a person wearing a coat, suggesting that the person took off the coat before entering the shooting area of ​​camera #2.

[0147] Furthermore, as shown in Figure 18, the post-event display unit 143 displays multiple motion images 22 taken after the event occurred. In this example, motion images 22 taken by cameras #4 and #5 are displayed. In the motion images 22 taken by cameras #4 and #5, the person is captured in the same state as in the motion image 22 taken by camera #1 after the change, so it is assumed that the person moved to the shooting area of ​​cameras #4 and #5 while still wearing their coat.

[0148] Furthermore, the image detail display unit 134 displays an enlarged view of both the motion image 22 before the change and the motion image 22 after the change when the event occurs. In addition, the image detail display unit 134 superimposes a detection frame 147 (frame image) surrounding the area where a partial change in the motion has occurred onto the motion image 22 before the change. In this example, since the person in the motion image 22 before the change is not wearing the coat that the person in the motion image 22 after the change is wearing, the detection frame 147 is displayed in the areas of the person's torso, both legs, and both arms, as well as in the area of ​​the carried item (coat).

[0149] Next, we will explain the change location confirmation screen 151 related to the second type of event displayed on the monitoring terminal 3. Figure 19 is an explanatory diagram showing the change location confirmation screen 151.

[0150] In the change confirmation screen 151 shown in Figure 19, the image display unit 152 displays either the motion image 22 before the change or the motion image 22 after the change, depending on the user's operation on the timeline confirmation screen 131 (see Figures 17 and 18). In addition, the image display unit 152 overlays the detection frame 147 (frame image) of the carried object related to the detected event (in this example, the movement of the carried object) onto the motion image 22.

[0151] Furthermore, on the change confirmation screen 151, the timeline display section 155 displays the image 23 of the item in the corresponding cell, based on the camera 1 that took the picture of the item and the time the item appeared. In this example, the image 23 of the item 23, which shows a coat, is displayed.

[0152] Next, we will describe the screen displayed on the monitoring terminal 3 when an overall change in a person's appearance (clothing) (the third type of event) is detected.

[0153] First, we will explain the event occurrence notification screen 121 related to the third type of event displayed on the monitoring terminal 3. Figure 20 is an explanatory diagram showing the event occurrence notification screen 121.

[0154] In the event notification screen 121 shown in Figure 20, the event content display unit 122 displays that the event to be notified has occurred, the location where the event occurred (in this example, the toilet), and the camera 1 that captured the event (in this example, camera #1). The detection image display unit 123 displays the motion image 22 after the event was detected. The pre-detection image display unit 124 displays the motion image 22 before the event was detected, as well as the motion image 22 from before that.

[0155] In this example, the appearance (clothing) of the person captured in the moving image 22 taken by camera #1 has changed. Therefore, it can be assumed that the person changed clothes during the period when they temporarily disappeared from the shooting area of ​​camera #1 (in this example, the entrance to the toilet), specifically while they were using the toilet. Note that the moving images 22 taken by other cameras #2 and #3 show the same person as in the moving image 22 taken by camera #1 before the change.

[0156] Next, we will explain the timeline confirmation screen 131 related to the third type of event displayed on the monitoring terminal 3. Figures 21 and 22 are explanatory diagrams showing the timeline confirmation screen 131. Figure 21 is the timeline confirmation screen 131 immediately after the event occurs. Figure 22 is the timeline confirmation screen 131 after a certain amount of time has passed since the event occurred.

[0157] In the timeline confirmation screen 131 shown in Figures 21 and 22, the event occurrence display unit 141 displays multiple motion images 22 taken by the same camera 1 (detection camera) when the event occurred. In this example, the event (change in the person's appearance) is detected in the image taken by camera 1 #1, and two motion images 22 are displayed showing the partial change in the motion related to the detected event, before and after the detected image. In the two motion images 22, the person's clothing (top and bottom) has changed because they changed clothes.

[0158] Furthermore, the pre-event display unit 142 displays multiple motion images 22 taken before the event occurred. In this example, motion images 22 taken by cameras #2 and #3 are displayed. Since the motion images 22 taken by cameras #2 and #3 show the same person as in the pre-change motion image 22 taken by camera #1, it is assumed that the person entered the shooting area of ​​camera #1 from the shooting area of ​​camera #3, through the shooting area of ​​camera #2, while remaining unchanged.

[0159] Furthermore, as shown in Figure 22, the post-event display unit 143 displays multiple motion images 22 taken after the event occurred. In this example, motion images 22 taken by cameras #4 and #5 are displayed. In the motion image 22 taken by camera #4, the person is in the same state as in the motion image 22 taken by camera #1 after the change, so it is assumed that the person entered the shooting area of ​​camera #4 in the same state. On the other hand, in the motion image 22 taken by camera #5, the person has returned to the same state as in the motion image 22 taken by camera #1 before the change, so it is assumed that the person changed clothes before moving into the shooting area of ​​camera #5.

[0160] Furthermore, the image detail display unit 134 displays an enlarged view of both the motion image 22 before the change and the motion image 22 after the change when the event occurs. In addition, the image detail display unit 134 superimposes a detection frame 147 (frame image) surrounding the area where a partial change in the motion occurred onto the motion image 22 before the change. In this example, since the person changed their top and bottom clothes, the detection frame 147 is displayed over the areas of the person's torso, legs, and arms.

[0161] In this embodiment, a detection frame 147 (frame image) is superimposed on the moving image 22 to indicate the location where a partial change occurs in the moving image 22. However, the location where a partial change occurs may be highlighted by another method. For example, the area of ​​the moving image 22 other than the area where the partial change occurs may be filled in with a semi-transparent color, and the area where the partial change occurs may be highlighted.

[0162] Next, we will explain the change location confirmation screen 151 related to the third type of event displayed on the monitoring terminal 3. Figure 23 is an explanatory diagram showing the change location confirmation screen 151.

[0163] In the change confirmation screen 151 shown in Figure 23, the image display unit 152 displays either the motion image 22 before the change or the motion image 22 after the change, depending on the user's operation on the timeline confirmation screen 131 (see Figures 21 and 22). In addition, the image display unit 152 overlays a detection frame 147 (frame image) on the motion image 22, indicating the part of the person where the partial change occurred in relation to the detected event (in this example, a change in the person's appearance).

[0164] Furthermore, on the change confirmation screen 151, the timeline display section 155 displays the person image 24 in the corresponding cell, based on the camera 1 that photographed the person and the time the person appeared. The person image 24 is extracted from the area of ​​the person in the image taken by camera 1, and if there are no objects being carried, the person image 24 is the same as the moving image 22.

[0165] Next, we will explain the fourth type of event that is subject to event detection performed by the facility management server 2. Figure 24 is an explanatory diagram illustrating the fourth type of event.

[0166] In this embodiment, a state in which a person's appearance differs from that of other people is detected as a fourth type of event. The first, second, and third types of events are temporal partial changes in the appearance and belongings of the same person, but the fourth type of event is a partial change in the appearance of the person in question, based on the appearance of a standard person. For example, if a facility requires its staff to wear a predetermined standard dress code, and the person in question is not wearing the standard dress code, then a state in which part of the person's appearance differs from that of other people (a fourth type of event) is detected.

[0167] In the example shown in Figure 24(A), the facility being monitored is a factory. In this case, camera 1 is installed at the entrance to the changing room and films people entering and leaving the changing room. In the factory changing room, people change into common work clothes. Since each person enters the changing room in various clothes, their appearance (clothing) does not match upon entry, but since each person changes into the common work clothes in the changing room, their appearance (clothing) is similar upon exit. Here, if some people do not change into the common work clothes, it is detected as a state in which a person's appearance differs from that of others (fourth type of event). In this example, person E does not change into the common work clothes, so it is detected as a fourth type of event. In this case, person E is determined to be a rule violator, and this fact is notified to the manager.

[0168] In the example shown in Figure 25(B), the facility being monitored is a school. In this case, camera 1 is installed, for example, at the school gate to photograph people arriving at school. At the school, people wear a common uniform when arriving. If some people are not wearing the common uniform, their appearance is detected as different from others (a fourth type of event). In this example, person E is not wearing the common uniform, so this is detected as a fourth type of event. In this case, person E is determined to be an outsider (suspicious person), and this is notified to the monitor. Note that the system detects people whose appearance (clothing) does not match the common work clothes or uniform, but there are multiple patterns of work clothes or uniforms (for example, different for men and women, different for each season, etc.), and people who do not match each of these patterns can be detected and notified. In addition to work clothes or uniforms, people whose commonly used name tags, facility symbols (company emblems, school emblems), logos, names, etc. are not detected may also be targeted for notification. In this case, for example, a person who is not permitted to wear a name tag with a specific strap can be identified as a suspicious person.

[0169] Next, we will explain the detection result database related to the fourth type of event managed by the facility management server 2. Figure 25 is an explanatory diagram showing the contents registered in the detection result database.

[0170] Facility management server 2 registers and manages the detection results of motion detection processing in the detection results database.

[0171] The detection results database registers the serial number, detection date and time (year, month, day, hour, minute, second, millisecond), camera information, overall image, and the number of changed attributes. The camera information and overall image are the same as the example shown in Figure 6. The number of changed attributes is the total number of changes in attributes that appeared on the target person, based on the standard attributes of the person's appearance.

[0172] Here, the facility management server 2 sets standard attributes (standard clothing characteristics) of the appearance of a person (moving object) in the camera 1's shooting area (surveillance area) based on the detection results of motion detection processing over a predetermined period (e.g., 1 hour, 1 day) that have been registered in the detection result database in advance. For example, it obtains standard attributes of a person based on the detection results of motion detection processing for one day, targeting the previous day excluding holidays (in this example, November 19, 2021 (Friday)).

[0173] In motion detection processing, facility management server 2 detects a person (moving object) from the image captured by camera 1 and extracts attributes related to the appearance of the person. Next, facility management server 2 compares the attributes of the person with standard attributes and counts the number of changed attributes, that is, the total number of changes in the attributes that appeared on the person. Next, facility management server 2 compares the number of changed attributes with an alert threshold (for example, 2), and if the number of changed attributes is equal to or greater than the alert threshold, it issues an alert. When extracting attributes related to the appearance of a person, it is advisable to classify them into tops (clothing worn on the upper body) and bottoms (clothing worn on the lower body).

[0174] Next, we will explain the event occurrence notification screen 121 related to the fourth type of event displayed on the monitoring terminal 3. Figure 26 is an explanatory diagram showing the event occurrence notification screen 121.

[0175] On the monitoring terminal 3, when the facility management server 2 detects that a person's appearance differs from that of other people (the fourth type of event), the event occurrence notification screen 121 is displayed on the overall display screen 101. In this example, when a person wearing clothing different from the school uniform is detected among those entering the school grounds from the school gate, the event occurrence notification screen 121 is displayed.

[0176] In this example, the target facility is a school, and on the overall display screen 101, the target area can be selected from the school gate, the first area of ​​the school, the second area of ​​the school, and the perimeter of the grounds. In this example, the school gate is set as the target of monitoring.

[0177] In the event notification screen 121 shown in Figure 26, the event details display unit 122 displays that the event to be notified has occurred, the location where the event occurred (in this example, the school gate), and the camera 1 that captured the event (in this example, camera #1 installed at the main gate).

[0178] Furthermore, the detection image display unit 123 displays a motion image 22 taken at the time the event was detected. In this example, based on the image captured by camera #1 installed at the school gate (main gate), a person not wearing the common uniform is detected as a fourth type of event, and the detection image display unit 123 displays a motion image 22 showing the person not wearing the common uniform.

[0179] Next, we will explain the timeline confirmation screen 131 related to the fourth type of event displayed on the monitoring terminal 3. Figures 27 and 28 are explanatory diagrams showing the timeline confirmation screen 131. Figure 27 is the timeline confirmation screen 131 immediately after the event occurs. Figure 28 is the timeline confirmation screen 131 after a certain amount of time has passed since the event occurred.

[0180] In the timeline confirmation screen 131 shown in Figures 27 and 28, the event occurrence display unit 141 displays the motion image 22 taken by camera 1 (detection camera) when the event occurred. In this example, an event is detected in the image captured by camera 1 #1, and the motion image 22 of the person related to the detected event is displayed.

[0181] The pre-event display unit 142 displays multiple motion images 22 taken before the event occurred. In this example, motion images 22 taken by cameras #3 and #4 are displayed. This suggests that the person moved sequentially through the shooting areas of camera #4 and camera #3 on the outer perimeter of the site before passing through the main gate, which is the shooting area of ​​camera #1.

[0182] Furthermore, as shown in Figure 28, the post-event display unit 143 displays multiple motion images 22 taken after the event occurred. In this example, motion images 22 taken by cameras #5 and #6 are displayed. This suggests that the person left the main gate and then moved sequentially through the shooting areas of camera #5 and camera #6 on the outer perimeter of the site.

[0183] Furthermore, the timeline display unit 132 displays the number of change attributes, that is, the total number of changes that appeared in the person related to the event, corresponding to each of the moving images 22. The pre-event display unit 142 displays the number of change attributes based on the first moving image 22 in the event-occurrence display unit 141. The post-event display unit 143 displays the number of change attributes based on the last moving image 22 in the event-occurrence display unit 141. In this example, since all the change attribute counts are "0", it is assumed that the person did not change clothes during the event.

[0184] Furthermore, on the timeline confirmation screen 131, a detection frame 147 (frame image) surrounding the area where a partial change in the moving body has occurred is superimposed on the moving body image 22 on the video playback unit 133. In this example, the detection frame 147 is displayed in the area of ​​a part of the body that is different from other people (for example, the legs). This detection frame 147 is set for the image captured by camera 1 in the person detection process of the motion detection process.

[0185] Next, we will explain the map confirmation screen 161 related to the fourth type of event displayed on the monitoring terminal 3. Figure 29 is an explanatory diagram showing the map confirmation screen 161.

[0186] On the map confirmation screen 161, icons 163 indicating the installation locations of cameras 1 installed in the monitoring area are superimposed on a map image 162 showing the layout of the monitoring area. In this example, the facility to be monitored is a school. Multiple cameras 1 are installed at the school. Camera 1 photographs the school gates (main gate, back gate) and the perimeter of the school grounds.

[0187] Furthermore, on the map confirmation screen 161, the camera 1 that displayed the motion image 22 on the timeline confirmation screen 131 (see Figures 27 and 28), that is, the camera 1 that captured the detected event (a state in which a person's appearance differs from other people) (detection camera), and the cameras 1 that captured motion (people) related to the event before or after the detected event occurred (surrounding cameras) are highlighted. In this example, the icon 163 of camera 1 #1, which captured the detected event, is highlighted as a detection camera due to its high importance, while the icons 163 of cameras #3, #4, #5, and #6 are highlighted as surrounding cameras.

[0188] As described above, embodiments have been explained as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these embodiments and can be applied to embodiments that have been modified, replaced, added, or omitted. Furthermore, it is possible to create new embodiments by combining the components described in the above embodiments. [Industrial applicability]

[0189] The facility management device, facility management system, and facility management method according to the present invention have the effect of enabling users to easily and appropriately grasp the situation of an event by detecting predetermined events occurring in a monitoring area in real time, allowing users to immediately check the status of the event that occurred in the monitoring area, and furthermore, by allowing users to check the previous status of moving objects related to the event. These are useful as facility management devices, facility management systems, and facility management methods that detect predetermined events occurring in a monitoring area based on images captured by multiple cameras that photograph the monitoring area within the facility. [Explanation of Symbols]

[0190] 1 Camera 2. Facility management server (facility management device, information processing device) 3. Monitoring terminal (terminal device) 11 Communications Department 12 Storage section 13 processors

Claims

1. A facility management device that uses a processor to perform a process to detect a predetermined event occurring in a monitoring area based on images captured by multiple cameras that photograph the monitoring area, The aforementioned processor, The aforementioned captured images are acquired at each time point, and moving objects, including people, that are the target of surveillance are detected from these captured images. The detection results of the aforementioned moving object are compared to detect partial changes in the moving object, and based on the state of these partial changes in the moving object, an event occurring in the monitoring area is detected. An event occurrence notification screen is output, which includes a first image containing the moving object extracted from the captured image at the time the event was detected, and a second image containing the moving object extracted from the captured image prior to the detection of the event. Furthermore, the facility management device is characterized in that, upon user operation to instruct confirmation of the event on the event occurrence notification screen, it outputs a timeline confirmation screen in which the captured images, including the moving object, taken at the time of the event and before and after the event, are arranged for each camera in order of the time of capture.

2. The aforementioned processor, The system detects the moving object, including the person and their belongings. The facility management device according to claim 1, characterized in that it detects a change in the moving body that involves either an increase or a decrease in the carried items.

3. The aforementioned processor, The system detects the moving object, including the person and their belongings. The facility management device according to claim 1, characterized in that it detects a change in the moving body that involves either the appearance or disappearance of the carried object.

4. The aforementioned processor, The facility management device according to claim 1, characterized in that it detects a change in appearance due to the putting on or taking off of clothing worn by a person as a partial change of the moving body.

5. The aforementioned processor, The facility management device according to claim 1, characterized in that it extracts and outputs the second image from images taken at multiple times prior to the detection of the aforementioned event.

6. The aforementioned processor, Furthermore, the facility management device according to claim 1 is characterized in that it superimposes an image on the captured image that shows the region in which a partial change of the moving object appears.

7. Multiple cameras to film the surveillance area, A facility management system comprising: a facility management device that performs a process to detect a predetermined event occurring in the monitoring area based on images captured by a plurality of cameras, The aforementioned facility management device is The aforementioned captured images are acquired at each time point, and moving objects, including people, that are the target of surveillance are detected from these captured images. The detection results of the aforementioned moving object are compared to detect partial changes in the moving object, and based on the state of these partial changes in the moving object, an event occurring in the monitoring area is detected. An event occurrence notification screen is output, which includes a first image containing the moving object extracted from the captured image at the time the event was detected, and a second image containing the moving object extracted from the captured image prior to the detection of the event. Furthermore, the facility management system is characterized in that, upon user operation to instruct confirmation of the event on the event occurrence notification screen, it outputs a timeline confirmation screen in which the captured images, including the moving object, taken at the time of the event and before and after the event, are arranged for each camera in order of the time of capture.

8. A facility management method in which an information processing device performs a process to detect a predetermined event occurring in a monitoring area based on images captured by multiple cameras that photograph the monitoring area, The aforementioned captured images are acquired at each time point, and moving objects, including people, that are the target of surveillance are detected from these captured images. The detection results of the aforementioned moving object are compared to detect partial changes in the moving object, and based on the state of these partial changes in the moving object, an event occurring in the monitoring area is detected. An event occurrence notification screen is output, which includes a first image containing the moving object extracted from the captured image at the time the event was detected, and a second image containing the moving object extracted from the captured image prior to the detection of the event. Furthermore, the facility management method is characterized in that, upon user operation to instruct confirmation of the event on the event occurrence notification screen, a timeline confirmation screen is output in which the captured images, including the moving object, taken at the time of the event and before and after the event, are arranged for each camera in order of the time of capture.

Images