Video monitoring device, video monitoring system, and video monitoring method

Abstract

An image monitoring device, an image monitoring system, and an image monitoring method suppress a cost of introduction while monitoring an abnormality of a target device, and assist a user in a coping action for the detected abnormality. The image monitoring device includes: a camera image analysis section that detects a device abnormality from an image captured by a monitoring camera, and determines an abnormality content of the detected abnormality; a coping action selection section that acquires a coping action for the determined abnormality content, based on coping action information that associates coping actions for abnormalities according to the abnormality content of the abnormality; and a visualization processing section that generates display data for causing a customer operation terminal to display a monitoring screen that displays an image of a display device, and causes the customer operation terminal to display the monitoring screen using the display data, and in a case where a device abnormality is detected, generates abnormality-time display data that includes display of an image in which the abnormality is detected and a prompt of a coping action for the abnormality, and displays the monitoring screen in a manner that enables a decision operation of the coping action.

Description

Technical Field

[0001] This invention relates to video surveillance devices, video surveillance systems, and video surveillance methods, and is suitable for use in video surveillance devices, video surveillance systems, and video surveillance methods that use images captured by surveillance cameras to monitor abnormalities of target equipment. Background Technology

[0002] Previously, the following technology was known: when an anomaly was detected by an anomaly detection sensor such as a fire detector and a camera set up in association with the sensor, the image captured by the camera was displayed on a terminal for user operation and a notification was sent to a pre-set notification destination (for example, Patent Document 1).

[0003] However, the anomaly reporting system described in Patent Document 1 has the following problems: users can only confirm the images when an anomaly occurs, which is not convenient enough for building managers who need to monitor various devices within the building. Furthermore, the anomaly reporting system described in Patent Document 1 requires the separate installation of anomaly detection sensors and cameras on each device within the building to be monitored, increasing implementation costs.

[0004] Patent Document 1: Japanese Patent Application Publication No. 2013-089092 Summary of the Invention

[0005] This invention was made with the above considerations in mind, and proposes an image monitoring device, image monitoring system, and image monitoring method that can assist users in responding to detected anomalies.

[0006] To address this issue, the present invention provides an image monitoring device that monitors for abnormalities of a device by capturing images from a surveillance camera. The image monitoring device includes: a camera image acquisition unit that acquires image data of the device captured by the surveillance camera; a camera image analysis unit that detects abnormalities in the image data and determines the abnormal content of the detected abnormality; a storage device that stores customer information and response action information, wherein the customer information is obtained by associating information related to customers who have signed maintenance contracts for monitoring the device, the device being monitored under those contracts, and the surveillance camera capturing the images of the device; and the response action information is obtained by associating the response actions to be taken in the event of an abnormality detection with the abnormal content of the abnormality; and a response action selection mechanism. The system includes a response unit that acquires a response action corresponding to the anomaly content determined by the camera image analysis unit based on the response action information stored in the storage device; and a visualization processing unit that generates display data for displaying a monitoring screen on a customer operation terminal operated by the customer, and uses the display data to display the monitoring screen on the customer operation terminal, the monitoring screen displaying an image of the image data acquired by the camera image acquisition unit. If the camera image analysis unit detects an anomaly in the device, the visualization processing unit generates display data for anomaly detection, including a display of the image detecting the anomaly and a prompt for the response action obtained by the response action selection unit in response to the anomaly detection. Using the display data for anomaly detection, the monitoring screen is displayed in a manner that allows the operation of determining the response action to be performed from the customer operation terminal.

[0007] Furthermore, to address this issue, the present invention provides an image monitoring system for monitoring anomalies in a device, comprising: a surveillance camera that captures images of the device; a camera image recording device that records image data of the device captured by the surveillance camera; a customer operation terminal operated by a customer who has signed a maintenance contract for monitoring the device; and an image monitoring device that uses the image data of the device to detect anomalies in the device, and, upon detecting an anomaly, prompts the customer operation terminal with the appropriate action to be taken. The image monitoring device includes: a camera image acquisition unit that acquires image data of the device from the camera image recording device; and a camera image analysis unit that detects anomalies in the device from the image data and determines the content of the detected anomaly. The system includes a storage device that stores customer information and response action information. The customer information is obtained by associating information related to the customer who signed the maintenance contract, the device being monitored under the maintenance contract, and the surveillance camera that captures the device. The response action information is obtained by associating response actions for the anomaly with the anomaly content of the anomaly. A response action selection unit retrieves a response action corresponding to the anomaly content determined by the camera image analysis unit based on the response action information stored in the storage device. A visualization processing unit generates display data for displaying a monitoring screen on a customer operation terminal operated by the customer, and uses this display data to display the monitoring screen on the customer operation terminal. The monitoring screen displays an image of the image data acquired by the camera image acquisition unit. In this video surveillance system, when the camera image analysis unit detects an anomaly in the device, the visualization processing unit generates display data for anomaly detection, which includes the display of the image showing the detected anomaly and the prompts for the corresponding actions obtained by the response action selection unit in response to the detection of the anomaly. Using the display data for anomaly detection, the monitoring screen is displayed in a manner that allows the customer operation terminal to perform an operation to determine the response action to be executed. The customer operation terminal determines the response action to be executed from the response actions prompted in the monitoring screen.

[0008] Furthermore, to address this issue, the present invention provides an image monitoring method for an image monitoring device that uses images captured by a surveillance camera to monitor an abnormality of the device being monitored. The image monitoring device stores customer information and response action information in a storage device. The customer information is obtained by associating information related to a customer who has signed a maintenance contract for monitoring the device, the device being monitored under the maintenance contract, and the surveillance camera capturing the image of the device. The response action information is obtained by associating the response actions to be taken in the event of an abnormality with the content of the abnormality. The image monitoring method includes the following steps: a camera image acquisition step, in which the image monitoring device acquires image data of the device captured by the surveillance camera; a camera image analysis step, in which the image monitoring device detects an abnormality of the device from the image data and determines the content of the detected abnormality; and a response action step. In the selection step, the video monitoring device obtains the response action corresponding to the abnormal content determined in the camera image analysis step based on the response action information stored in the storage device; and in the visualization processing step, the video monitoring device generates display data for displaying the monitoring screen on a customer operation terminal operated by the customer, and uses the display data to display the monitoring screen on the customer operation terminal. The monitoring screen displays the image of the image data obtained in the camera image acquisition step. If an abnormality of the device is detected in the camera image analysis step, in the visualization processing step, the video monitoring device generates abnormality display data including the display of the image that detected the abnormality and a prompt for the detection of the response action obtained in the response action selection step. Using the abnormality display data, the monitoring screen is displayed in a manner that allows the operation of determining the response action to be performed from the customer operation terminal.

[0009] According to the present invention, it is possible to assist users in responding to detected anomalies. Furthermore, issues, structures, and effects other than those described above will become clear through the explanation of the embodiments described below. Attached Figure Description

[0010] Figure 1 This is a block diagram illustrating a structural example of an image monitoring system 100 according to an embodiment of the present invention.

[0011] Figure 2 This is a diagram illustrating an example of customer information.

[0012] Figure 3 This is an example of a diagram representing information about response actions.

[0013] Figure 4This is a diagram illustrating an example of the overall processing flow in the video surveillance system 100.

[0014] Figure 5 It is a diagram that represents data references, etc., in image data analysis and processing.

[0015] Figure 6 It is a diagram that represents data references and other information obtained during the processing of response actions.

[0016] Figure 7 This is a diagram used to illustrate the screen structure of a surveillance video.

[0017] Figure 8 This is an example of a monitoring screen display when an anomaly occurs.

[0018] Figure 9 This is another example of a monitoring screen displaying an abnormal situation.

[0019] Symbol Explanation

[0020] 1. Video surveillance device;

[0021] 2. Buildings;

[0022] 3. Customer operation terminal;

[0023] 4.5 Communication networks;

[0024] 11. Customer Information Input Department;

[0025] 12 storage devices;

[0026] 12a Customer Information Database;

[0027] 12b camera image management database;

[0028] 12c exception record database;

[0029] 12d Response Action DB;

[0030] 13. Camera Image Acquisition Department;

[0031] 14. Visualization Processing Department;

[0032] 15. Camera Image Analysis Department;

[0033] 16. Response Action Options Department;

[0034] 17 Ministry of Communications;

[0035] 21. Camera image recording device;

[0036] 22 surveillance cameras;

[0037] 23 pieces of equipment;

[0038] 100 video surveillance system;

[0039] 121 Customer Information;

[0040] 124 Response Action Information;

[0041] 300 surveillance footage. Detailed Implementation

[0042] In the following description, "interface device" may refer to one or more interface devices. These one or more interface devices may be at least one of the following.

[0043] • One or more I / O (Input / Output) interface devices. An I / O interface device is an interface device between at least one of an I / O device and a remote display computer. The I / O interface device for the display computer can also be a communication interface device. At least one I / O device can be a user interface device, such as an input device like a keyboard or pointer, and an output device like a display device.

[0044] • One or more communication interface devices. One or more communication interface devices can be one or more of the same type of communication interface devices (e.g., one or more NICs (Network Interface Cards)) or two or more of different types of communication interface devices (e.g., NICs and HBAs (Host Bus Adapters)).

[0045] Additionally, in the following description, "memory" refers to one or more memory devices, typically main memory devices. At least one memory device in the memory can be either a volatile memory device or a non-volatile memory device.

[0046] Additionally, in the following description, "persistent storage device" refers to one or more persistent storage devices. Persistent storage devices are typically non-volatile storage devices (such as secondary storage devices), specifically, for example, HDDs (Hard Disk Drives) or SSDs (Solid State Drives).

[0047] Additionally, in the following description, "storage device" refers to at least one of the aforementioned memory or permanent storage devices.

[0048] Additionally, in the following description, "processor" refers to one or more processor devices. At least one processor device typically refers to a microprocessor device such as a CPU (Central Processing Unit), but can also be other types of processor devices such as a GPU (Graphics Processing Unit). At least one processor device can be either single-core or multi-core. Furthermore, at least one processor device can also be a processor core. Alternatively, at least one processor device can also be a broader processor device, such as a portion or all of the hardware circuitry performing the processing (e.g., an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit)).

[0049] Additionally, in the following descriptions, the term "kkk part" is sometimes used to describe functions. However, a function can be implemented by a processor executing more than one computer program, by more than one hardware device (such as an FPGA or ASIC), or by a combination of them. The description of each function is just one example; multiple functions can also be combined into one function, or one function can be divided into multiple functions.

[0050] Furthermore, in the following descriptions, the term "xxxDB" is sometimes used to describe information that yields output from input. This information can be any constructed data (e.g., structured or unstructured data) or a learning model such as a neural network that produces output from input. Therefore, "xxxDB" can be referred to as "xxx information." Additionally, in the following descriptions, the structure of each DB is an example; one DB can be divided into two or more DBs, and all or part of two or more DBs can be implemented by one DB. Furthermore, "DB" is an abbreviation for database.

[0051] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification, when describing elements of the same kind (elements with the same function), the common part of the reference numerals including subscripts and branch numbers (excluding the subscripts and branch numbers) is sometimes used; when describing elements of the same kind, reference numerals including subscripts and branch numbers are sometimes used. For example, when describing surveillance cameras without specifically distinguishing them, it is sometimes referred to as "surveillance camera 22," while when describing each surveillance camera 22 separately, it is sometimes referred to as "surveillance camera 22a," "surveillance camera 22b," and "surveillance camera 22c." Furthermore, repetition of descriptions of elements having the same function as described above is omitted. Additionally, descriptions of general components constituting an elevator, specifically the main hoisting cable, counterweight, car, winch, security camera, and image recording device, are omitted.

[0052] (1) Structure

[0053] Figure 1 This is a block diagram illustrating a structural example of an image monitoring system 100 according to an embodiment of the present invention. Figure 1 As shown, the video surveillance system 100 includes: a video surveillance device 1, a camera image recording device 21 and a surveillance camera 22 in a building 2, and a customer operation terminal 3. The video surveillance device 1 is communicatively connected to the camera image recording device 21 via a communication network 4, and is communicatively connected to the customer operation terminal 3 via a communication network 5.

[0054] In this embodiment, the equipment 23 within building 2 (specifically, roof 23a, elevator 23b, fire-fighting equipment 23c, automatic door 23d, and parking lot 23e) are the monitoring targets, and surveillance cameras 22a to 22e are installed to capture images of each equipment 23. The images captured by each surveillance camera 22 are stored in a camera image recording device 21. The camera image recording device 21 can be, for example, a general computer or storage device, as long as it has at least storage functionality for recording image data and communication functionality for sending the recorded image data to the image monitoring device 1. Furthermore, Figure 1 The device 23 shown is an example, but is not limited to it. For example, the device 23 can also be used to monitor external passages, garbage collection points, etc.

[0055] Furthermore, the building 2 equipped with surveillance cameras 22 that monitor the device 23 being monitored is not limited to one; there can be multiple such buildings. Additionally, in this embodiment, the surveillance camera 22 is treated as a single management unit, but this is not a limitation. For example, a group of buildings consisting of multiple buildings 2 can also be used as a management unit. Furthermore, the device 23 being monitored is not limited to equipment within a building; equipment in structures other than buildings, such as factories or stations, can also be monitored, and these structures can also be used as management units.

[0056] The customer operation terminal 3 is an information processing terminal held or used by a customer, and includes a display device, an input device, and an interface device. The input device may be, for example, a keyboard and a mouse, but may also be, for example, a touch panel integrated with the display device. Specific examples of the customer operation terminal 3 are not particularly limited, and may include, for example, a personal computer, tablet computer, or smartphone. Furthermore, the customer operation terminal 3 is not limited to one, and may include multiple terminals. In this embodiment, "customer" refers to a person or organization (contractor) who has signed a contract (hereinafter, maintenance contract) related to the maintenance and management of building 2.

[0057] The communication network 4 connecting the video monitoring device 1 and the camera video recording device 21, and the communication network 5 connecting the video monitoring device 1 and the customer operation terminal 3, can be the same type of communication network or different types of communication networks. Communication networks 4 and 5 can be, for example, the Internet, a mobile communication network, a wired LAN (Local Area Network), a wireless LAN, a dedicated line, or a combination thereof. Alternatively, communication networks 4 and 5 can also be a single communication network (e.g., the Internet).

[0058] The video surveillance device 1 can be an information processing device having an interface device (not shown), a storage device 12, and a processor connected to them, or it can be a device implemented on the device (e.g., a cloud platform). The aforementioned interface device performs data input and output relative to the video surveillance device 1. Specifically, a keyboard and a display can be provided on the video surveillance device 1, or an operating terminal (not shown) used by operators or salespersons can be used as the interface device for the video surveillance device 1.

[0059] In the video surveillance device 1, the processor is, for example, a CPU (Central Processing Unit). The storage device 12 includes memory, such as ROM (Read Only Memory) and RAM (Random Access Memory). The video surveillance device 1 reads and executes the program code stored in the ROM through the CPU, and the software and hardware cooperate to realize the functions of the processing units (customer information input unit 11, camera image acquisition unit 13, visualization processing unit 14, camera image analysis unit 15, and response action selection unit 16) described later. In addition, the storage device 12 may also include a permanent storage device such as an HDD (Hard Disk Drive) in addition to the ROM and RAM mentioned above. Furthermore, the communication unit 17 is implemented by an interface device with communication functions.

[0060] like Figure 1 As shown, the video monitoring device 1 includes: a customer information input unit 11, a storage device 12, a camera image acquisition unit 13, a visualization processing unit 14, a camera image analysis unit 15, a response action selection unit 16, and a communication unit 17.

[0061] The customer information input unit 11 has the function of inputting information (customer information) related to customers who have signed maintenance contracts for video surveillance of device 23 and the objects they monitor into the customer information DB12a of the storage device 12. The customer information can include the person receiving the maintenance service, the contract holder, the building ID used to identify building 2, information indicating the building address, etc. (To be discussed later...) Figure 2 The following is a specific example of customer information. In this embodiment, information related to the monitoring device 23 of the monitored object in building 2 and information related to the monitoring camera 22 of the monitoring device 23 are registered in the customer information in association.

[0062] The storage device 12 has the function of storing predetermined information. In this embodiment, the storage device 12 is configured with different DBs (databases) according to the categories of stored information. Specifically, the storage device 12 has: customer information DB12a, camera image management DB12b, anomaly records DB12c, and response actions DB12d.

[0063] The customer information DB12a stores information (customer information) related to customers who have signed maintenance contracts for the video surveillance of device 23 and the objects they monitor. As described above, customer information is input from the customer information input unit 11, as will be discussed later. Figure 2The following are specific examples. The camera image management DB12b stores information related to the image data captured by each surveillance camera 22 (camera image information). The camera image information is obtained from the camera image recording device 21 and stored in the camera image management DB12b. The anomaly recording DB12c stores information related to anomalies detected by analyzing the camera image information of each surveillance camera 22 (anomaly information). The analysis of the camera image information is performed by the camera image analysis unit 15. The response action DB12d stores information related to response actions (response action information) that are options for responding to detected anomalies. (Further details will follow later.) Figure 3 The text provides specific examples of information regarding response actions.

[0064] The camera image acquisition unit 13 connects to the camera image recording device 21, which records image data from each surveillance camera 22, via the communication network 4, based on the customer information stored in the customer information DB12a, according to the customer information of each camera (surveillance camera 22). Furthermore, the camera image acquisition unit 13 periodically acquires surveillance images (image data) from the device 23 based on each surveillance camera 22 from the camera image recording device 21, and registers the camera image information based on the acquired image data in the camera image management DB12b. Alternatively, the camera image acquisition unit 13 may be configured to be always connected to the camera image recording device 21. Alternatively, the camera image acquisition unit 13 may directly register the image data acquired from the camera image recording device 21 as camera image information in the camera image management DB12b, but it is preferable to register it as camera image information in the camera image management DB12b with additional predetermined information associated with the image data (e.g., information related to the image data itself, information about the shooting environment of the image data registered in the customer information, etc.).

[0065] The visualization processing unit 14 has the function of processing the data to be output to the customer operation terminal 3, and the function of generating screen data to be output to the customer operation terminal 3. Specifically, for example, the visualization processing unit 14 can generate screens that require input of username and password, monitoring screens of equipment 23 of the monitored objects in the building 2, and response selection screens when anomalies are detected, and provide them to the customer operation terminal 3.

[0066] The camera image analysis unit 15 has the function of detecting anomalies and determining the type (anomaly category) of the detected anomalies based on camera image information from the surveillance camera 22 obtained from the camera image recording device 21. In analyzing the camera image information (i.e., detecting the anomaly and determining the anomaly category as described above), the camera image analysis unit 15 refers to the customer information DB12a and the camera image management DB12b, inputs the necessary data into a predetermined algorithm for detecting anomalies, and thereby obtains the analysis results. Then, the camera image analysis unit 15 registers the obtained analysis results as anomaly information in the anomaly record DB12c. Furthermore, the camera image analysis unit 15 also has the function of outputting the obtained analysis results to the visualization processing unit 14.

[0067] The response action selection unit 16 has the following functions: receiving the analysis results (anomaly detection, anomaly category determination) from the camera image analysis unit 15, and obtaining recommended response actions from the response actions corresponding to the determined anomaly category. Specifically, the response action selection unit 16 refers to the anomaly record DB12c and the response action DB12d, and selects and obtains information related to the action (response action) corresponding to the anomaly recorded in the anomaly information. In addition, the response action selection unit 16 also has the function of outputting the obtained response action-related information to the visualization processing unit 14.

[0068] The communication unit 17 functions as an input / output unit, which is used for data input and output between the various parts of the image monitoring device 1 and the camera image recording device 21, the customer operation terminal 3, or the operation terminal of the maintenance personnel or salesperson (not shown).

[0069] (2) Various information

[0070] The following is a detailed description of the various information stored in the storage device 12, focusing on customer information and response action information set before the execution of video surveillance.

[0071] First, let's explain customer information. Customer information refers to information about customers who have signed maintenance contracts. It is registered in customer information DB12a through customer information input unit 11 based on input operations from sales staff who conduct business activities and maintenance staff who perform maintenance work.

[0072] Figure 2 This is a diagram illustrating an example of customer information. Figure 2 The customer information 121 shown is a specific example of customer information registered in customer information DB12a. In the case of a video surveillance maintenance contract signed with a customer on a building 2 basis, customer information 121 is formed by summarizing customer records that contain information related to the customer and the monitored object for each building 2 that has completed the video surveillance maintenance contract.

[0073] Specifically, in Figure 2 In customer information 121, the customer record consists of items including Customer 1211, Customer ID 1212, Building ID 1213, Building Address 1214, Building Equipment 1215, Building Equipment ID 1216, Camera 1217, and Camera ID 1218. Furthermore, Figure 2 The item shown is just one example; the customer information in this embodiment may also include... Figure 2 Customer records are formed for items other than those listed.

[0074] The customer ID 1211 registers the name of the customer who signed the maintenance contract, and the customer ID 1212 registers the identifier assigned to uniquely identify the customer. The building ID 1213 registers the identifier assigned to uniquely identify building 2, which has the video surveillance equipment under the maintenance contract, and the building address 1214 registers the address of building 2. The building equipment 1215 registers the name of the video surveillance equipment (equipment 23), and the building equipment ID 1216 registers the identifier assigned to identify equipment 23. The camera 1217 registers the name of the surveillance camera 22 that monitors the video surveillance equipment (equipment 23), and the camera ID 1218 registers the identifier assigned to identify the surveillance camera 22.

[0075] However, although Figure 1 Although not illustrated, in this embodiment, the user database may also be stored in the storage device 12. In this embodiment, "user" refers to a person belonging to a customer (e.g., an employee of the customer's company). The user database may have user records per user, preferably each user record containing a user ID, password, and customer ID. The user ID and password are authentication information used to identify users who can access the video surveillance device 1.

[0076] Here, the usage of the user ID and password will be explained. The customer operation terminal 3 connects to the video monitoring device 1 via the communication network 5 through user operation. Based on this connection, the visualization processing unit 14 requests the user to input their user ID and password. That is, the visualization processing unit 14 outputs a message to the customer operation terminal 3 urging the input of the user ID and password. Furthermore, only when the user ID and password registered in the user DB are input from the customer operation terminal 3 can the visualization processing unit 14 output (e.g., display) various information obtained from the customer information DB12a, camera image management DB12b, anomaly record DB12c, and response action DB12d to the customer operation terminal 3. Moreover, the output data produced by the visualization processing unit 14 is not limited to a specific data format; specifically, it can be in the form of images, animations, HTML, XML, or CSV, etc.

[0077] Furthermore, in this embodiment, the description mainly focuses on the image data of the surveillance camera 22, the anomalies detected by analyzing the image data (anomaly category, anomaly content), and the actions taken in response to the anomalies (response actions). However, the video monitoring device 1 of the present invention can also replace this information or, based on this information, display building equipment inquiry responses, maintenance schedules, or maintenance performance, etc. With this configuration, the video monitoring device 1 of the present invention can process various contract information related to building equipment.

[0078] Next, the response action information will be explained. Response action information refers to information related to recommended actions (response actions) detected by the video surveillance equipment (equipment 23) under the maintenance contract. One or more response actions are pre-set based on the content of the anomaly. The response action information only needs to be registered and placed in the response action DB12d before video surveillance by the video surveillance device 1 is performed. Various methods can be used for registration. Specifically, for example, the system administrator can pre-register the response action information in the response action DB12d, or the manufacturer can register response action information for representative anomalies in the response action DB12d during the manufacture or sale of the video surveillance device 1. Alternatively, the response action information can be appropriately customized by the customer side (e.g., the owner or management company of building 2).

[0079] Figure 3 This is an example of a diagram representing information about response actions. Figure 3The response action information 124 shown is a specific example of response action information registered in response action DB12d. Response action information 124 is formed by response action records, which are obtained by associating the equipment 23 of the building 2 that has completed the maintenance contract for video surveillance based on the surveillance camera 22 with the abnormal content, and associating one or more response actions to be taken next when the abnormality is detected.

[0080] Specifically, in Figure 3 In the response action information 124, the response action record consists of items including building equipment 1241, building equipment ID 1242, anomaly category 1243, anomaly content 1244, anomaly content priority 1245, action number 1246, action content 1247, and contact destination 1248. Furthermore, Figure 3 The example shown is just one instance; the response information in this implementation may also include... Figure 3 A record of response actions will be created for items other than those listed. Additionally, in Figure 3 In the document, response action records are shown for "Fire Protection Equipment" under Building Equipment ID "S001" and "Elevator" under Building Equipment ID "E001". However, it can also be assumed that in the actual response action information 124, the records for "Fire Protection Equipment" under Building Equipment ID "S001" and "Elevator" under Building Equipment ID "E001" are displayed respectively. Figure 2 Customer information 121 shows the registration and response records for all equipment shown in building equipment 1215.

[0081] Building equipment ID 1241 and building equipment ID 1242 and Figure 2 The building equipment 1215 and building equipment ID 1216 in the customer information 121 shown correspond to each other. Anomaly category 1243 indicates the category of anomalies that can be detected through image data analysis by the camera image analysis unit 15. Figure 3 In the case of "smoke or fire" which is an anomaly belonging to fire and "abnormal behavior" which is an anomaly belonging to human behavior, examples are given.

[0082] The exception content 1244 registers the specific event representing the exception. This exception content can also be understood as a detailed classification of exception category 1243. Furthermore, the priority corresponding to each exception content registered in exception content 1244 is registered in exception content priority 1245. The priority corresponding to each exception content is preset, but can be changed by administrators during operation. Alternatively, exception content priority 1245 can also be set according to exception category 1243; however, considering the possibility of exceptions belonging to multiple exception categories occurring simultaneously, it is preferable to set the priority for all exception content 1244 belonging to all exception categories 1243. Figure 3In the example shown, as anomaly content belonging to "smoke or fire," three types are registered: "smoke detection," which indicates the detection of smoke from the image data; "fire detection," which indicates the detection of fire from the image data; and "no smoke or fire action," which indicates that no smoke or fire is detected from the image data but smoke or fire is inferred from other conditions in the image (such as flashing lights). As for anomaly content belonging to "abnormal action," three types are registered: "abnormal action (riot) detection," which indicates the detection of a riot from the image data; "abnormal action (stagnation) detection," which indicates the detection of a person remaining in the image data; and "abnormal action (other) detection," which indicates the detection of human actions other than riots and stagnation from the image data. Furthermore, for the above six types of anomaly content 1244, a priority level of "1" to "6" is set in the anomaly content priority 1245, with a lower priority value indicating a higher priority. For example, among the above 6 types of abnormal content 1244, "fire detection" is the most dangerous abnormality, therefore, it is set to the highest priority "1".

[0083] In Action Content 1247, register one or more recommended response actions in the event of detecting Anomaly Content 1244. In Contact Destination 1248, register the contact destinations contacted when executing these response actions. Additionally, in Action Number 1246, for the one or more response actions recommended based on Anomaly Content 1244 (Action Content 1247), register the order in which the recommended response actions are performed. Figure 3 In cases such as "smoke detection," as a response action associated with the recommended action number 1246 ("1"), "call the fire truck" is registered in action content 1247, and the fire department's telephone number "119" is registered in contact destination 1248 as the contact destination for this response action. Additionally, in the case of "smoke detection," as a response action associated with the recommended action number 1246 ("2"), "notify via in-store broadcast" is registered in action content 1247, but since a contact destination is not required for this response action, no specific contact destination is registered in contact destination 1248. In this embodiment, multiple response actions can be registered for a single anomaly, as described above, and by setting the recommended order for these response actions, customers can be easily and clearly informed about the appropriate action to take when an anomaly is detected.

[0084] In addition, Figure 3In the example, the anomaly category 1243 differs according to the building equipment 1241. Therefore, the anomaly content 1244 belonging to the anomaly category 1243 and the action content 1247 corresponding to the anomaly content 1244 also differ according to the building equipment 1241. However, even if different building equipment 1241 are involved, the response action information 124 of this embodiment can register common or repeated anomaly categories 1243. Furthermore, in the response action information 124 (response action record) of this embodiment, the action content 1247 can be determined separately for at least the anomaly content 1244.

[0085] Additionally, response action information 124 (response action DB12d) may be pre-registered by the system administrator. However, if the user is an employee of the management company, it may be necessary to register contact destinations that the maintenance company could not have foreseen, such as needing to contact the building owner when an anomaly occurs. Therefore, as a variation of this embodiment, the user may be able to change or newly register action number 1246, action content 1247, or contact destination 1248 in response action information 124.

[0086] In addition, as described above, abnormal information related to the abnormality detected from the image data is registered in the abnormal information DB12c. However, in this abnormal information, the data items related to the detected abnormality are in the data format corresponding to the response action information 124 (specifically, abnormality category 1243, abnormality content 1244, etc.).

[0087] (3) Processing

[0088] Hereinafter, we will explain how the video monitoring system 100 of this embodiment uses the video data of the surveillance camera 22 to provide a monitoring screen to the customer operation terminal 3, monitors the occurrence of anomalies based on the video data of the surveillance camera 22, and notifies the customer operation terminal 3 when an anomaly is detected to assist in the appropriate response action (e.g., notification) to the anomaly.

[0089] Figure 4 This is a diagram illustrating an example of the overall processing flow in the video surveillance system 100. Figure 4 The diagram focuses on the information display on the monitoring screen and the processing related to operations on the monitoring screen. Basic operations such as logging in and logging out of the system (video monitoring device 1) are omitted from the description.

[0090] according to Figure 4First, the camera image recording device 21 continuously acquires and records image data captured by the surveillance camera 22 installed in the building 2 from the device 23 (step S11). Then, the camera image acquisition unit 13 of the image monitoring device 1 continuously or periodically accesses the camera image recording device 21 to acquire the image data from the surveillance camera 22, and registers the camera image information based on the image data in the camera image management DB12b (step S21). In addition, in step S21, the camera image acquisition unit 13 can also acquire various information required to generate camera image information based on the image data from the camera image recording device 21 and the customer information DB12a.

[0091] Next, if the predetermined conditions in the video monitoring device 1 are met, the camera image analysis unit 15 begins image data analysis processing for each image data. Furthermore, the predetermined conditions can be, for example, a predetermined period of time after the monitoring screen is activated in the customer operation terminal 3, or a predetermined period of time unrelated to the function of the monitoring screen. In the former case, anomaly detection is performed only when the monitoring screen is activated in the customer operation terminal 3, thus reducing the processing load in the video monitoring device 1. On the other hand, in the latter case, image data analysis is performed periodically on all image data acquired from the camera image recording device 21, regardless of the operating status in the customer operation terminal 3; therefore, the processing load is higher, but the occurrence of anomalies can be detected quickly and reliably. Alternatively, predetermined conditions other than those mentioned above can also be determined.

[0092] In the image data analysis and processing, the camera image analysis unit 15 selects one unanalyzed image data from the image data of the analysis objects registered in the camera image management DB12b, and obtains the camera image information and customer information related to the selected image data from the camera image management DB12b and the customer information DB12a (step S22). For example, when the monitoring screen is activated in the customer operation terminal 3, the customer ID and building ID entered by the customer are sent to the image monitoring device 1 when the screen is activated. Therefore, the image data of the building equipment 1215 associated with these IDs in the customer information 121 (i.e., the image data captured by the camera 1217) is the image data of the analysis objects.

[0093] Next, the camera image analysis unit 15 uses the information obtained in step S22 to perform image analysis to check for anomalies in the image data of the analysis target (step S23). In the image analysis in step S23, the response action information 124 (see reference) is processed. Figure 3 The detection of anomalies corresponding to anomaly category 1243 and anomaly content 1244.

[0094] Next, the camera image analysis unit 15 determines whether an anomaly was detected in the image analysis of step S23 (step S24). If no anomaly was detected in step S24 (step S24 No), the image (still image or moving image) of the image data for which no anomaly was detected is sent to the visualization processing unit 14, proceeding to step S28 described later. If an anomaly was detected in step S24 (step S24 Yes), the camera image analysis unit 15 determines the category and content of the detected anomaly, generates anomaly information based on the analysis results containing the determined information, and registers the generated anomaly information in the anomaly record DB12c (step S25).

[0095] Furthermore, in the following section Figure 5 The image shown in step S22 to S25 illustrates images of data references and other data used in the image data analysis and processing.

[0096] After the processing in step S25, the response action selection unit 16 performs response action acquisition processing to obtain the response action corresponding to the detected anomaly.

[0097] In the response action acquisition process, the response action selection unit 16 refers to the anomaly information registered in the anomaly record DB12c in step S25, uses the anomaly category and anomaly content as keywords, and refers to the response action DB12d to obtain all response action records that match the anomaly category 1243 and anomaly content 1244 (step S26). Furthermore, if the video monitoring device 1 has a pop-up notification function to report anomaly detection, the response action selection unit 16 can also notify the visualization processing unit 14 of anomaly detection at the timing specified in step S26, and the visualization processing unit 14 can send a pop-up notification to the customer operation terminal 3 via the communication unit 17. The timing for the pop-up notification can be any of the timings in steps S27 to S29 described later, but by using an earlier timing as in step S26, the anomaly detection can be reported to the customer quickly.

[0098] Next, the response action selection unit 16, referring to the anomaly content priority 1245 of the response action record obtained in step S26, rearranges multiple anomaly detection-related information according to the anomaly content priority 1245 if anomalies were detected in the previously performed image data analysis of other image data (step S27). Step S27 determines which anomaly to notify first (e.g., the one displayed at the top of the monitoring screen) when anomalies are detected in multiple devices 23 simultaneously. Therefore, if no anomalies were detected in the previously performed image data analysis of other image data, step S27 can be skipped. After step S27 is completed, the response action selection unit 16 sends the reordered information of one or more response action records to the visualization processing unit 14, proceeding to step S28.

[0099] Furthermore, in the following section Figure 6 The image shows the data reference obtained during the processing of the response actions in steps S26 to S27.

[0100] In step S28, a predetermined processing unit of the image monitoring device 1 (for example, it could be the camera image analysis unit 15, or an overall control unit not shown) determines whether there is any unanalyzed image data remaining. If unanalyzed image data exists (yes in step S28), the process returns to step S22 and repeats. If image data analysis of all image data of the analysis target is completed (no in step S28), the process proceeds to step S29. Here, "all image data of the analysis target" can also be replaced with image data from the surveillance cameras 22 of all devices 23 in the building 2.

[0101] In step S29, the visualization processing unit 14 uses the various information received from the camera image analysis unit 15 and the response action selection unit 16 in steps S22 to S28 to generate display data for the monitoring screen. The generated display data includes display data related to devices 23 where anomalies were detected, as well as display data related to devices 23 where no anomalies were detected. Furthermore, the visualization processing unit 14 sends the display data to the customer operation terminal 3, thereby enabling the customer operation terminal 3 to display the received display data on the monitoring screen after startup (equivalent to step S33 described later).

[0102] In addition, Figure 4In the processing flow, after the image data analysis and processing (and response action acquisition processing) of all image data of the analysis object is completed, the visualization processing unit 14 generates display data. However, as a variation, display data may also be generated whenever the image data analysis and processing (and response action acquisition processing) for each image data is completed, and the display data may be sent to the customer operation terminal 3 at any stage.

[0103] When step S29 is completed, the video monitoring device 1, in principle, ends a series of processes. However, if the customer operation terminal 3 determines to perform a specific response action, the response action can also be performed on the video monitoring device 1 (step S30). For example, when a response action (not shown) that can be performed by the response action processing unit of the video monitoring device 1 is set (specifically, "notifying via in-hall broadcast", etc.), the response action processing unit performs the response action based on the operation shown on the monitoring screen of the customer operation terminal 3 (step S35 described later).

[0104] Next, while referring to Figure 4 The processing in customer operation terminal 3 will be explained. Furthermore... Figure 4 The processing flow shown is just one example of the processing in customer terminal 3, and is not limited to it. For example, the customer can also start the monitoring screen by operating customer terminal 3 before receiving the pop-up notification of an anomaly detection.

[0105] according to Figure 4 When a pop-up notification is received from the visualization processing unit 14 of the video monitoring device 1 in step S26, the customer operation terminal 3 sends a pop-up notification based on sound, display, or vibration to inform the customer of the detected anomaly (step S31). The pop-up notification function is a generally known technology, therefore, a detailed description is omitted.

[0106] Next, upon receiving the pop-up notification from step S31, or according to the customer's wishes, the customer performs a pre-defined operation, thereby activating the monitoring screen on the customer operation terminal 3 (step S32). When the monitoring screen is activated, the system requests the input of the customer ID (user ID) assigned to the customer and the building ID of the building 2 equipped with the monitoring device 23. These customer IDs and building IDs are transmitted to the video monitoring device 1 via the communication network 5. Additionally, the aforementioned authentication based on the user ID and password can also be performed when the monitoring screen is activated.

[0107] After the monitoring screen is activated, the customer operation terminal 3 uses the display data received from the visualization processing unit 14 of the video monitoring device 1 to display the monitoring screen (step S33).

[0108] Furthermore, the customer observes the monitoring screen displayed in step S33 to confirm whether an anomaly has been detected (step S34). If no anomaly is detected, the monitoring screen displays "Normal" (step S34 No), and the customer does not need to perform any special operation; the process returns to step S33, and the monitoring screen continues to display (including updating the displayed content). Additionally, the customer can also end the monitoring screen at this time. On the other hand, if an anomaly is detected, the monitoring screen displays "Abnormal" (step S34 Yes), in which case the customer is requested to perform the operation in step S35.

[0109] Furthermore, in the following section Figure 7 The image shows an example of the screen structure during "normal" and "abnormal" monitoring.

[0110] If an "abnormal situation" is displayed on the monitoring screen, in step S35, the customer confirms that an abnormal image has been detected and presses the response action display button (described later). Figure 7 The response action display button 3022) shows the response actions in a list (described later). Figure 7 The response actions to be taken are outlined in the 3203 list.

[0111] Next, the customer performs a predetermined operation to execute the response action determined in step S35, thereby executing the response action by either the customer operation terminal 3 or the video monitoring device 1 (step S30 or step S36). The predetermined operation to execute the response action is, for example, equivalent to pressing the "contact destination" button displayed on the monitoring screen. Furthermore, it is possible to consider which of the customer operation terminal 3 or the video monitoring device 1 is preset in the response action information (response action DB12d). Specifically, for example, in the case of a response action where a specific phone number is registered in the contact destination, it is more efficient for the customer operation terminal 3 to directly dial the number in step S36. Conversely, in the case of a response action such as making an in-building announcement in building 2 without a specific phone number registered in the contact destination, it is sometimes more efficient for the video monitoring device 1 to take the lead in executing the response action in step S30.

[0112] After taking action, customers can stop operating the monitoring screen, but it is preferable to continue displaying the information on the monitoring screen while it is being updated until the anomaly is actually eliminated.

[0113] Furthermore, various structures can be considered for the execution of response actions, as follows. For example, when contact is established with the destination according to the response action, images of the detected anomaly can be repeatedly played on the monitoring screen to facilitate the communication of the abnormal situation. Additionally, as an example, the response action in case of anomalies can be configured to connect directly or indirectly from the customer operation terminal 3 to the building's public address system or the intercom in the elevator car, enabling inquiries to users of building 2. Furthermore, as an example, the response action in case of anomalies can be configured to connect directly or indirectly from the customer operation terminal 3 to signage within building 2 or the LCD screen in the elevator car, displaying images of the anomaly to facilitate the communication of information to security personnel, emergency medical teams, etc. Additionally, as an example, the response action in case of anomalies can also include remote control of equipment (disaster prevention equipment, etc.) within building 2.

[0114] In addition, if no anomalies are detected, the monitoring screen can be updated periodically. Each update involves analyzing the image data and displaying the images on the monitoring screen.

[0115] Furthermore, the history information regarding the presence or absence of anomaly detection and the notification received during anomaly detection can be saved and stored in the storage device 12 of the video monitoring device 1. With this structure, the stored logs can be used for post-detection verification of anomalies.

[0116] In addition, Figure 4 The processing flow illustrates the process of launching the monitoring screen after a pop-up notification is received when an anomaly is detected in the customer operation terminal 3. However, the processing flow for anomaly detection in this embodiment is not limited to the above process. For example, if an anomaly is detected in the display of the monitoring screen, the detection of the anomaly can also be separately notified to the customer operation terminal 3 by a predetermined means such as a pop-up notification. Alternatively, if an anomaly is detected in the display of the monitoring screen, a pop-up notification may not be given, and the display of the monitoring screen may be switched to the method described later. Figure 8 , Figure 9 The monitoring screen shows the abnormal situation described in the document.

[0117] Furthermore, in the above description, as an example, the analysis of camera images (image data) in step S22 is explained based on the state of the customer operation terminal 3 when the monitoring screen is started, etc. However, the opportunity for analyzing image data in this embodiment is not limited to this. For example, whenever the camera image acquisition unit 13 acquires image data from the camera image recording device 21, the camera image analysis unit 15 may also perform image data analysis, etc.

[0118] As mentioned above, refer to Figure 4The overall processing in the video surveillance system 100 has been explained, but the following is a detailed explanation, with reference to the appendix. Figure 1 The analysis and processing of image data and the processing of response actions in the image monitoring device 1, as well as the display of the monitoring screen in the customer operation terminal 3, are explained.

[0119] (3-1) Image Data Analysis and Processing

[0120] Figure 5 It is a graph representing data references, etc., in image data analysis and processing. Figure 5 As shown inside the camera image analysis unit 15, Figure 5 The references shown (referencing, obtaining, sending, recording) and Figure 4 The corresponding steps S22 to S25 are processed.

[0121] While referring to Figure 5 On the one hand, the flow of data in image data analysis and processing is explained.

[0122] First, when a customer requests to display a monitoring screen based on their operation of the customer's operating terminal 3, the visualization processing unit 14, which receives the request via the communication unit 17, sends the customer ID (user ID) of the customer operating the customer's operating terminal 3 and the building ID of the building 2 being monitored, contained in the request, to the camera image analysis unit 15.

[0123] Next, the camera image analysis unit 15 uses the received customer ID and building ID as keywords and refers to the customer information DB12a to obtain the camera ID 1218 of the matching customer record. Furthermore, the camera image analysis unit 15 uses the obtained camera ID 1218, the aforementioned customer ID, and building ID as keywords and refers to the camera image management DB12b to obtain the "image recording date and time (start)" and "image recording date and time (end)" representing the recording date and time of the object's image data, as well as the "image data" which is the main body of the object's image data, as information required for analyzing the image data.

[0124] Next, the camera image analysis unit 15 performs image analysis (image data analysis) on the acquired image data to determine if any anomalies have occurred. Furthermore, the camera image analysis unit 15 is equipped with a program that includes anomaly detection algorithms for camera images (still or moving images) based on machine learning and other methods. These anomaly detection algorithms can detect, for example, equipment malfunctions such as unclosed automatic doors, smoke, fires, abnormal behavior of building users (riots, loitering), and suspicious behavior (graffiti, pranks, etc.).

[0125] When the camera image analysis unit 15 detects equipment malfunctions such as an open automatic door, smoke, fire, abnormal behavior of building users (riots, loitering), or suspicious behavior (graffiti, pranks) through image data analysis, it determines the anomaly category and content, generates anomaly information based on the analysis results, and registers it in the anomaly record DB12c. The specific data structure of the anomaly information is as follows: Figure 5 As shown, in addition to the anomaly category and content, and the various information obtained from the customer information DB12a and camera image management DB12b in step S22, the system also includes the anomaly detection date and time, and image data containing the image of the anomaly state. Furthermore, the "image data" of the anomaly information registered in the anomaly record DB12c does not need to be the entire image data that has undergone image analysis. For example, it is preferable to generate image data representing images 5 seconds before and after (a total of 10 seconds) from the anomaly detection date and time, and use this as the "image data" of the anomaly information. In this way, by using a portion of the original image data that has undergone image analysis as the "image data" of the anomaly information, it is possible to ensure that the image representing the detected anomaly is present while suppressing the increase in the data size of the anomaly information.

[0126] (3-2) Response actions achieved

[0127] Figure 6 It is a diagram that represents data references and other information obtained during the processing of response actions. Figure 6 The references (references, acquisitions, transmissions) shown in the data are related to... Figure 4 The corresponding processing steps S26 to S27 are as follows.

[0128] While referring to Figure 6 On the one hand, it explains the flow of data obtained during the response operation.

[0129] After the image data analysis by the camera image analysis unit 15 is analyzed and the analysis results are saved as anomaly information in the anomaly record DB12c, the response action selection unit 16 refers to the anomaly record DB12c to obtain the anomaly category and anomaly content of the detected anomaly. Next, the response action selection unit 16 uses the obtained anomaly category and anomaly content as keywords and refers to the response action DB12d to obtain the action number 1246, action content 1247, and contact destination 1248 of the matching response action record.

[0130] Then, the response action selection unit 16 will obtain the abnormal information (such as) from the abnormal record DB12c. Figure 5As shown, information including a portion of customer information and response information obtained from response action DB12d are combined and sent to the visualization processing unit 14. Furthermore, the visualization processing unit 14 uses the received information to generate display data for the monitoring screen, which is then output to the customer operation terminal 3 via the communication unit 17 and the communication network 5. This displays information related to the detected anomaly on the monitoring screen of the customer operation terminal 3. On the other hand, if no anomaly is detected, according to... Figure 5 The camera image analysis unit 15 obtains information from the camera image management DB12b, and the visualization processing unit 14 generates display data for the monitoring screen, which is then displayed on the monitoring screen of the customer operation terminal 3 under normal conditions. In this way, the customer (user) can confirm the camera image captured by the surveillance camera 22 of device 23 through the monitoring screen, whether in normal or abnormal conditions. In abnormal conditions, the customer can also confirm the response actions to the abnormality, as described later.

[0131] (3-3) Monitoring screen

[0132] Figure 7 This diagram illustrates the screen structure of a monitoring screen. The monitoring screen 300 displayed on the customer operation terminal 3 is, for example, a GUI (Graphical User Interface). Based on the display data generated by the visualization processing unit 14, it displays the image of the device 23 captured by the surveillance camera 22, anomalies detected from the image of the device 23, and corresponding response actions.

[0133] like Figure 7 As shown, the monitoring screen 300 prepares different screen structures in the monitoring screen 301 when it is normal (when no abnormality is detected) and the monitoring screen 302 when it is abnormal (when more than one abnormality is detected).

[0134] The monitoring screen 301 under normal conditions consists of: a building information display bar 3011, which displays information related to the building 2 where the monitored devices 23 are installed; a message display bar 3012, which displays a message indicating that no abnormality has been detected; and a camera image display bar 3013, which displays the name of each monitored device 23 and the camera images (still images or moving images) captured by the device 23.

[0135] The monitoring screen 302 during an anomaly is configured such that, in addition to the building information display bar 3011 and camera image display bar 3013, which are the same as those in the monitoring screen 301 during normal operation, it also includes: an anomaly detection information display bar 3021, which displays information related to the detected anomaly; a response action display button 3022, which displays a list of recommended response actions (actions to be taken next) for the detected anomaly; and a response action list 3023, which displays a list of response actions when the response action display button 3022 is pressed, and can handle operations to decide on the execution of response actions.

[0136] Here, the anomaly detection information display bar 3021 includes at least the name of the device that detected the anomaly, the image of when the anomaly occurred, the date and time of the anomaly, and the content of the anomaly. Furthermore, to allow for intuitive identification of abnormal situations, the anomaly detection information display bar 3021 preferably uses a display method that emphasizes the anomaly more than during normal operation. In this case, the anomaly detection information display bar 3021 is displayed in a different manner than the message display bar 3012 displayed during normal operation; specifically, the display frame showing the detected anomaly is displayed in red (normally it is, for example, green or black) or flashing (normally it does not flash). Additionally, the character color of the characters displayed in the anomaly detection information display bar 3021 can be set to a different color than during normal operation. Furthermore, in the monitoring screen 302 during anomalies, such as... Figure 7 As shown, the display method of the anomaly detection information display bar 3021 that displays the camera image of the device that was detected to be abnormal is preferably displayed in a position above the camera image display bar 3013 that displays the camera image of the device that was not detected to be abnormal (in other words, in an emphasized display mode).

[0137] Additionally, the response action overview 3023 can display response actions that are action content 1247 associated with the detected abnormal content; the number of displayed actions can be one or more. Furthermore, similarly to the abnormal detection information display bar 3021, the response action overview 3023 can also use red or other colors to indicate an abnormality. Additionally, the monitoring screen 302 during an abnormality includes... Figure 7 In addition to the structure shown, you can also set buttons to retrieve the image when the error occurred and the current image and switch between them.

[0138] The display areas 3011-3013 and 3021-3023 in the monitoring screens 301 and 302 mentioned above are implemented by GUI components, for example.

[0139] By constructing a monitoring screen 300 as described above, it can display situations where no abnormalities are detected under normal conditions, and can intuitively and easily display the detected abnormal status and corresponding response actions when an abnormality occurs. Therefore, it can assist building managers and other customers in their management operations.

[0140] Figure 8 This is an example of a monitoring screen display during an anomaly. Figure 8 The image shows monitoring screen 303 when an anomaly of "smoke detection" is detected in fire-fighting equipment, and monitoring screen 304 when an anomaly of "abnormal behavior (riot)" is detected in an elevator. However, the screen structure of these monitoring screens 303 and 304 follows the pattern shown in the image. Figure 7 The image structure of monitoring screen 302 is described in the description. Furthermore, in... Figure 8 and the following Figure 9 In the middle, in Figure 2 The monitoring screen 300, which displays the camera images of the monitored objects, is described using customer records in customer information 1211 where customer 1211 is “○○ Management Group”, customer ID 1212 is “001”, and building ID 1213 is “Building A”.

[0141] Monitoring screen 303 is an example of a display when an anomaly of "smoke detection" is detected in fire-fighting equipment.

[0142] In monitoring screen 303, the building ID "Building A" and building address "Ibaraki Prefecture □□ City ○○1-2-3" obtained from customer information DB12a are output in building information display bar 3031 (specific values ​​are recorded in...). Figure 2 (Building ID 1213 and building address 1214).

[0143] The anomaly detection information display bar 3032 outputs the building equipment "fire protection equipment" obtained from the anomaly record DB12c (the specific value is recorded in...). Figure 2 The customer information (121), image data, anomaly detection date and time, and the anomaly content "smoke detection" are included. Additionally, in... Figure 7 In the monitoring screen 302 shown during an anomaly, as described above, the anomaly detection information display bar 3032 is preferably set to a different display mode than during normal operation, so that an anomaly can be intuitively identified. Specifically, for example, if the display frame of the camera image of other devices (such as "automatic door" in the figure) that have not detected an anomaly is green, consider setting the display frame of the camera image displayed in the anomaly detection information display bar 3032 to red, etc.

[0144] In the response action list 3033, the values ​​of action number 1246, action content 1247, and contact destination 1248 in the response action record corresponding to the combination of the anomaly category "smoke or fire" and the anomaly content "smoke detected" are output. Specifically, in the response action list 3033, "call the fire truck" is displayed as the action content corresponding to action number "1", and the telephone number "119" is displayed as its contact destination. In addition, "notify via the building's public address system" is displayed as the action content corresponding to action number "2", and a button marked "broadcast executed" is displayed instead of the contact destination. Here, when using a smartphone in the customer operation terminal 3, the customer can make a phone call to the displayed contact destination or execute the building's public address system by clicking "119" or "broadcast executed" displayed in the response action list 3033.

[0145] Monitoring screen 304 is an example of a display when an anomaly (riot) is detected in the elevator. The screen structure of monitoring screen 304 (e.g., building information display bar 3041, anomaly detection information display bar 3042, response action overview 3043) is the same as that of monitoring screen 303. The acquisition of various information output to monitoring screen 304 can be considered in the same way as in monitoring screen 303, therefore, specific explanations are omitted.

[0146] Figure 9 This is another example of a monitoring screen displaying an abnormal situation. Figure 9 The monitoring screen 305 shown is an example of the monitoring screen 300 when an anomaly is detected in multiple devices 23 of the monitored object.

[0147] Specifically, in Figure 9 In monitoring screen 305, anomalies were detected in the "fire detection" of the fire-fighting equipment and the "smoke detection" of the automatic door. Furthermore, in this case, in the response action information 124 registered in response action DB12d, the anomaly content of the fire detection of the fire-fighting equipment has a higher priority 1245 than the anomaly content of the smoke detection of the automatic door.

[0148] At this time, Figure 9In the monitoring screen 305, a first anomaly detection information display panel 3052 is configured below the building information display panel 3051, and a second anomaly detection information display panel 3054 is configured below it. The first anomaly detection information display panel 3052 displays information related to the anomaly with the highest priority among those detected at the current time, while the second anomaly detection information display panel 3054 displays information related to the anomaly with the next lower priority. That is, the first anomaly detection information display panel 3052 displays fire detection in fire-fighting equipment, and the second anomaly detection information display panel 3054 displays smoke detection in automatic doors. Furthermore, below the second anomaly detection information display panel 3054 is a normal information display panel 3055 indicating images of devices 23 (elevators, parking lots, rooftops) where no anomalies were detected.

[0149] In the monitoring screen 305 displayed as described above, for example, if the response action display button corresponding to the first anomaly detection information display bar 3052 is pressed, a list of response actions associated with fire detection in the fire-fighting equipment is displayed in the response action list 3053. The customer can select and execute a response action from options such as "call the fire truck" or "notify via the building's public address system." Similarly, if the response action display button corresponding to the second anomaly detection information display bar 3054 is pressed, a list of response actions associated with smoke detection in the automatic door is displayed in the response action list, and the customer can select the response action to execute.

[0150] As shown in monitoring screen 305, when multiple anomalies are detected, the anomalies with higher importance are displayed in a prominent position (in a conspicuous place) according to the priority of the anomaly content set in the response action information 124. In this way, customers can intuitively identify the anomalies that should be dealt with first and can efficiently execute response actions to the anomalies.

[0151] As explained above, the video surveillance system 100 (video surveillance device 1) according to this embodiment does not require anomaly detection sensors. It can determine anomalies by analyzing the images of the surveillance camera 22 associated with the monitored device 23. By displaying the information of the next action to be taken, i.e., the response action, along with the detected anomaly image, on the user's (customer's) customer operation terminal 3, it can assist the user in responding to detected anomalies while suppressing implementation costs. In addition, at this time, the customer operation terminal 3 provides an intuitive and easy-to-understand display screen in addition to the images of the surveillance camera 22, thereby reducing the management burden on building managers and others.

[0152] Furthermore, the present invention is not limited to the embodiments and modifications described above, but also includes various modifications. For example, the embodiments described above are those that have been explained in detail for ease of understanding of the present invention, and are not limited to having all the structures described. In addition, for a part of the structure of the embodiments, other structures can be added, deleted, or replaced.

[0153] Furthermore, some or all of the aforementioned structures, functions, processing units, and processing modules can be implemented in hardware, for example, through integrated circuit design. Alternatively, the aforementioned structures and functions can be implemented in software by a processor interpreting and executing programs that implement each function. The programs, tables, files, and other information implementing each function can be stored in recording devices such as memory, hard disks, SSDs (Solid State Drives), or recording media such as IC cards, SD cards, and DVDs.

[0154] Additionally, the control lines and information lines shown in the attached diagrams are only those deemed necessary for the explanation and do not necessarily represent all control lines and information lines on the product. In fact, it can be assumed that almost all structures are interconnected.

Claims

1. A video surveillance device that uses images captured by a surveillance camera of a device being monitored to monitor for abnormalities in the device, characterized in that, The video monitoring device has the following features: The camera image acquisition unit acquires image data of the device captured by the surveillance camera; The camera image analysis unit detects anomalies in the image data of the device and determines the content of the detected anomalies. A storage device that stores customer information and response information, wherein the customer information is obtained by associating information related to customers who have signed maintenance contracts for monitoring the device, the device that is the object of monitoring under the maintenance contracts, and the surveillance camera that captures the device; and the response information is obtained by associating the response actions to be taken next in the event of the anomaly detected with the anomaly content of the anomaly. The response action selection unit obtains the response action corresponding to the abnormal content determined by the camera image analysis unit based on the response action information stored in the storage device. as well as The visualization processing unit generates display data for displaying a monitoring screen on a customer operation terminal operated by the customer. Using this display data, the customer operation terminal displays the monitoring screen, which shows an image of the image data acquired by the camera image acquisition unit. When the camera image analysis unit detects an anomaly in the device, the visualization processing unit generates display data for anomaly detection, which includes the display of the image that detected the anomaly and the prompts for the response actions obtained by the response action selection unit in response to the detection of the anomaly. Using the display data for anomaly detection, the monitoring screen is displayed in a manner that allows the operation of deciding the response action to be performed from the customer operation terminal.

2. The video surveillance device according to claim 1, characterized in that, In the response action information, each response action corresponding to one instance of abnormal content can be assigned different action priorities to register multiple response actions. When the multiple response actions in the response action information correspond to the abnormal content determined based on the anomalies detected by the camera image analysis unit. The response action selection unit obtains all matching responses from the plurality of responses. The visualization processing unit generates display data for use in case of an anomaly, so that the multiple response actions obtained by the response action selection unit are displayed on the monitoring screen in a display manner corresponding to the action priority.

3. The video surveillance device according to claim 1, characterized in that, If the camera image analysis unit detects the anomaly, the visualization processing unit performs a predetermined output process to notify the customer operation terminal of the occurrence of the anomaly.

4. The video surveillance device according to claim 1, characterized in that, When the monitoring screen is requested to be started from the customer's operating terminal, the visualization processing unit allows the monitoring screen to be displayed to the customer's operating terminal only if the authentication using the customer information is appropriate.

5. The video surveillance device according to claim 1, characterized in that, When multiple devices are monitored under the maintenance contract, the visualization processing unit generates the display data by displaying the images of each of the multiple devices on the monitoring screen. If the camera image analysis unit detects an anomaly in any of the plurality of devices... The visualization processing unit generates display data for use when the anomaly occurs, so that the image of the device that detected the anomaly is displayed in the monitoring screen in a way that emphasizes the image of the device that did not detect the anomaly more than the image of the device that did not detect the anomaly.

6. The video surveillance device according to claim 5, characterized in that, If the camera image analysis unit detects anomalies in two or more of the multiple devices, it determines the content of each detected anomaly. The response action selection unit, based on the response action information, obtains response actions corresponding to each detected anomaly. The visualization processing unit generates display data for use when the anomaly occurs, so that, according to the detected anomaly, the monitoring screen displays an image of the detected anomaly and prompts the corresponding response actions for the anomaly.

7. The video surveillance device according to claim 5, characterized in that, Each of the multiple anomalous entries registered in the response action information is assigned an anomalous priority. When the camera image analysis unit detects anomalies in two or more of the multiple devices, and determines different anomaly content for each anomaly, The response action selection unit, based on the response action information, obtains response actions corresponding to each detected anomaly. The visualization processing unit generates display data for the anomaly, so that, according to the detected anomaly, the image of the detected anomaly and the prompt for the corresponding response action are displayed on the monitoring screen in a display order corresponding to the anomaly priority assigned to the anomaly content of each anomaly.

8. The video surveillance device according to claim 1, characterized in that, The video monitoring device also has: The response action processing unit executes the specific response action when it determines the execution of a specific response action based on the display data used in the abnormality and the monitoring screen displayed on the customer's operating terminal.

9. A video surveillance system for monitoring anomalies in equipment, characterized in that, have: A surveillance camera that captures images of the device; A camera image recording device that records image data of the device captured by the surveillance camera; A customer operating terminal, operated by a customer who has signed a maintenance contract for monitoring the equipment; and The video monitoring device uses the image data of the device to detect abnormalities in the device, and when an abnormality is detected, it prompts the customer's operating terminal with the appropriate response actions. The video monitoring device has the following features: The camera image acquisition unit acquires image data of the device from the camera image recording device; The camera image analysis unit detects anomalies in the device from the image data and determines the content of the detected anomalies. A storage device that stores customer information and response action information, wherein the customer information is obtained by associating information related to the customer who signed the maintenance contract, the device that is the object of the maintenance contract, and the surveillance camera that captures the device, and the response action information is obtained by associating the response actions to the anomaly with the anomaly content of the anomaly. The response action selection unit obtains the response action corresponding to the abnormal content determined by the camera image analysis unit based on the response action information stored in the storage device. as well as The visualization processing unit generates display data for displaying a monitoring screen on a customer operation terminal operated by the customer. Using this display data, the customer operation terminal displays the monitoring screen, which shows an image of the image data acquired by the camera image acquisition unit. If the camera image analysis unit detects an abnormality in the device. The visualization processing unit generates display data for anomaly detection, including a display of the image showing the detected anomaly and prompts for the corresponding response actions obtained by the response action selection unit in response to the detection of the anomaly. Using this display data, the monitoring screen is displayed in a manner that allows the operation of determining the response action to be performed from the customer's operating terminal. The customer operating terminal determines the response action to be executed from the response actions prompted on the monitoring screen.

10. An image monitoring method for an image monitoring device that uses images captured by a surveillance camera to monitor anomalies of a device being monitored, characterized in that, The video surveillance device stores customer information and response information in a storage device. The customer information is obtained by associating information related to customers who have signed maintenance contracts for monitoring the equipment, the equipment being monitored under the maintenance contracts, and the surveillance camera that captures the equipment. The response information is obtained by associating the response actions to be taken after detecting the anomaly with the anomaly content. The image monitoring method includes the following steps: In the camera image acquisition step, the image monitoring device acquires the image data of the device captured by the surveillance camera; The camera image analysis step involves the image monitoring device detecting anomalies in the image data and determining the content of the detected anomalies. In the response action selection step, the video monitoring device obtains the response action corresponding to the abnormal content determined in the camera image analysis step based on the response action information stored in the storage device. as well as In the visualization processing step, the image monitoring device generates display data for displaying the monitoring screen on a customer operation terminal operated by the customer. This display data is then used to display the monitoring screen on the customer operation terminal. The monitoring screen displays an image of the image data acquired in the camera image acquisition step. If an anomaly is detected in the camera image analysis step, in the visualization processing step, the image monitoring device generates display data for anomaly detection, which includes the display of the image in which the anomaly was detected and the prompt for the response action obtained in the response action selection step. Using the display data for anomaly detection, the monitoring screen is displayed in a manner that allows the operation of deciding the response action to be performed from the customer operation terminal.

Images