Monitoring system, monitoring method, and monitoring program

Abstract

To allow an operator to easily manage or control a plurality of cameras simultaneously through an interface which can be intuitively and efficiently operated.SOLUTION: In a monitoring system including a smartphone which displays video acquired from a plurality of installed monitoring cameras in a consolidated manner, a management server 3 includes: a camera installation information database 320b which stores spatial arrangements of the monitoring cameras, as coordinates; a coordinate conversion unit 314a which converts the coordinates into a two-dimensional planar arrangement, on the basis of the stored coordinates of the monitoring cameras for a monitoring target; and a touch panel which displays video screens captured by the monitoring cameras in accordance with the planar arrangement converted by the coordinate conversion unit 314a, and sequentially switches the displayed video screens by scrolling the video screens in a vertical or lateral direction according to the planar arrangement.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a surveillance system, a surveillance method, and a surveillance program including a surveillance terminal that displays integrated images acquired from multiple imaging means such as surveillance cameras installed in buildings, facilities, and areas such as shopping malls and high-rise apartment buildings. [Background technology]

[0002] In recent years, the evolution of surveillance systems using AI (artificial intelligence) cameras has been remarkable. Advances in video recognition and image analysis technology have made it possible to go one step further than the conventional simple video recording and realize more advanced functions such as detecting abnormal behavior, identifying people, and tracking objects. This AI camera is an advanced surveillance device that uses image recognition and machine learning technology to automatically detect and identify abnormalities and target movements occurring within the area being monitored. Its merits include the ability to process large amounts of video data in real time and perform advanced analysis. In addition, it can identify abnormal behavior, identify people, and perform facial recognition depending on the settings. This can greatly reduce the difficulty of humans performing 24-hour surveillance and improve the efficiency and accuracy of surveillance.

[0003] As an example of a system equipped with such an AI camera, Patent Document 1 discloses a system that centrally manages and controls a large number of surveillance cameras. In this system, an image (reference image) sent from the camera is acquired, and a fixed object area (reference area) corresponding to a fixed object contained therein is set. Then, a real-time camera image (live image) is acquired and displayed on a surveillance screen. A user can switch to a desired screen (for example, an intrusion detection area setting screen) by operating a screen switching tab provided on the basic setting screen. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2023-035448 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the system disclosed in the above-mentioned Patent Document 1 switches to a desired screen (for example, an intrusion detection area setting screen) by operating a screen switching tab when managing and controlling a large number of cameras in a centralized manner. Therefore, in the system disclosed in Patent Document 1, in order to manage a large number of cameras in a centralized manner, it is necessary to understand the spatial arrangement of each camera in the entire monitored area and the relative positional relationship between the cameras, making it difficult to operate intuitively and to grasp the overall situation.

[0006] Therefore, the present invention is intended to solve the above problems, and aims to provide a surveillance system, a surveillance method, and a surveillance program that make it easy for an operator to manage and control multiple cameras simultaneously through an interface that enables intuitive and efficient operation. [Means for solving the problem]

[0007] In order to solve the above problems, the present invention provides a monitoring system including a monitoring terminal that displays images acquired from a plurality of installed image capturing means in an integrated manner, a spatial arrangement storage means for storing the spatial arrangement of each of the imaging means as coordinates; a coordinate conversion unit that converts the coordinates of each imaging means stored in the spatial arrangement storage means into a two-dimensional planar arrangement; a display unit that displays the images captured by the image capturing means in accordance with the planar layout converted by the coordinate conversion unit; a screen switching unit that sequentially switches the video screen displayed on the display unit by scrolling the video screen in a vertical or horizontal direction according to the planar layout; The present invention is characterized by comprising:

[0008] The present invention also provides a monitoring method for displaying images captured by a plurality of image capturing means on a monitoring terminal in an integrated manner, the method comprising: a coordinate conversion step in which a spatial arrangement storage means stores the spatial arrangement of each of the image capturing means as coordinates, and a coordinate conversion unit converts the spatial arrangement of each of the image capturing means stored in the spatial arrangement storage means into a two-dimensional planar arrangement based on the coordinates of each of the image capturing means; a screen switching step in which a display unit displays the image screens captured by the image capturing means in accordance with the planar arrangement converted by the coordinate conversion unit, and a screen switching unit scrolls the image screens displayed on the display unit in an up-down or left-right direction in accordance with the planar arrangement to switch the images sequentially; Includes.

[0009] In the above invention, the display unit is a touch panel having an operation panel superimposed on a display screen, The screen switching unit has a function of detecting a touch operation on the touch panel and determining a direction of the operation on the screen. It is preferred.

[0010] In addition, in the above invention, it is preferable that the screen switching unit groups the multiple imaging means by hierarchical level according to the height position of their arrangement, switches the video screens of each imaging means belonging to the same level by scrolling them horizontally on the display unit, and the video screens belonging to the same level are arranged in a loop so as to circulate horizontally and switch sequentially.

[0011] Furthermore, in the above invention, it is preferable to further comprise vertical direction recognition means for recognizing the vertical direction of the images displayed on the image screens and rotating or inverting the images on each image screen. Effect of the Invention

[0012] As described above, these inventions make it easy for an operator to manage and control multiple cameras simultaneously through a highly user-friendly interface that enables the operator to intuitively understand the spatial arrangement of multiple cameras, and enable rapid response to the detection of abnormal behavior, the identification and tracking of people and places, etc., thereby significantly improving the operability and convenience of the surveillance system. [Brief description of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an overall configuration of a monitoring system according to an embodiment. [Diagram 2] 1 is a block diagram showing the internal configuration of a camera constituting a surveillance system according to an embodiment of the present invention; [Diagram 3] 2 is a block diagram showing an internal configuration of a wireless base station that constitutes the monitoring system according to the embodiment. FIG. [Figure 4] 2 is a block diagram showing an internal configuration of a monitor terminal device constituting the monitoring system according to the embodiment; FIG. [Diagram 5] 2 is a block diagram showing the internal configuration of a management server that constitutes the monitoring system according to the embodiment. FIG. [Figure 6] FIG. 4 is a flowchart showing the operation when installing a surveillance camera in the surveillance system according to the embodiment. [Figure 7] FIG. 4 is a flowchart showing a monitoring operation by a monitoring camera in the monitoring system according to the embodiment. [Figure 8] 4 is a flowchart showing the operation of the monitoring system according to the embodiment when an abnormality is detected by a monitoring camera. [Figure 9] FIG. 13 is a flowchart showing the operation when information about a viewing camera is disclosed. [Figure 10] FIG. 4 is an explanatory diagram showing image processing according to the embodiment. [Figure 11] FIG. 4 is an explanatory diagram showing image processing according to the embodiment. [Figure 12] FIG. 4 is an explanatory diagram showing image processing according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] An embodiment of a monitoring system according to the present invention will be described in detail below with reference to the accompanying drawings. Fig. 1 shows the overall configuration of a monitoring system according to this embodiment. Note that the embodiment shown below is an example of an apparatus for embodying the technical idea of ​​this invention, and the technical idea of ​​this invention does not limit the material, shape, structure, arrangement, etc. of each component to those described below. The technical idea of ​​this invention can be modified in various ways within the scope of the claims.

[0015] (Overview of monitoring services) The surveillance service of this embodiment is a service that monitors specific locations and areas using images, video, audio and other sensor detection data by installing a large number of surveillance cameras, and also provides storage, analysis and management of each recorded data used for surveillance (including video, audio and other sensor detection data).

[0016] The recorded data used for monitoring mainly includes images and videos captured by the surveillance camera 2 as shown in Fig. 2, as well as sound recorded by a built-in audio microphone, and detection data obtained from various sensors such as an infrared sensor, an illuminance sensor, a human sensor, a temperature sensor, and an odor / smoke sensor. The images or videos captured by the surveillance camera 2 are automatically recognized by the image monitoring unit 25d on the surveillance camera 2 side and the image recognition processing unit 314b on the management server 3 side using AI (artificial intelligence) and other technologies to automatically recognize people, actions, or changes in the images and detect abnormalities in the images. Then, incidents such as accidents and incidents are discovered and reported.

[0017] (Overall configuration of the monitoring system) In order to realize the uses and functions described above, the surveillance system according to this embodiment, as shown in Fig. 1, performs surveillance using a number of surveillance cameras 2 that capture images or videos, and is installed facing surveillance targets scattered around the country through a communication network 7 including wireless communication, and is managed and operated by a management server 3. Specifically, the surveillance information management system in this embodiment has the management server 3, a wireless base station 72, the surveillance cameras 2 that can communicate wirelessly or wired via the wireless base station 72 or a communication satellite 73, and a smartphone 1 that is a general-purpose terminal device used by a surveillance person, all arranged on the communication network 7.

[0018] The communication network 7 is a communication network such as a public telephone network or the Internet, and includes a wired network and a wireless communication network, and these networks include a distributed communication network constructed by interconnecting various communication lines (public lines such as telephone lines, ISDN lines, ADSL lines, and optical lines, dedicated lines, and wireless communication networks such as 5G, 4G, LTE, and 3G) using the communication protocol TCP / IP. Note that the communication network 7 also includes intranets (intra-company networks) using 10BASE-T or 100BASE-TX, and LANs and Wi-Fi networks within a specific area (such as a company or home).

[0019] The management server 3 is an application server that is placed on the communication network 7 and remotely manages and controls each surveillance camera 2, including application management, device management, user management, operating status management, surveillance and control management, setting management, remote management, and data collection management for each surveillance camera 2. With these management functions, the management server 3 executes operation control for the surveillance cameras 2 installed in remote locations, issues control notifications in application settings and remote management of surveillance information management applications, and collects data from the surveillance cameras 2.

[0020] Specifically, this management server 3 is a server device operated by the provider of the surveillance information management service, and for the surveillance service, registers in advance the identification information of the owner or manager (installer) of each surveillance camera and the location information regarding the installation location of the surveillance camera, and performs management services for each surveillance information (recorded data, etc.) based on each piece of information transmitted from the surveillance camera 2.

[0021] 5 and 6 show examples of the spatial arrangement of the surveillance cameras 2, and the surveillance device control unit 314 of the management server 3 records the spatial arrangement of each surveillance camera 2 on a coordinate system as installation position information.

[0022] The wireless base station 72 is a device that is connected to the communication network 7 via the relay device 71, establishes a wireless communication connection with the surveillance camera 2, and provides telephone calls and data communication by the surveillance camera 2. Note that the wireless base station 72 in this embodiment functions as a location information acquisition unit that manages which cell each mobile terminal is in on the communication network 7. Specifically, it identifies the location of the surveillance camera 2 based on a location registration request transmitted from the surveillance camera 2. The relay device 71 is a node device such as a modem, terminal adapter, or gateway device for connecting to the communication network 7, and selects a communication path, converts data (signals) between each other, and performs relay processing between the wireless base station 72 and the communication network 7.

[0023] Smartphone 1 is an information processing terminal used by a monitor or the owner / installer of surveillance camera 2, and is equipped with a CPU for calculation processing functions and a communication processing function via a communication interface. It can be realized, for example, by a general-purpose computer such as a personal computer, or a dedicated device with specialized functions, and includes not only smartphones, but also mobile computers, mobile phones, etc.

[0024] (Internal structure of each device) Next, the internal structure of each device constituting the above-mentioned surveillance information management system will be described. Fig. 2 shows the appearance and internal structure of a surveillance camera 2 constituting the surveillance information management system according to this embodiment, Fig. 3 shows the internal structure of a smartphone 5, and Fig. 4 shows the internal structure of a management server 3 according to this embodiment. Note that the term "module" used in the description refers to a functional unit configured by hardware such as a device or equipment, or software having the functions thereof, or a combination of these, for achieving a specified operation.

[0025] (1) Surveillance camera 2 The surveillance camera 2 is an imaging device that optically captures images and videos and has the function of recording audio and sound, and as shown in FIG. 1, it is equipped with an input interface 21 and an output interface 22 as user interface modules. The input interface 21 is an interface to which, in addition to devices for inputting user operations such as operation buttons, images and videos captured by a CCD camera 21a and sound acquired by an audio microphone 21b are input. The input interface 21 also has a built-in acceleration sensor 21c that detects the vertical direction, and various built-in or external sensors such as an infrared sensor, an illuminance sensor, a human sensor, a temperature sensor, and an odor / smoke sensor are connected. The output interface 22 is an interface to which devices for outputting video and sound such as a display and a speaker are connected.

[0026] Furthermore, the surveillance camera 2 includes a location information acquisition unit 23. The location information acquisition unit 23 is a module that measures the current location of the camera itself, and is capable of, for example, calculating the coordinates of the current location based on a GPS signal from a communication satellite 73, and acquiring location information of the camera itself on the network from the base station identifier of the base station with which communication is established by the connection processing unit 27 and radio wave conditions such as the signal strength from this base station.

[0027] The surveillance camera 2 also includes, as communication modules, a wireless interface 26 and a connection processing unit 27. The wireless interface 26 includes a wireless communication function based on a protocol for mobile communication for making calls and data communications, and a wireless communication function based on a protocol for data communications such as a wireless LAN.

[0028] In this embodiment, an identifier is set for each of the wireless base station 72, which is an access point, and the surveillance camera 2, which is a client. Through these identifiers, a surveillance camera within the wireless communication range of the wireless base station 72 is detected, and a link is automatically set to form a wireless LAN hotspot, enabling data communication.

[0029] The connection processing unit 27 is a module for connecting to the wireless base station 72 via wireless communication, and transmits an identifier (telephone number, IP address, user ID) for identifying the device to the wireless base station 72 to be connected, and performs authentication. When establishing a wireless LAN connection with the wireless base station 72, the connection processing unit 27 may monitor wireless signals to detect an identifier for identifying the wireless base station 72, in order to form a link for transmitting and receiving data.

[0030] Furthermore, the surveillance camera 2 includes a control unit 25 and a memory 24 as application execution system modules. The control unit 25 is an arithmetic processing device such as a CPU, and each functional module is virtually constructed by executing various programs on this control unit 25. In this control unit 25, an execution processing unit 25c, an execution condition setting unit 25a, and a monitoring status notifying unit 25b are constructed as application execution system modules.

[0031] The execution processing unit 25c is a module that executes an application related to monitoring information management, connects the monitoring camera 2 and the management server 3, and realizes various functions and services by cooperating with the monitoring camera 2 and the management server 3. The application executed by the execution processing unit 25c includes a video monitoring unit 25d. The video monitoring unit 25d is a module that recognizes changes in people, actions, or events captured in the recorded data captured by the monitoring camera 2 and detects abnormalities captured in the recorded data. In this embodiment, the functions of the video monitoring unit are also provided on the management server 3 side, and depending on the specifications of the monitoring camera 2, all of the functions of the video monitoring unit 25d may be executed on the video monitoring unit 314d side of the management server 3, or the functions may be realized by cooperation between the video monitoring units 25d and 314d.

[0032] In addition, a UI control unit is provided in the application executed by the execution processing unit 25c. This UI control unit is a module that provides a user interface, for example, a web page written in a language such as HTTP, through the connection processing unit 27 and the wireless interface 26. Through this web page, it is possible to set execution conditions and distribute images and videos captured by the camera.

[0033] The monitoring status notification unit 25b is a module that notifies the management server 3, user terminal, etc. of the situation as a monitoring status notification when an abnormality is detected in the monitoring range, such as a sudden change being detected in a captured image or excessive sound being detected. Possible methods of this notification include, for example, sending data to the management server 3 using a predetermined protocol, sending an email containing a predetermined character string, or sending and receiving predetermined information data using a protocol such as HTTP through a web page provided by the UI control unit. This monitoring information status notification includes an identifier (telephone number, terminal ID) for identifying the monitoring camera 2, and can automatically perform authentication processing, and the transmitted monitoring information status can be associated with the monitoring camera and stored in the management server 3.

[0034] The execution condition setting unit 25a is a module that sets conditions (execution conditions) related to the execution of an application and regulates the execution of the application by the execution processing unit 25c in each surveillance camera. The conditions set by the execution condition setting unit 25a include a condition for detecting an abnormality.

[0035] (2) Smartphone 5 3, the smartphone 5 includes, as user interface modules, an input interface 51 and an output interface 52. The input interface 51 is a device for inputting user operations, such as an operation button, a touch panel, or a jog dial.

[0036] The output interface 52 is a device that outputs video and audio, such as a display and a speaker. In particular, the output interface 52 includes a touch panel 52a, and a GUI (Graphical User Interface) is displayed on the display unit of the touch panel 52a by the application execution unit 53. The touch panel 52a is a device in which an operation panel is superimposed on a display screen, and has a function of detecting a touch operation on the display screen and determining the operation direction on the screen.

[0037] The application execution unit 53 is a module that executes applications such as a general OS, browser software, office software such as word processing software, image display software, etc., and is usually realized by a CPU, etc. In this embodiment, by executing the browser software, it is possible to connect to the surveillance camera 2 and management server 3 and display web pages provided by these devices, thereby implementing a function of accepting user operations and transmitting and receiving data.

[0038] The smartphone 5 also includes a communication interface 54. The communication interface 54 is a communication interface for performing data communication, and has a function of performing non-contact communication by radio or the like, and contact (wired) communication by a cable, adapter means, or the like. In this embodiment, the communication interface 54 transmits various requests and authentication information to the management server 3 through the communication network 7, and receives information data such as web pages from the management server 3.

[0039] (3) Management Server 3 The management server 3 is a server device having a user management function, a camera control function, and a content distribution function, and can be realized by a group of servers with multiple functions distributed. This management server 3 includes a web server function, and is capable of distributing content such as web pages that are composed of material data such as HTML (HyperText Markup Language), video, audio, and images in a document system such as WWW (World Wide Web), and is necessary for measuring call connection rates. Specifically, the management server 3 includes a communication module, a camera control module, a user interface module, a group of databases, etc., as shown in Fig. 4.

[0040] The management server 3 includes a communication unit 302 and a connection processing unit 301 as communication modules, and includes a recorded data collection unit 303 and a control unit 310 as user interface modules. The control unit 310 is also connected to a storage unit 320 that records various data, and a clock unit 304 that acquires the current time.

[0041] The communication unit 302 is a communication device that transmits and receives various data to and from the communication network 7 by transmitting and receiving packet data. In this embodiment, the communication unit 302 has a web server function that distributes web pages on the Internet and an e-mail receiving function, and can transmit and receive e-mail to and from the surveillance camera 2 using protocols such as HTTP and SMTP, and can obtain surveillance information status notifications through these protocols. The connection processing unit 301 is a module that establishes wireless communication between the wireless base station 72 and the surveillance camera 2 that is present within a predetermined area under the jurisdiction of the wireless base station 72.

[0042] The storage unit 320 is a database that accumulates information related to the surveillance information management system, and in this embodiment, it is composed of a group of multiple databases. The group of databases may be a single database, or may be a relational database in which the functions are divided into multiple databases and the data are linked to each other by setting mutual relations. In this embodiment, the storage unit 320 includes a user database 320a that holds information about users such as the owner, installer, or manager of the camera, a camera installation information database 320b that includes information about the spatial arrangement of the installed cameras, and a recorded data accumulation unit 320c that accumulates the captured recorded data.

[0043] The user database 320a is a storage device that stores personal information such as the user's name, address, mobile phone number, and email address in association with a user identification number that identifies a user such as a person who installed (or owns) the camera or an administrator. In particular, in this embodiment, the user database 320a stores setting information regarding access authority and privacy control for the smartphone 5 used by the user in association with the user identification information.

[0044] The camera installation information database 320b is a storage device that accumulates the latitude, longitude, altitude, direction (azimuth), address, and other location information of each installed camera. This camera installation information includes location information (including latitude, longitude, altitude, address, building name, position within the building, installation angle, etc., and captured images (still images, videos, thumbnails, etc.)) related to the places where the surveillance cameras 2 are installed (commercial facilities 41, high-rise apartment buildings 42) as shown in Fig. 5 or 6, and information related to the users who installed the surveillance cameras 2 (including user IDs, necessary personal information, access rights, etc.).

[0045] The recorded data storage unit 320c is a storage device that stores images or videos captured by the surveillance camera 2 as recorded data in association with location information of the captured images. The recorded data (videos, still images, audio, and other sensor detection data) stored in the recorded data storage unit 320c can be viewed as the user's own data through a surveillance app on the smartphone 5, by a user whose access right has been confirmed by authentication processing.

[0046] The authentication processing unit 313 is a module that acquires device identification information and installer identification information transmitted from the surveillance camera 2 or smartphone 5, and executes authentication processing based on these identification information. Specifically, the authentication processing unit 313 is a computer that verifies the legitimacy of an accessing person, or software having that function, and acquires installer identification information via the communication network 7 and collates it with the user database 320a to confirm whether the accessing person has the right to access and whether the accessing person is the person himself / herself. Then, the authentication processing unit 313 transmits authentication result information to the interface providing unit 312 or the surveillance device control unit 314.

[0047] The user management unit 311 is a module that accepts user registration for receiving the surveillance information management service by the system and registration of surveillance devices such as surveillance cameras, and records various information related to the registration in the user database 320a. Here, when registering a user, the user management unit 311 registers an ID, which is a user identification number, as an identifier for identifying the installer, and registers a name, address, smartphone / mobile phone number, email address, etc., as user information, and also registers the installer's access authority, authentication information, etc. When registering a surveillance camera, the user management unit 311 registers an ID, which is a user identification number, as an identifier for identifying the installer, and registers a name, address, smartphone / mobile phone number, email address, etc., as installer information, and also registers the installer's access authority, authentication information, etc.

[0048] The recorded data collection unit 303 collects recorded data such as video data and audio data from each monitoring camera, and executes processes such as anomaly detection through the monitoring device control unit 314, while recording the video data in the recorded data storage unit 320c.

[0049] The output data delivery unit 305 is a module that delivers the recorded video data, which is the monitoring data, via the GUI generated by the interface provision unit 312 from the communication unit 302 through the communication network 7 to the smartphones 5 and the like of general users. Specifically, this output data delivery unit 305 is realized by a Web server function having a content delivery function, and is configured with material data such as HTML (HyperText Markup Language), video, audio, and images in a document system such as the WWW (World Wide Web), and is capable of delivering content such as Web pages necessary for measuring call connection rates.

[0050] The control unit 310 is a calculation module configured by hardware such as a processor such as a CPU or a DSP (Digital Signal Processor), a memory, and other electronic circuits, or software such as a program having the functions, or a combination of these, and by appropriately reading and executing a program, various functional modules are virtually constructed, and each constructed functional module controls the operation of each unit and performs various processes for user operations. In this embodiment, the control unit 310 includes a user management unit 311, an authentication processing unit 313, an interface providing unit 312, and a monitoring device control unit 314. The clock unit 304 is a module that constantly acquires the current time, and in this embodiment, acquires the time of authentication processing executed in the authentication processing unit 313, the time of recording or viewing of recorded data, etc. as time information, and transmits the acquired current time to the control unit 310.

[0051] The interface providing unit 312 manages the camera installation information stored in the camera installation information database 320b, constructs a GUI (Graphical User Interface) based on the camera installation information, and allows the user to view recorded data such as videos recorded in the recorded data storage unit 320c through this GUI. The interface providing unit 312 has a recorded data management unit 312a and a GUI control unit 312b. The recorded data management unit 312a is a module that distributes the recorded data stored in the recorded data storage unit 320c to the user through the GUI generated and controlled by the GUI control unit 312b.

[0052] Based on the spatial arrangement information stored in the camera installation information database 320b, the GUI control unit 312b virtually arranges each piece of video data on a two-dimensional plane, makes the video of each camera scrollable up, down, left, and right, and presents it so that it can be viewed via the communication network 7. The two-dimensional arrangement of the video data in the GUI control unit 312b is set by a coordinate conversion process by the coordinate conversion unit 314a.

[0053] The notification unit 314e is a module that notifies the current situation information based on the result of the comparison by the image recognition processing unit 314b. The notification of the current situation information by the notification unit 314e includes outputting a sound or an image from the smartphone 1 or the touch panel 52a of the smartphone 1, and reporting to the smartphone 5 or the management server 3 of a preset resident through the communication network 7.

[0054] Specifically, when the comparison result indicates that there is an unauthorized intruder, the notification unit 314e outputs an alarm sound to the visitor from the smartphone 5 included in the smartphone 1. On the other hand, when the comparison result indicates that there is a legitimate visitor, the notification unit 314e can have a daily conversation with the visitor by using a voice dialogue function. Furthermore, when the comparison result indicates that there is an unauthorized intruder, the notification unit 314e transmits an image taken by the security camera 2 or the like to the management server 3 together with the device ID. In addition, the notification unit 314e notifies the registered smartphone 5 of current status information indicating that an unauthorized intruder has invaded. At this time, when the notification unit 314e performs a comparison using a connection protocol, which is the above-mentioned comparison process, the notification unit 314e also notifies the current status information including the result of the comparison or the success or failure of the connection protocol.

[0055] The monitoring device control unit 314 is a module that manages and controls monitoring devices such as monitoring cameras installed for each monitoring target. Specifically, the monitoring device control unit 314 accesses monitoring devices such as the monitoring camera 2, sets the operating conditions, etc., monitors the operating status, and collects and accumulates data captured, recorded, and detected by each monitoring device. In this embodiment, the monitoring device control unit 314 includes a coordinate conversion unit 314a, an image recognition processing unit 314b, a synchronization processing unit 314c, a video monitoring unit 314d, and an alarm unit 314e.

[0056] The coordinate conversion unit 314a is a module that converts the spatial arrangement of each imaging means into a two-dimensional planar arrangement based on the coordinates of the spatial arrangement of each surveillance camera 2 stored in the camera installation information database 320b, which is a spatial arrangement storage means. The coordinate conversion process in the coordinate conversion unit 314a integrates the video screens captured by the surveillance cameras 2 and arranges the images in a planar manner, and displays the video screens according to this planar arrangement on a liquid crystal panel superimposed on the touch panel 52a, which is a display unit. In addition, the operation panel superimposed on the touch panel 52a, which is a screen switching unit, detects a user operation such as a swipe, which is a contact operation on the touch panel, and scrolls up and down or left and right according to the planar arrangement, thereby switching the video screens sequentially.

[0057] The image recognition processing unit 314b is a module that automatically identifies specific objects or patterns in the video. For example, it is possible to identify and identify images of people, vehicles, or specific actions, and automatically issue an alert for specific suspicious actions or abnormal situations through the notification unit 314e. In addition, the image recognition processing unit 314b also has a vertical direction recognition function that recognizes the vertical direction of the image displayed on the video screen and rotates or flips the image on each video screen.

[0058] The synchronization processing unit 314c is a module that synchronizes the operation of the user interface on the smartphone 5 side and the settings made by the user with the interface control processing on the management server 3 side. The synchronization processing unit 314c causes all operations and setting changes made by the user through the user interface of the smartphone 5 to be reflected in the management server 3. Specifically, when the user changes the display method of the video of the surveillance camera 2, the information is transmitted to the management server 3 by the synchronization processing unit 314c, and the management server 3 performs appropriate video processing based on the information, thereby enabling the video display desired by the user. In addition, setting changes such as when the user adds a new surveillance camera 2 to the system or changes the position of an existing surveillance camera 2 are also reflected in the management server 3 in the same manner.

[0059] The video monitoring unit 314d is a module that recognizes changes in people, actions, or events captured in the recorded data captured by the surveillance camera 2, and detects abnormalities captured in the recorded data. In this embodiment, the functions of this video monitoring unit are also provided on the surveillance camera 2 side, and depending on the specifications of the surveillance camera 2, all of the functions of this video monitoring unit 314d may be executed on the video monitoring unit 25d side of the surveillance camera 2, or the functions may be realized by cooperation between the video monitoring units 25d and 314d.

[0060] The video monitoring units 25d and 314d are equipped with AI (artificial intelligence), which can automatically perform recognition processing and can improve recognition accuracy by machine learning such as deep learning. The basic functions of the AI ​​equipped in these video monitoring units 25d and 314d include a watch function and a security function that continuously monitors and analyzes images to detect abnormalities, a function that reports and alerts when an abnormality is detected or periodically, a counting function that counts people and objects, a function that identifies men and women, ordinary cars, light cars, bicycles, etc., and a recognition function that recognizes specific individuals, car types, etc. Other basic functions include a motion sensor function that detects specific movements, and a sensor sensitivity function for temperature, humidity, brightness, etc.

[0061] Furthermore, the AI ​​equipped in the video monitoring units 25d, 314d has a statistical analysis function based on the collected big data, and this statistical analysis function includes a function for visualizing and graphing the results of the statistical analysis, a calculation function, a machine learning function that uses the model obtained as a result of the statistical analysis as training data, and a behavior prediction function and an outcome prediction function based on the generated model.

[0062] (Operations of the image recognition processing unit 314b and the coordinate conversion unit 314a) Here, the processing operations of the image recognition processing unit 314b and the coordinate conversion unit 314a will be described in detail. (1) Processing operation in the image recognition processing unit 414b In this embodiment, the vertical direction recognition function will be described when the surveillance camera 2 is equipped with an acceleration sensor 21c. The video monitoring unit 25d directly measures the direction and inclination of the surveillance camera 2 using the detection result of the acceleration sensor 21c. This allows the display direction of the image to be accurately corrected according to each situation, such as when the surveillance camera 2 is installed facing upright as shown in FIG. 8, when it is installed 180 degrees upside down as shown in FIG. 9, or when it is installed tilted at 90 degrees as shown in FIG. 10 or FIG. 11. To perform these processes, the image recognition processing unit 314b uses an advanced image recognition algorithm. This allows the camera direction to be estimated from the direction, position, or shape of an object in the video, and the video to be adjusted.

[0063] Specifically, as shown in FIG. 8(a), when the surveillance camera 2 is installed facing up and down correctly, the image recognition processing unit 314b recognizes the vertical direction of the image based on the data from the acceleration sensor 21c, and the image is displayed on the display unit in the same orientation as shown in FIG. 8(b). Also, when the surveillance camera 2 is installed tilted at 90 degrees as shown in FIG. 10(a), the acceleration sensor 21c detects the tilt, the image recognition processing unit 314b recognizes the vertical direction of the image, and issues a command to rotate the image 90 degrees (left or right). As a result, as shown in FIG. 9(c), the image is displayed on the display unit in the rotated state. Furthermore, as shown in FIG. 9(a), when the surveillance camera 2 is installed 180 degrees upside down, the acceleration sensor 21c detects the upside down state, the image recognition processing unit 314b recognizes the vertical direction of the image, and issues a command to rotate the image 180 degrees as shown in FIG. 9(c). As a result, the image is displayed on the display unit in the inverted state.

[0064] In the case of a surveillance camera that does not have an acceleration sensor, the image recognition processing unit 314b recognizes the vertical direction based on the image captured by the surveillance camera 2, and displays the image on the display unit as is. When the surveillance camera is installed tilted by 90 degrees, the image recognition processing unit 314b recognizes the vertical direction of the image based on a specific feature contained in the image, such as the tilt of an object that is affected by gravity, and rotates the image by 90 degrees (left or right). When the surveillance camera is installed upside down by 180 degrees: the image recognition processing unit 314b recognizes the vertical direction of the image based on a specific feature contained in the image, such as an object being shown upside down, and issues a command to flip the image 180 degrees, and the image is displayed on the display unit in the flipped state.

[0065] (2) Processing operations in the coordinate conversion unit 314a The coordinate conversion unit 314a has a function of converting the spatial arrangement of the imaging means into two-dimensional coordinates based on the coordinates of the spatial arrangement of each surveillance camera 2, thereby integrating the captured image screens and displaying them on a plane.

[0066] As shown in FIG. 5(a), when multiple surveillance cameras 2 (201-209) are arranged on a complicated route such as floor A1 of a shopping mall, the coordinate conversion unit 314a acquires the coordinates of the spatial arrangement of each surveillance camera 2 stored in the camera installation information database 320b. These coordinates indicate the position on the floor map of each floor of the shopping mall, and reflect where each surveillance camera 2 (201-209) is arranged and in which direction it faces. Based on these acquired coordinates, the coordinate conversion unit 314a converts the spatial arrangement of each surveillance camera 2 (201-209) into a two-dimensional planar arrangement as shown in FIG. 5(b), and the video screens captured by the surveillance cameras 2 (201-209) scattered on floor A1 of the shopping mall are displayed in an integrated manner on the planar map. Here, the video screens of each surveillance camera 201-209 are arranged horizontally so as to circulate horizontally in the order of the route of the surveillance cameras 201-209 arranged on the route of floor A1.

[0067] On the other hand, when multiple surveillance cameras 2 are arranged in the corridors of each floor of the high-rise apartment A2 as shown in FIG. 6(a), the coordinate conversion unit 314a acquires the coordinates of the spatial arrangement of each surveillance camera 2 from the camera installation information database 320b. These coordinates indicate the floors of each floor in the high-rise apartment A2, and reflect the position of each surveillance camera 2 on which floor and in which direction it is facing. Based on these acquired coordinates, the coordinate conversion unit 314a converts the spatial arrangement of each imaging means into a two-dimensional planar arrangement as shown in FIG. 6(b). As a result, the video screens are arranged in the vertical direction so that the transition between each floor of the high-rise apartment becomes a vertical scroll, and the video screens are arranged in the horizontal direction so that the transition between surveillance cameras arranged on the same floor becomes a horizontal scroll, and the video screens taken from the surveillance cameras 2 scattered on each floor of the high-rise apartment A2 are displayed in an integrated manner for each floor on a planar map.

[0068] The spatial arrangement of each surveillance camera can be set arbitrarily by the user. In this case, the user directly inputs the coordinates of the spatial arrangement of each surveillance camera 2 to the system through an application running on the smartphone 5, and records them in the camera installation information database 320b on the management server 3 side. For example, the position and orientation of the camera are recorded when the surveillance camera 2 is physically installed, and are manually input into the application. Then, the coordinate conversion unit 314a converts the spatial arrangement of each surveillance camera 2 into a two-dimensional planar arrangement based on these manually input coordinates.

[0069] Another method for setting the spatial arrangement of the surveillance cameras is to use the radio wave strength of the wireless communication by the surveillance cameras 2. In this method, the relative positions of the surveillance cameras 2 are estimated based on the measurement of the radio wave strength between the wireless devices communicating with each other. Specifically, by utilizing the characteristic that the strength of a wireless signal attenuates with distance, the strength of the wireless signals transmitted by each surveillance camera 2 to each other is measured, and the relative distance between the cameras is estimated based on the strength. Based on the estimation result, the coordinate conversion unit 314a converts the spatial arrangement of the surveillance cameras 2 into a two-dimensional planar arrangement.

[0070] Another method for setting the spatial arrangement of the surveillance cameras is to use the image recognition results by the image recognition processing unit 414b. The image recognition results by the image recognition processing unit 414b are used to estimate the relative positions and spatial arrangement of the multiple surveillance cameras 2. Specifically, as shown in Fig. 12, the image recognition processing unit 414b has a function of automatically identifying a specific object T2 or pattern in the video captured from each surveillance camera 2, and this identification result is used to estimate the relative positions and spatial arrangement of each surveillance camera 2.

[0071] Specifically, when the same object or pattern is identified in the video of each surveillance camera 2, the relative arrangement of each camera is estimated based on the position and direction of the object. For example, when two cameras are shooting the same object T2 from different angles, the angle and distance between each camera are estimated based on the image of the object. In this way, by utilizing the identification result of the image recognition processing unit 414b, the coordinate conversion unit 314a can more precisely estimate the specific spatial arrangement of each surveillance camera 2 and convert it into a more accurate two-dimensional planar arrangement.

[0072] (Monitoring information management method) A monitoring information management method can be implemented by operating a monitoring information management system having the above configuration. Fig. 7 is a flow chart showing the monitoring information management method according to this embodiment. Note that the processing procedure described below is merely an example, and each process may be modified as much as possible. Also, steps may be omitted, replaced, or added to the processing procedure described below as appropriate depending on the embodiment.

[0073] First, the surveillance camera 2 is installed at a predetermined location in the location to be monitored (A1 or A2) (S101). After installation, the connection processor 27 establishes a communication connection with the management server 3 used by the user (monitor) (S102 and S103). This communication establishment places the new surveillance camera 2 within a common Wi-Fi network. Next, using this established communication, the smartphone 5 starts recording and sending the surveillance records to the management server 3, and the management server 3 stores the received surveillance records (S105 and S104).

[0074] Meanwhile, the user accesses the management server 3 from the smartphone 5, and transmits user identification information such as a user ID as an administrator from the smartphone 5 to the management server 3 (S106). On the management server 3 side that receives this user identification information (S107), the authentication processing unit 313 performs authentication processing using the received user identification information (S108 and S109), and if the authentication is successful, the user management unit 311 permits access from the smartphone 5 to the management server 3 side.

[0075] Next, the installation location information is transmitted from the surveillance camera 2 to the management server 3 (S110). Methods for determining this installation location information include a method in which the surveillance camera 2 acquires the location information itself, and a method in which the location information of the surveillance camera 2 detected by the smartphone 5 or the management server 3 is used. Then, while the surveillance operation (S111) continues, the surveillance status notification unit 25b of the surveillance camera 2 sequentially transmits recorded data to the management server 3, and on the management server 3 side that receives this, the surveillance device control unit 314 accumulates the recorded data received by the recorded data collection unit 303 in the recorded data accumulation unit 320c.

[0076] In addition, in the management server 3, the interface providing unit 312 constructs a GUI for viewing the recorded data (S111a). In constructing this GUI, the user settings are referenced (S111b) and a coordinate conversion process is performed (S111a). This generated GUI is distributed to the user from the output data distribution unit 305 and the communication unit 302 via the communication network 7 (S207).

[0077] In this monitoring operation (S111), an abnormality detection process is executed in the video monitoring unit 25d of the monitoring camera 2 or the video monitoring unit 314d of the management server 3. Specifically, normal imaging process is continued until an abnormality is detected, and when an abnormality is detected, an abnormality occurrence notification is generated and transmitted to the management server 3 or the smartphone 5 via the monitoring status notification unit 25b or the interface providing unit 312. This abnormality occurrence notification includes a notification that an abnormality has occurred, its type, and image data or audio data files that serve as the basis for the abnormality (or address information such as a URL indicating the storage location of these data files). The smartphone 5 that receives this abnormality occurrence notification executes an alert display / output process, and the management server 3 accumulates information regarding the abnormality that has occurred.

[0078] In addition, in the monitoring operation (S111), the image recognition processing unit 314b performs analysis of the images transmitted from each monitoring camera 2, and recognizes the characteristics of objects and scenes present in the images. If an acceleration sensor is built into the camera, the vertical direction of the image is identified based on information from the sensor, and the image is rotated or inverted as necessary. Furthermore, even if an acceleration sensor is not installed, the vertical direction of the image is estimated based on the image characteristics, and appropriate image processing is performed. Meanwhile, the coordinate conversion unit 314a refers to the spatial arrangement coordinates of each monitoring camera 2 stored in the camera installation information database 320b (S111b), and converts these into an arrangement on a two-dimensional plane (S111a). This constructs a GUI that displays images from multiple cameras in an integrated manner on a single display surface. At this time, the spatial arrangement of the monitoring cameras 2 is arbitrarily set by the user, or is estimated based on the radio wave intensity of the wireless communication between the cameras.

[0079] Through the above series of operations, the surveillance camera system performs an integrated analysis of the images from multiple cameras, arranges them on a GUI that combines them on a single two-dimensional plane, and distributes this as output data (S111c) and makes it possible to display it (S111d). This enables the user to efficiently understand the images from each surveillance camera 2 and take appropriate action.

[0080] (Monitoring Information Management Program) The above-mentioned monitoring information management system and monitoring information management method according to the present invention can be realized by executing a monitoring information management program of the present invention written in a specific language on a computer. That is, by installing the program according to this embodiment in an IC chip or memory device of a general-purpose computer such as a portable terminal device, a smartphone, a wearable terminal, a mobile PC or other information processing terminal, a personal computer or a server computer, and executing it on a CPU, a monitoring information management system having the above-mentioned functions can be constructed and the monitoring information management method according to the present invention can be implemented.

[0081] Furthermore, the program of the present invention can display images captured by a plurality of image capturing means in an integrated manner on a monitoring terminal. This program has a spatial arrangement storage means for storing the spatial arrangement as coordinates, and can convert the spatial arrangement into a two-dimensional planar arrangement using a coordinate conversion unit. After that, the display unit can display the image screens captured by each image capturing means. Also, the screen switching unit can be used to scroll the displayed image screens up and down or left and right according to the planar arrangement and sequentially switch between them.

[0082] In particular, the surveillance program according to this embodiment can determine the operation direction on the screen using a touch panel on which the operation panel is superimposed. Furthermore, the screen switching unit can group multiple imaging means according to their height positions, and can cycle through video screens belonging to the same layer in the horizontal direction to switch between them in sequence. These functions enable more intuitive and efficient operation, making it easier to manage and control the surveillance system.

[0083] Furthermore, the monitoring program according to this embodiment not only manages the monitoring information, but can also effectively utilize it. For example, the monitoring information can be used to detect and deal with fraudulent acts and problems at an early stage. The monitoring information can also be used to improve business processes, such as improving production lines and enhancing quality control. In general, the monitoring program according to this embodiment can be executed on various devices, can be intuitively and efficiently operated, and can easily collect, analyze, and visualize monitoring information, thereby contributing to the management and control of the monitoring system and the improvement of business processes.

[0084] The surveillance program according to the present embodiment also provides various functions for managing surveillance information. For example, the information processing terminal can be used to recognize the vertical direction of the image displayed on the video screen, and rotate or invert the image on each video screen. Also, based on the coordinates of each surveillance camera stored in the surveillance target, a coordinate conversion unit can be used to convert the coordinates into a two-dimensional planar arrangement. After that, the display unit can be used to display the video screens captured by each surveillance camera. In this way, the present program can effectively manage surveillance information.

[0085] Furthermore, the program of the present invention can determine the operation direction on the screen using a touch panel on which the operation panel is superimposed. For example, the screen switching unit has a function of detecting a touch operation on the touch panel and determining the operation direction on the screen. Also, the screen switching unit can group a plurality of imaging means according to their height positions, and can cycle through video screens belonging to the same layer in the horizontal direction and switch them sequentially. These functions enable more intuitive and efficient operation, making it easier to manage and control the surveillance system.

[0086] The monitoring information management program of the present invention can be distributed, for example, through a communication line, and can be transferred as a package application that runs on a stand-alone computer by recording it on a computer-readable recording medium. Specifically, the recording medium can be a magnetic recording medium such as a flexible disk or cassette tape, an optical disk such as a CD-ROM or DVD-ROM, or a RAM card. The computer-readable recording medium on which the program is recorded makes it possible to easily implement the above-mentioned system and method using a general-purpose computer or a dedicated computer, and also makes it easy to store, transport, and install the program.

[0087] (Action and effect) According to this embodiment, a highly operable interface that enables an operator to intuitively understand the spatial arrangement of multiple cameras makes it easy for the operator to manage and control multiple cameras simultaneously, and enables rapid response to the detection of abnormal behavior, the identification and tracking of people and places, and the like, thereby significantly improving the operability and convenience of the surveillance system.

[0088] The present invention is not limited to the above-described embodiment, and the components can be modified without departing from the spirit of the invention. Various inventions can be formed by appropriately combining the components disclosed in the above-described embodiment. For example, some components may be deleted from all the components shown in the embodiment. [Explanation of symbols]

[0089] A1…Shopping mall floor A2: High-rise apartment building T2…Object 1. Smartphone 2. Surveillance camera 3. Management Server 4. Target of surveillance 5. Smartphone 7. Communication Network 21...Input interface 21a…CCD camera 21b…Audio microphone 21c…Accelerometer 22...Output interface 23…Location information acquisition unit 24…Memory 25...Control section 25a…Execution condition setting section 25b…Monitoring status notification section 25c...execution processing section 25d…Video surveillance section 26...Wireless interface 27...Connection processing unit 41…Commercial facilities 42…High-rise apartment building 51...Input interface 52...Output interface 52a…Touch panel 53…Application execution unit 54…Communication interface 71...Relay device 72...Radio base station 73…Communications satellite 201~209…Surveillance camera 301...Connection processing unit 302…Communications Department 303…Recording data collection section 304…Timekeeping section 305…Output data distribution unit 310...Control unit 311...User Management Department 312…Interface provision department 312a…Recorded data management section 312b...GUI control section 313...Authentication processing unit 314...Monitoring device control section 314a... Coordinate conversion unit 314b...Image recognition processing unit 314c...Synchronization processing section 314d…Video Surveillance Department 314e…Information Department 320...Storage section 320a...User database 320b…Camera installation information database 320c…recorded data storage unit 414b...Image recognition processing unit

Claims

1. A monitoring system including a monitoring terminal that displays images acquired from a plurality of installed imaging means in an integrated manner, a spatial arrangement storage means for storing the spatial arrangement of each of the imaging means as coordinates; a coordinate conversion unit that converts the coordinates of each imaging means stored in the spatial arrangement storage means into a two-dimensional planar arrangement; a display unit that displays the images captured by the image capturing means in accordance with the planar layout converted by the coordinate conversion unit; a screen switching unit that sequentially switches the video screen displayed on the display unit by scrolling the video screen in a vertical or horizontal direction according to the planar layout; A monitoring system comprising:

2. the display unit is a touch panel having an operation panel superimposed on a display screen, The screen switching unit has a function of detecting a touch operation on the touch panel and determining a direction of the operation on the screen.

2. The monitoring system according to claim 1 .

3. The surveillance system described in claim 1, characterized in that the screen switching unit groups the multiple imaging means by level according to the height position of their arrangement, switches the video screens of each imaging means belonging to the same level by scrolling them horizontally on the display unit, and the video screens belonging to the same level are arranged in a loop so that they are circulated horizontally and switched sequentially.

4. 2. The monitoring system according to claim 1, further comprising vertical direction recognition means for recognizing a vertical direction of the images displayed on said video screens and rotating or inverting the images on each video screen.

5. A monitoring method for displaying images captured by a plurality of imaging means on a monitoring terminal in an integrated manner, comprising: a coordinate conversion step in which a spatial arrangement storage means stores the spatial arrangement of each of the image capturing means as coordinates, and a coordinate conversion unit converts the spatial arrangement of each of the image capturing means stored in the spatial arrangement storage means into a two-dimensional planar arrangement based on the coordinates of each of the image capturing means; a screen switching step in which a display unit displays the image screens photographed by the image capturing means in accordance with the planar arrangement converted by the coordinate conversion unit, and a screen switching unit scrolls the image screens displayed on the display unit in an up-down or left-right direction in accordance with the planar arrangement to switch the images sequentially; A monitoring method comprising:

6. the display unit is a touch panel having an operation panel superimposed on a display screen, In the screen switching step, the screen switching unit detects a touch operation on the touch panel and determines a direction of the operation on the screen.

6. The monitoring method according to claim 5.

7. The monitoring method described in claim 5, characterized in that in the screen switching step, the screen switching unit groups the multiple imaging means by hierarchy according to the height position of their arrangement, switches the video screens of each imaging means belonging to the same hierarchy by scrolling them horizontally on the display unit, and loops the video screens belonging to the same hierarchy so that they are circulated horizontally and switched sequentially.

8. The monitoring method according to claim 5, further comprising a vertical direction recognition step in which, in the coordinate conversion step, a vertical direction recognition means recognizes the vertical direction of the images displayed on the video screens and rotates or inverts the images on each video screen.

9. A surveillance program for displaying images acquired from a plurality of installed image capturing means in an integrated manner, comprising: a spatial arrangement storage means for storing the spatial arrangement of each of the imaging means as coordinates; a coordinate conversion unit that converts the coordinates of each imaging means stored in the spatial arrangement storage means into a two-dimensional planar arrangement; a display unit that displays the images captured by the image capturing means in accordance with the planar layout converted by the coordinate conversion unit; a screen switching section for scrolling the video screen displayed on the display section in a vertical or horizontal direction in accordance with the planar arrangement and successively switching the video screen; A monitoring program that functions as a

10. the display unit is a touch panel having an operation panel superimposed on a display screen, The screen switching unit has a function of detecting a touch operation on the touch panel and determining a direction of the operation on the screen.

10. The monitoring program according to claim 9.

11. The surveillance program of claim 9, characterized in that the screen switching unit groups the multiple imaging means by level according to the height position of their arrangement, switches the video screens of each imaging means belonging to the same level by scrolling them horizontally on the display unit, and is arranged in a loop so as to circulate the video screens belonging to the same level horizontally and switch them sequentially.

12. The information processing terminal, 10. The monitoring program according to claim 9, further functioning as vertical direction recognition means for recognizing a vertical direction of the image displayed on said image screen and rotating or inverting the image on each image screen.

Images