Monitoring system, radar device, and method for controlling monitoring system

Abstract

This monitoring system: acquires radar detection information from a radar device and acquires camera detection information from a camera, the camera detection information including video in which an imaging object appears and an imaging condition; determines, on the basis of imaging condition information and positional coordinates at which a radar detection object included in the radar detection information is detected, whether the radar detection object and the imaging detection object are the same; associates the radar detection object with the imaging object when it is determined that the radar detection object and the imaging object are the same; displays, on a monitor upon receiving an operation for designating a past time, the position of the radar detection object at the designated time by superimposing said position on map information indicating a monitoring object region; and displays video in which the imaging object determined to be the same as the radar detection object at the designated time on the monitor.

Description

Surveillance system, radar device, and method for controlling surveillance system

[0001] The present invention relates to a surveillance system, a radar device, and a method for controlling a surveillance system.

[0002] Patent Document 1 discloses the following: "A building radar camera system including a camera and a radar system, wherein one or more images captured by the camera include first positions within the one or more images of one or more points on a world plane, and the radar system is configured to capture radar data indicating second positions on the world plane of the one or more points. The building radar camera system further includes one or more processing circuits configured to receive correspondences between the first and second positions of the one or more points and generate a sphere-to-plane homography, the sphere-to-plane homography being configured to translate between the points captured by the camera modeled on a unit sphere and the world plane based on the correspondences between the first and second positions, and to translate the points captured by the camera or one or more additional points captured by the radar system between the unit sphere and the world plane based on the sphere-to-plane homography." (Abstract excerpt)

[0003] U.S. Patent No. 11061132

[0004] According to the building radar camera system of Patent Document 1, images captured by the camera and radar data can be aligned within the world plane.

[0005] However, when a user plays back images from a camera going back in time, although they can view past footage, there is a problem in that it is difficult for the user to grasp the location on the map or movement trajectory of objects captured in the past footage at the time the image was captured.

[0006] The present invention has been made to solve the above-mentioned problems, and aims to provide a surveillance system, a radar device, and a method for controlling a surveillance system that combine radar and a camera and are suitable for verifying the movements of suspicious individuals at any point in the past.

[0007] In order to achieve the above object, the present invention employs the configurations described in the claims. As an example, the present invention provides a surveillance system comprising: a radar device that irradiates radio waves to an object in a surveillance area, detects the object based on the radio wave reflected by the object, and outputs radar detection information; a first camera that captures an image of the object and outputs camera detection information; a memory that stores map information indicating the surveillance area; a processor; and a monitor, wherein the radar detection information includes radar-detected object identification information that identifies the radar-detected object detected by the radar device, a detection time, and a position coordinate at which the radar-detected object was detected; and the camera detection information includes image information generated by the first camera and an image of the object captured in the image information. The processor determines whether the radar-detected object and the imaged object are the same based on the position coordinates at which the radar-detected object was detected and the imaged object information, and if it determines that they are the same, links the radar-detected object identification information with the camera detection information. When an operation to specify a past time is accepted, the processor causes the position of the radar-detected object at the specified time to be superimposed on the map information and displayed on the monitor, and causes the monitor to display the video information of the imaged object that was determined to be the same as the radar-detected object at the specified time.

[0008] The present invention also provides a radar device connectable to a monitor via a network, the radar device irradiating an object in a monitored area with radio waves, detecting the object based on the radio wave reflected by the object, and outputting radar detection information, a memory for storing map information indicating the monitored area, a processor, and a communication interface for establishing a communication connection with a camera, the radar detection information including radar-detected object identification information for identifying the radar-detected object detected by the radar device, the detection time, and the position coordinates at which the radar-detected object was detected, the radar device receiving camera detection information output by the camera after capturing an image of the object, the camera detection information being video information generated by the camera. and imaged object identification information that identifies the imaged object captured in the video information, the image capture time, and image capture condition information for specifying the position and field of view of the camera. The processor determines whether the radar-detected object and the imaged object are the same based on the position coordinates at which the radar-detected object was detected and the image capture condition information, and if it determines that they are the same, links the radar-detected object identification information with the camera detection information. When an operation to specify a past time is accepted, the processor causes the position of the radar-detected object at the specified time to be superimposed on the map information and displayed on the monitor, and the video information of the imaged object determined to be the same as the radar-detected object at the specified time to be displayed on the monitor.

[0009] The present invention also provides a method for controlling a monitoring system, the method including the steps of: acquiring radar detection information output by a radar device after irradiating an object in a monitoring area with radio waves and detecting the object based on the radio wave reflected by the object, the radar detection information including radar-detected object identification information for identifying the radar-detected object detected by the radar device, a detection time, and position coordinates at which the radar-detected object was detected; and acquiring camera detection information output by a camera after capturing an image of the object, the camera including image-captured object identification information for identifying video information generated by the camera and the image-captured object captured in the video information, the image-capture time, and the position and field of view of the camera. the step of acquiring the camera detection information including imaging condition information for determining whether the radar-detected object and the imaged object are the same based on the position coordinates at which the radar-detected object is detected and the imaging condition information, and if it is determined that they are the same, linking the radar-detected object identification information with the camera detection information; and the step of receiving an operation to specify a past time, superimposing the position of the radar-detected object at the specified time on map information indicating the area to be monitored and displaying on the monitor the video information of the imaged object determined to be the same as the radar-detected object at the specified time.

[0010] According to the present invention, it is possible to provide a surveillance system, a radar device, and a method for controlling a surveillance system that are suitable for verifying the movements of a suspicious person at any time in the past by combining a radar and a camera. Objects, configurations, and effects other than those described above will be clarified in the following embodiments.

[0011] 1 is a diagram showing an example of the system configuration of a monitoring system according to a first embodiment. FIG. 1 shows a UI screen example 1 displayed on the monitoring system. FIG. 2 shows a UI screen example 2 displayed on the monitoring system. FIG. 3 shows a UI screen example 3 displayed on the monitoring system. Functional block diagram of a radar. Functional block diagram of a PTZ camera. Functional block diagram of a server. Flowchart showing the flow of initial setup processing. Flowchart showing the flow of normal operation processing 1. Flowchart showing the flow of normal operation processing 2. Flowchart showing the flow of real-time display and recording processing to a recorder. Explanatory diagram of ID synthesis processing. Explanatory diagram of handover processing. Explanatory diagram of handover processing. Explanatory diagram of handover processing. Explanatory diagram of handover processing. Sequence diagram of handover (when handover is possible). Sequence diagram of handover (when handover is not possible). Diagram showing an example data structure of radar detection information. Diagram showing an example data structure of camera imaging condition information. Diagram showing an example data structure of point cloud trajectory information. Flowchart showing the flow of video playback processing. Diagram showing a playback screen image. Diagram showing an example operation screen during playback. Diagram showing an example event search screen 1. Diagram showing an example event search screen 2. Functional block diagram of a monitoring system according to a second embodiment.

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In all the drawings for explaining the embodiment, the same components are generally designated by the same reference numerals, and repeated description thereof will be omitted.

[0013] First Embodiment FIG. 1 is a diagram showing an example of the system configuration of a monitoring system 100 according to a first embodiment.

[0014] The monitoring system 100 shown in FIG. 1 is configured by at least one radar device 10 and at least one camera 20, each of which is communicatively connected to a server 30 and a recorder 40 via a network 90.

[0015] 1 correspond to the radar device 10. Also, each of the PTZ cameras 1, 2, ..., M (PTZ is an abbreviation of the initials of Panoramac Tilt Zoom) and the fixed cameras, ..., O, correspond to the camera 20, T. The numbers of radars and cameras are merely examples, and there should be at least one of each.

[0016] The radar device 10 is a device that irradiates radio waves to objects in a monitored area, detects the objects based on the radio waves reflected by the objects, and outputs radar detection information. In this embodiment, a millimeter-wave radar is used as the radar device 10, but microwave radar, infrared radar, or LiDAR (Light Detection and Ranging) may also be used.

[0017] Radar detection information is a general term for information output by the radar device 10, and includes, for example, radar detection object identification information that identifies the radar detection object detected by the radar device 10, detection time information, and the position coordinates (which may be point cloud coordinate information) at which the radar detection object was detected.

[0018] Furthermore, when the radar device 10 implements a so-called AI function that uses a trained model to determine the type of object (hereinafter referred to as "radar-detected object") detected by the radar device 10, the radar detection information may include attribute information indicating the type (label) of the radar-detected object. Furthermore, when the radar device 10 implements a function that analyzes the speed from changes in the coordinates of the detection point within a scan cycle of the radar-detected object, the radar detection information may include attribute information indicating the moving speed.

[0019] The camera 20 is a device that captures an object and outputs camera detection information, and may be a visible light camera or an infrared camera. A PTZ camera, used as an example of the camera 20, can remotely control the rotation of the PTZ camera's housing in the panoramic and / or tilt directions, as well as the zoom magnification of the imaging unit. The above rotation control and zoom control, which changes the zoom magnification, are collectively referred to as PTZ control. A PTZ camera is a suitable example of a camera used in the monitoring system 100. However, cameras that only allow rotation control but do not have a zoom function, or fixed cameras that do not have a rotation function and only allow the zoom magnification to be changed, may also be used. A camera equipped with an ultra-high-resolution fisheye lens and a zoom function may also be used as the fixed camera.

[0020] Camera detection information is a general term for information output by camera 20. The camera detection information includes, for example, video information generated by camera 20 by capturing an image of an object, image-captured object identification information that identifies the imaged object included in the video information, image capture time information, and image capture condition information that specifies the position and field of view of the camera at the image capture time.

[0021] Furthermore, if the camera 20 implements a so-called AI function that uses a learned model to determine the type of subject (hereinafter referred to as the "imaged object") captured by the camera 20 and included in the video information, the camera detection information may include attribute information indicating the type (label) of the captured object.

[0022] In the monitoring system 100, a recorder 40, a security guard terminal 80, and a security drone 81 are connected to a network 90.

[0023] The server 30 includes a server processor 31 including a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) suitable for AI processing, RAM (Random Access Memory), ROM (Read Only Memory), a memory 32 that functions as a data storage area and a work area for the server processor 31, a database 33, and a communication I / F (interface) 34, which are connected via a bus.

[0024] The server processor 31 is connected to an operation device 60 that is operated by a user of the monitoring system 100. The server processor 31 acquires an operation input signal for the monitoring system 100 via the operation device 60.

[0025] The server processor 31 is connected to a monitor 70. The monitor 70 displays a user interface (UI) screen provided by the monitoring system 100.

[0026] The memory 32 stores map information of the area to be monitored in advance.

[0027] The server 30 determines whether the radar-detected object and the captured image object are the same, and if so, performs processing to link the radar-detected object identification information with the camera detection information. Furthermore, when an operation input signal based on a user's operation to specify a past time is received from the operation device 60, the server 30 superimposes the position of the radar-detected object at the specified time on map information and displays on the monitor 70 the image information of the captured image object that has been determined to be the same as the radar-detected object at the specified time.

[0028] The server 30 records, in the recorder 40, linking information that links the radar-detected object identification information with the camera-detected information.

[0029] In addition, when a radar-detected object enters the monitored area, the server 30 instructs the camera 20 to track and photograph the object, displays an alarm on the monitor 70, and issues an alarm to at least one of the security guard terminal 80 or security drone 81.

[0030] When the camera 20 captures an image of a radar-detected object, the radar-detected object and the captured imaged object are the same. However, when the camera 20 captures an image of an object other than the radar-detected object, the radar-detected object and the captured imaged object are different. Therefore, the monitoring system 100 has a monitoring function that determines whether the radar-detected object and the captured imaged object are the same, and based on the result of the identity determination, transmits a notification signal to the server 30 notifying the server 30 of the presence of an intruder or the occurrence of an abnormality. Therefore, in FIG. 1 , the server 30 corresponds to the monitoring device. In a second embodiment described later, with reference to FIG. 26 , an example will be described in which the radar device 10 is equipped with a monitoring function and functions as the monitoring device.

[0031] The recorder 40 is a storage (Network Attached Storage) connected to the network 90. ​​The recorder 40 records and records radar detection information and camera detection information. The recorder 40 links the radar detection information with the camera images, and can display the position of an object on a map and the camera images linked to the object in a linked manner on the monitor 70. Next, with reference to FIGS. 2 to 4, an example of the linked display of the position of an object on a map and the camera images linked to the object will be described.

[0032] FIG. 2 is a diagram showing a UI screen example 1 (UI screen 201) displayed on the monitoring system 100. As shown in FIG.

[0033] The UI screen 201 shown in Figure 2 is a UI screen in which the radar detection range 210 of the radar device 10, the imageable area 220 of the camera 20, the field of view area 221 of the camera 20 at time t1 within the imageable area 220, the installation position 231 of the radar device 10, and each installation position 232 of the camera 20 are superimposed against a map screen 290 that displays map information stored in the memory 32 at time t1.

[0034] The user sets a no-entry line L1 on the map screen 290 in advance, prohibiting the entry of objects. The set no-entry line L1 is superimposed on the map screen 290. Note that, on the UI screen 201 (see FIG. 2 ), the area surrounded by the no-entry line L1 is set as a no-entry area. In the example shown in FIG. 2 , if an object crosses the no-entry line L1 (i.e., enters from outside the area surrounded by the no-entry line L1) and enters the no-entry area, the camera 20 continues to capture the object that has entered the no-entry area by automatic tracking and capturing. The server 30 may also issue an alarm, for example, indicating that an object has crossed the no-entry line L1. An object non-detection mask may be set outside the no-entry area on the map screen 290 to prevent unnecessary alarms from being issued. Furthermore, the area outside the no-entry area on the map screen 290 may be displayed as a blur mask to improve the visibility of the intrusion detection area. In other words, the no-entry line L1 is a line that, when the radar device 10 detects the occurrence of an object that has crossed the no-entry line L1, causes the object to be monitored by the camera 20. Note that the detection of an object's intrusion may not be based on the object passing through the no-entry line L1, but on the object's intrusion into a preset no-entry area.

[0035] It is preferable that the area within the monitoring area where radar detection is not possible (blind spot) be as small as possible. Therefore, it is desirable that the positions and orientations of the multiple radar devices 10 be such that the outer edge areas of each radar detection range 210 overlap.

[0036] When the radar device 10 detects a radar-detected object, the server 30 displays a radar detection point icon 241 indicating the position of the radar-detected object on the map screen 290.

[0037] Furthermore, when the attribute of the radar-detected object is determined, an icon indicating the attribute may be displayed on the map screen 290 instead of the radar detection point icon 241. For example, if the attribute of the radar-detected object is a person, an attribute icon 242 is displayed instead of the radar detection point icon. Since the radar-detected object is a person, a human-shaped icon is used as the attribute icon 242. As another example of the attribute icon 242, if the attribute of the radar-detected object is a bicycle, a bicycle-shaped icon is used.

[0038] The server 30 adds a moving speed 242v of the radar-detected object to the attribute icon 242 and a dotted line indicating a moving trajectory 242tr of the radar-detected object indicated by the attribute icon 242, and displays them superimposed on the map screen 290.

[0039] Furthermore, when the user operates the operation device 60 to click the attribute icon 242, the server 30 pinpoints and superimposes a camera screen 245 capturing an image of the person indicated by the attribute icon 242 on the UI screen 201. The server 30 also displays, together with the camera screen 245, the type "Human", the movement speed "3 km / h", and the score "score 80" as attribute information 245at of the attribute icon 242 on the UI screen 201. The score "score 80" corresponds to an index indicating the probability that the person is "Human" when the camera 20 performs a camera AI detection process.

[0040] FIG. 3 is a diagram showing a UI screen example 2 (UI screen 202) displayed on the monitoring system 100. As shown in FIG.

[0041] 2 in that, instead of the pinpoint display of the camera screens on the UI screen 201 in FIG. 2, the UI screen 202 shown in FIG. 3 displays a list of camera screens 261, ..., 26N of multiple cameras 20. Any number of camera screens can be arranged on the UI screen 202.

[0042] When the user clicks on the viewing area 221 superimposed on the map screen 290 using the operation device 60, the server 30 performs PTZ control of the camera 20 corresponding to the viewing area 221. This changes the field of view (angle of view) of the camera 20 from the viewing area 221bef to the viewing area 221aft, allowing other objects to be photographed. Other information superimposed on the map screen 290 is the same as that on the UI screen 201. Furthermore, the viewing areas 221 on the UI screen 201 and a UI screen 203 (described later) can also function as operation icons for PTZ control in the same manner as described above.

[0043] FIG. 4 is a diagram showing a UI screen example 3 (UI screen 203) displayed on the monitoring system 100.

[0044] The UI screen 203 shown in FIG. 4 displays the attribute icon 243 of the alarm-required object in a different manner. The change in the attribute icon may be achieved, for example, by changing the display color or by flashing the icon. Furthermore, when the server 30 issues an alarm, it automatically displays a camera screen 249 capturing an image of the alarm-required object and attribute information 249at on the monitor 70 alongside the map screen 290. Furthermore, the color of the field of view 221alt capturing the camera screen 249 may be changed upon alarm activation to make the location of the alarm-required object easier to understand. Furthermore, when an alarm is issued, the server 30 may transmit the UI screen 203 to the security guard terminal 80 to alert the security guard.

[0045] FIG. 5 is a functional block diagram of the radar device 10.

[0046] The radar device 10 includes a radar processor 11 , a memory 12 , a radar module 13 , and a communication I / F 14 .

[0047] The radar processor 11 may include a CPU, a GPU suitable for AI processing, etc. The radar processor 11 controls detection by the radar module 13, communication via the communication I / F 14, and overall processing of the radar device 10.

[0048] The memory 12 includes RAM, ROM, flash memory, etc., and functions as a data storage area and a work area for the radar processor 11.

[0049] The radar module 13 includes a radar IC (integrated circuit) 131, multiple radar transmitting antenna arrays ATx1 to ATxm, and radar receiving antenna arrays ARx1 to ARxn. A scan cycle corresponds to the time from when the radar receiving antenna arrays ARx1 to ARxn sequentially irradiate radar toward the radar detection range 210 until the radar receiving antenna arrays ARx1 to ARxn receive the reflected waves. The radar receiving antenna arrays ARx1 to ARxn output reception information of the reflected waves received during one scan cycle to the radar IC 131.

[0050] The radar processor 11 includes a radar attribute determination unit 15 and a radar detection information processing unit 16. The radar processor 11 configures the radar attribute determination unit 15 and the radar detection information processing unit 16 by executing programs that realize the functions of each unit.

[0051] The radar attribute determination unit 15 determines the attributes of a radar-detected object based on the sensor output (point cloud data) of the radar module 13. In the following description, the attributes of a radar-detected object are referred to as "radar attributes," and data indicating the "radar attributes" are referred to as "radar attribute labels." The attributes of an imaged object are referred to as "camera attributes," and data indicating the "camera attributes" are referred to as "camera attribute labels." The process of determining attributes is referred to as "labeling."

[0052] The radar attribute determination unit 15 includes an AI calculation processing unit 151 and a learning model memory 152, and performs labeling using AI (Artificial Intelligence).

[0053] The learning model memory 152 stores a learned model obtained by performing machine learning on an AI algorithm using point cloud data of radar-detected objects such as people, cars, and animals that the surveillance system 100 may detect, and training data annotated with the types of the radar-detected objects. The type of radar-detected object is an example of an object attribute.

[0054] The AI ​​calculation processing unit 151 inputs the sensor output received from the radar module 13 into the learned model stored in the learning model memory 152, detects the judgment score of the radar attribute of the radar-detected object, and performs labeling by classifying the attributes of the radar-detected object.

[0055] The radar detection information processing unit 16 performs processing such as tracking radar-detected objects based on the positions of the detection points and storing radar detection information.

[0056] FIG. 6 is a functional block diagram of the camera 20.

[0057] The camera 20 includes a camera processor 21 , a memory 22 , an imaging unit 23 , a communication I / F 24 , and a camera driving mechanism 27 .

[0058] The camera processor 21 includes a CPU and a GPU that performs AI processing, and controls imaging control by the imaging unit 23, communication control via the communication I / F 24, PTZ control of the imaging unit 23 by the camera drive mechanism 27, and overall processing of the camera 20.

[0059] The memory 22 includes RAM, ROM, flash memory, etc., and functions as a data storage area and a work area for the camera processor 21. The memory 22 may store information on calculation formulas used for mutual coordinate conversion between the coordinates of the radar coordinate system, the coordinates of the camera coordinate system, and the coordinates of the image coordinate system.

[0060] The imaging unit 23 includes an imaging element and a lens unit. The imaging element may include, for example, a complementary metal-oxide-semiconductor (CMOS) or a charge-coupled device (CCD).

[0061] If the camera 20 is a PTZ camera, the camera drive mechanism 27 includes a rotation motor 271 for panning and tilting the imaging unit 23, and a zoom motor 272 for controlling zoom by extending or retracting the lens unit. If the camera 20 is a fixed camera, only the zoom motor 272 is included.

[0062] In response to an instruction from the server 30, the camera processor 21 calculates a target position of the rotary motor 271 suitable for the imaging unit 23 to capture an image of an object located at the radar-detected object position detected by the radar device 10, and operates the rotary motor 271 to the calculated target position. At this time, the camera processor 21 may calculate, along with the target position of the rotary motor 271, a target position of the zoom motor 272 that provides a magnification suitable for the imaging unit 23 to capture an image of the object located at the radar-detected object position, and operate the zoom motor 272 to the calculated target position, thereby performing PTZ control.

[0063] As an example of PTZ control, the camera processor 21 operates the rotation motor 271 held by the PTZ camera until it reaches a target position calculated based on the relationship between the position (e.g., current position) of the rotation motor 271 and the field of view in real space, and on camera installation information (e.g., the camera installation position and camera imaging direction of the PTZ camera). Also, the camera processor 21 operates the zoom motor 272 to a target position calculated based on the relationship between the position (e.g., current position) of the zoom motor 272 held by the PTZ camera and the field of view in real space, and on the camera installation information.

[0064] As another example of PTZ control, a target position of the rotation motor 271 suitable for the imaging unit 23 to capture an image of an object at a radar-detected object position detected by the server 30 or the radar device 10 may be transmitted to the camera 20, and the rotation motor 271 may be operated to the target position. Furthermore, a target position of the zoom motor 272 with a magnification suitable for the imaging unit 23 to capture an image of an object at a radar-detected object position detected by the server 30 or the radar device 10 may be transmitted, and the zoom motor 272 may be operated.

[0065] The first camera attribute determination unit 25, the second camera attribute determination unit 26, and the camera detection information processing unit 28 are configured by the camera processor 21 executing a program that realizes each function of each unit.

[0066] The first camera attribute determination unit 25 includes an AI calculation processing unit 251 and a learning model memory 252.

[0067] The learning model memory 252 stores a learned model that has been machine-learned by an AI algorithm using training data annotated with images of objects that the surveillance system 100 may detect, such as people, cars, and animals, and the type of object that is imaged.

[0068] The AI ​​calculation processing unit 251 inputs the image information of the imaged object obtained from the imaging unit 23 into the learned model stored in the learning model memory 252, detects the judgment score of the camera attributes of the imaged object, and performs labeling by classifying the attributes of the imaged object.

[0069] The second camera attribute determination unit 26 determines the camera attributes by an alternative process different from the AI ​​algorithm when labeling using AI by the first camera attribute determination unit 25. As an example, the second camera attribute determination unit 26 performs background subtraction processing.

[0070] The camera detection information processing unit 28 executes processing such as tracking the imaged object and storing the camera detection information.

[0071] The AI ​​algorithms used by the radar attribute determination unit 15 and the first camera attribute determination unit 25 may be, for example, SVM (Support Vector Machine), deep learning, CNN (Convolutional Neural Network), or R-CNN (Region Based Convolutional Neural Network), and are not limited thereto.

[0072] FIG. 7 is a functional block diagram of the server 30.

[0073] The server processor 31 functions as a calibration unit 35, a detected object identification unit 36, a display control unit 37, a camera determination unit 38, and a monitoring function unit 39. Each of these units is configured by the server processor 31 executing a program that realizes the function of each unit. Hereinafter, the programs executed by the server processor 31 may be collectively referred to as the "video monitoring playback program."

[0074] The calibration unit 35 generates a conversion formula for converting the position coordinates of a radar-detected object defined in the radar coordinate system into the map coordinate system used in the map information. Also, the calibration unit 35 generates a conversion formula for converting the position coordinates of an imaged object defined in the camera coordinate system into the map coordinate system used in the map information. The conversion formula generation process may use any known technology as appropriate.

[0075] For example, the installation position of the radar device 10 is set as the origin of the radar coordinate system, and the map coordinates of the installation position are acquired. Furthermore, the radar device 10 detects a calibration point whose map coordinates are known, and acquires the coordinates of the calibration point in the radar coordinate system. The calibration unit 35 calculates a coordinate transformation formula for converting the radar coordinate system into the map coordinate system as a radar coordinate transformation formula, using the map coordinate system coordinates of the origin of the radar coordinate system and the radar coordinate system coordinates and map coordinate system coordinates of the calibration point. The radar coordinate transformation formula is generated for each radar device 10.

[0076] The calibration unit 35 also acquires map coordinates of the installation position of the camera 20, with the installation position being the origin of the camera coordinate system. Furthermore, the camera 20 captures an image of a calibration point whose map coordinates are known, and calculates the amount of rotation and translation of the calibration point relative to the installation position of the camera 20, based on shooting condition information including the pan angle, tilt angle, and zoom magnification at the time the image was captured. The calibration unit 35 calculates, as a camera coordinate transformation formula, a coordinate transformation formula for converting the camera coordinate system to the map coordinate system, using the map coordinate system coordinates of the origin of the camera coordinate system, the amount of rotation and translation of the calibration point, and the map coordinate system coordinates of the calibration point. The camera coordinate transformation formula is generated for each camera 20.

[0077] The detected object identification unit 36 ​​determines whether the radar-detected object and the captured image object are the same object. The determination of identity is based on the position of the radar-detected object in the map coordinate system and the radar attributes labeled by the radar device 10, and the position of the camera-detected object in the map coordinate system and the camera attributes labeled by the camera 20, and their identity and positions.

[0078] The detected object identification unit 36 ​​converts the coordinates of the radar coordinate system of the radar-detected object into the map coordinate system using a radar coordinate conversion formula, and converts the coordinates of the camera coordinate system of the imaged object into the map coordinate system using a camera coordinate conversion formula, and compares the positions of the radar-detected object and the imaged object.

[0079] The display control unit 37 generates display data for the UI screens shown in FIGS. 2 to 4 and FIGS. 23 to 25 (described later), and controls the display on the monitor 70.

[0080] The display control unit 37 also obtains the coordinates of the radar-detected object along a time series, calculates the movement trajectory of the radar-detected object, and displays it superimposed on the map information.

[0081] The camera determination unit 38 determines which camera will capture an image of a radar-detected object. The camera determination unit 38 also executes a process (handover process) to switch the camera that captures an image of a radar-detected object (image capture object).

[0082] The monitoring function unit 39 includes an intrusion determination unit 391 and a notification processing unit 392. The intrusion determination unit 391 determines whether or not a radar-detected object has crossed the intrusion no-entry line L1.

[0083] The notification processing unit 392 notifies the camera 20 of the position of the object (intruding object) that has crossed the no-entry line L1, and the camera 20 then begins tracking and photographing the intruding object. The notification processing unit 392 also notifies the display control unit 37 that an intruding object has occurred, and the display control unit 37 displays the UI screen (FIG. 4) that is displayed when an alarm is issued. The notification processing unit 392 then notifies the security guard terminal 80 and security drone 81 of the occurrence of the intruding object.

[0084] (Initial Setting) FIG. 8 is a flowchart showing the flow of the initial setting process.

[0085] Map information of the area to be monitored is registered in the server 30 (S101). Furthermore, device information of the radar device 10 and the camera 20 is registered in the server 30 (S102). The device information includes the product number, model name, IP address, login ID, password, etc. of the radar device 10 and the camera 20.

[0086] The server 30 transmits a mode transition instruction to each of the radars 1, 2, ..., N to transition to the calibration mode (S103). The operation modes of the radar device 10 include a normal operation mode and a calibration mode.

[0087] The normal operation mode of the radar device 10 is an operation mode in which the radar device 10 outputs only the position data of the radar-detected object at the time the object is detected as radar detection information. Therefore, the radar detection information output in the normal operation mode does not include past position data of the radar-detected object. This makes it easy to grasp the current position of the radar-detected object. On the other hand, the calibration mode of the radar device 10 is an operation mode in which the radar detection information includes the position data of the radar-detected object from the start of operation in the calibration mode to the present time. This leaves a trace of the movement of the radar-detected object, making it easy to align it with map coordinates. Note that, in order to explain the normal operation mode and the calibration mode of the camera 20 from step S107 onwards (described later), the calibration mode of the radar device 10 and the calibration mode of the camera 20 may be referred to as the radar calibration mode and the camera calibration mode in order to distinguish them.

[0088] Each of the radars 1, 2, ..., N operates in radar calibration mode in response to the mode transition instruction, detects an object whose position in a map coordinate system is known (referred to as a calibration object), and acquires the movement trajectory of the calibration object (S104). Then, each of the radars 1, 2, ..., N transmits the movement trajectory and depression angle information of the calibration object to the server 30 (S105). The depression angle of each of the radars 1, 2, ..., N is detected by an internal sensor provided in each radar.

[0089] The server 30 aligns the installation position and direction of each radar 1, 2, ..., N on the map based on the movement trajectory and depression angle information of the calibration object acquired from each of the radars 1, 2, ..., N. The server 30 also acquires the depression angle information and accepts an operation input signal for the installation height of each of the radars 1, 2, ..., N from the operation device 60 (S106). The calibration unit 35 generates a radar coordinate transformation formula for matching the position and direction of the radar coordinate system of the calibration object with the map coordinates of the calibration object.

[0090] The server 30 also transmits a mode transition instruction to each of the PTZ cameras 1, ..., M to transition to the camera calibration mode (S107). The camera 20 has a normal operation mode and a camera calibration mode.

[0091] The normal operation mode of the camera 20 is an operation mode in which the camera 20 outputs image information each time it captures an image of an object. On the other hand, the camera calibration mode is an operation mode in which the camera 20 outputs the PTZ position at the time the camera 20 captured an image of the calibration object in addition to outputting the image information. As a result, by capturing images of a calibration object whose positions at two different points on the map coordinates are known, it is possible to obtain the PTZ positions corresponding to the map coordinates of the calibration object located at each of the two known points (S108).

[0092] Each of the PTZ cameras 1, . . . , M transmits the two PTZ positions acquired in step S108 to the server 30 (S109).

[0093] The server 30 aligns the installation positions and directions of the PTZ cameras 1, ..., M on the map with the map coordinate system. The server 30 also receives operation input signals specifying the depression angles and heights of the PTZ cameras 1, ..., M (S110). The calibration unit 35 generates a camera coordinate transformation formula for matching the position and direction of the camera coordinate system of the calibration object with the map coordinate system of the calibration object.

[0094] Furthermore, the server 30 transmits a mode transition instruction to each of the fixed cameras 1, 2, ..., and O to cause them to transition to the camera calibration mode (S111). Each of the fixed cameras 1, 2, ..., and O operates in the camera calibration mode in response to the mode transition instruction, acquires the position of the calibration object expressed in the camera coordinate system (including the coordinates on the screen) (S112), and transmits it to the server 30 (S113).

[0095] The server 30 aligns the installation positions and directions of the fixed cameras 1, 2, ..., O on the map with the map coordinate system. The server 30 also receives an operation input signal specifying the depression angle and height of each of the fixed cameras 1, 2, ..., O (S114). The calibration unit 35 generates a camera coordinate transformation formula for matching the position and direction of the camera coordinate system of the calibration object with the map coordinate system of the calibration object.

[0096] In the initial setting, in addition to map information, the server 30 also registers information obtained by calibration, such as the installation position on the map, height, and front direction of each device, as well as radar coordinate conversion formulas and camera coordinate conversion formulas. Radar detection information and linking information for recording are used for real-time display, but are not recorded in the server 30 but in the recorder 40.

[0097] (Normal Operation Process 1) Fig. 9 is a flowchart showing the flow of normal operation process 1. Normal operation process 1 is a process for making an intrusion detection determination using both radar and a camera.

[0098] The recorder 40 starts recording (S201). Also, each of the PTZ cameras 1, ..., M and each of the fixed cameras 1, 2, ..., O start capturing images (S202, S203). The recorder 40 receives video information from each of the PTZ cameras 1, ..., M and each of the fixed cameras 1, 2, ..., O (S204, S205).

[0099] Each of the radars 1, 2, ..., N performs radar detection processing and transmits radar detection information to the server 30 (S206). The display control unit 37 of the server 30 displays the received object positions on a map (S207). The radar detection information includes point cloud coordinate information, radar-detected object identification information (hereinafter referred to as "ID"), and attribute information of the radar-detected object.

[0100] Furthermore, each of the radars 1, 2, ..., N transmits the radar detection information to the recorder 40 (S208), and the recorder 40 records the radar detection information. The processing of steps S201 to S208 corresponds to the display processing of the UI screen 203 in FIG.

[0101] When any of the radars 1, 2, . . . , N detects that an object has passed through the no-entry line (S209), each of the radars 1, 2, .

[0102] The camera determination unit 38 of the server 30, which has received the "passing information," determines which PTZ camera should capture the radar-detected object (S211). In this step, the determination is made based on the image-capable area of ​​the camera, and if there are multiple candidates, the closest one is selected. In this step, the user may specify the camera when initially setting up the system.

[0103] The camera determination unit 38 of the server 30 calculates the PTZ motor position for capturing an image with the determined PTZ camera (S212), and transmits PTZ motor control information including the PTZ motor position to the determined camera (S213).

[0104] Upon receiving the PTZ motor control information, the PTZ camera drives the rotary motor 271 and zoom motor 272 of the camera drive mechanism 27 (S214) and starts capturing images.

[0105] The first camera attribute determination unit 25 performs AI detection based on the video information and transmits attribute information of the captured object to the server 30 (S215).

[0106] The detected object identification unit 36 ​​of the server 30 links the attributes of the radar-detected object with the attributes of the captured object (S216). In the linking process, for example, if the position coordinates on the map obtained by converting the position coordinates of the radar-detected object in the radar coordinate system into the map coordinate system are close to the position coordinates on the map obtained by converting the position coordinates of the camera-captured object in the camera coordinate system, and further if the radar attribute label of the radar-detected object matches the camera attribute label of the camera-captured object, the radar-detected object and the camera-captured object may be linked as being the same.

[0107] The intrusion determination unit 391 of the server 30 makes an intrusion detection determination (S217), and the notification processing unit 392 issues an alarm as necessary (S218), and records the history of the alarm issuance in the recorder 40 (S219). In this case, the following processing may be performed based on the information of the radar attribute label and the camera attribute label. For example, an alarm may be issued only when it is determined that the intruding object is a person based on the radar attribute label and the camera attribute label. Furthermore, if an intrusion is detected by the radar but the object cannot be captured by the camera, it may be determined that this is a false detection by the radar and no alarm may be issued.

[0108] When capturing an image of a radar-detected object using fixed cameras 1, 2, ..., O, zoom magnification control information for optimizing the zoom magnification is sent instead of PTZ control information (S220), camera AI detection processing is performed, and attribute information of the captured object is sent to server 30 (S221).

[0109] (Normal Operation Process 2) Fig. 10 is a flowchart showing the flow of normal operation process 2. Normal operation process 2 is a process for determining whether an intrusion has been detected using only radar.

[0110] Steps S201 to S210 are the same as those in the normal operation process 1, and therefore a duplicated explanation will be omitted.

[0111] The server 30 performs an intrusion detection determination (S231) based on the radar detection information and the information on passage of the no-entry line. If an intrusion is detected, an alarm is issued (S232), and the alarm issuance history is recorded in the recorder 40 (S233).

[0112] Thereafter, as in normal operation process 1, the server 30 determines which PTZ camera to use to capture the radar-detected object (S211), calculates the PTZ motor position for capturing the image with the selected PTZ camera (S212), and transmits PTZ motor control information to the designated PTZ camera (S213). The PTZ camera then drives the rotation motor 271 and zoom motor 272 (S214) to begin capturing the image. The process from step S211 onward continues until the alarm ceases. During this time, if it is necessary to switch cameras, the server 30 transmits PTZ motor control information to the target PTZ camera (S222), drives the rotation motor 271 and zoom motor 272 (S223), and captures the image with the selected PTZ camera.

[0113] (Real-time display and recording on a recorder) FIG. 11 is a flowchart showing the flow of processing for real-time display and recording on a recorder.

[0114] The recorder 40 starts recording (S301), and each of the fixed cameras 1, 2, ..., O captures an image (S302), and transmits the streaming of the video information and imaging condition information including the zoom magnification at that time to the server 30 and the recorder 40 (S303).

[0115] Each of the PTZ cameras 1, . . . , M also captures an image (S304), and transmits the streaming of the video information and imaging condition information including the pan angle, tilt angle, and zoom magnification at that time to the server 30 and the recorder 40 (S305).

[0116] The display control unit 37 of the server 30 processes the display of the video information received from the fixed cameras 1, 2, ..., O and the PTZ cameras 1, ..., M, and displays the camera's field of view on the map screen 290 based on the imaging condition information received from each camera (S306).

[0117] Each of radars 1, 2, ..., N performs radar detection, generates radar detection information including the coordinates of the point cloud in the radar coordinate system, an ID that uniquely identifies the radar detected object, and attributes of the radar detection information, and transmits it to the server 30 (S307, S308).

[0118] The detected object identification unit 36 ​​of the server 30 converts the coordinates of the point cloud in the radar coordinate system into coordinates in the map coordinate system, and the display control unit 37 displays the positions of the radar-detected objects on the map screen 290 (S309).

[0119] The detected object identification unit 36 ​​of the server 30 performs an ID synthesis process for the object IDs of the radar boundary (S310).

[0120] FIG. 12 is an explanatory diagram of the ID synthesis process.

[0121] In Figure 12, four radar devices 10 are installed such that adjacent radar detection ranges 210a, 210b, 210c, and 210d overlap with adjacent radar detection ranges at the boundaries of each radar detection range. In the overlapping areas, each radar device 10 detects the same object, assigns an ID to it, and transmits radar detection information to the server 30. As a result, multiple IDs are assigned to the same radar-detected object, and the IDs of the overlapping radar-detected objects are combined into one within the server 30. This allows the server 30 to manage the IDs using the same ID when an intruder 240 moves across multiple radar detection ranges.

[0122] Returning to the explanation of Fig. 11, when the server 30 detects an intruder 240 and performs automatic tracking and photographing using the camera 20, the server 30 displays the image of the camera 20 tracking the intruder 240 (S311) and transmits the radar ID and information about the camera tracking that ID to the recorder 40 (S312). As a result, information linking the radar ID and the information about the camera tracking that ID is stored in the recorder 40 (S313).

[0123] (Handover Processing) The handover processing will be described with reference to Fig. 13 to Fig. 17. Fig. 13, Fig. 14, and Fig. 15 are explanatory diagrams of the handover processing, Fig. 16 is a sequence diagram of handover (when handover is possible), and Fig. 17 is a sequence diagram of handover (when handover is not possible).

[0124] Scene 1 in Figure 13 shows the initial state. As shown in the overhead map 401, the image capture ranges 411 and 412 of PTZ camera 1 and PTZ camera 2 overlap at the camera boundary. The field of view areas 421 and 422 of PTZ camera 1 and PTZ camera 2 do not overlap in scene 1. A person 430 is outside detection line L2 (farther from PTZ camera 1 using detection line L2 as the reference) and is walking toward detection line L2.

[0125] In scene 1, the server 30 displays a screen 451 on the monitor 70, which displays a list of images from multiple cameras. In Fig. 13, the screen 451 displays a list of real-time images from PTZ cameras 1, 2, ..., N and fixed cameras 1, 2, ..., O connected to the server 30, respectively, on small screens a to l. The small screens for displaying the list may be any of the PTZ cameras 1, 2, ..., N and fixed cameras 1, 2, ..., O, and the number of small screens can also be set as appropriate.

[0126] In scene 2, it is detected that person 430 has passed through detection line L2. Server 30 transmits PTZ control information to PTZ camera 1 so that PTZ camera 1 begins tracking person 430 and so that person 430 is included in the field of view 421 of PTZ camera 1 and so that the face of person 430 is visible. The video captured by PTZ camera 1 is captured from a viewpoint from a direction in which the face of person 430 is visible. Server 30 displays screen 452, which shows the video captured by PTZ camera 1, on monitor 70.

[0127] In scene 3, as person 430 moves, camera determination unit 38 of server 30 transmits PTZ control information to PTZ camera 1 to track and capture images of person 430. PTZ camera 1 continues to track and capture images of person 430, and as person 430 continues to move across PTZ camera 1, an image captured by PTZ camera 1 from a side view of person 430 is displayed on screen 453. Gradually, only the back of person 430 becomes visible on screen 453.

[0128] In scene 4 of Fig. 14, when person 430 approaches outside the imaging range 411 of PTZ camera 1, PTZ camera 2 begins imaging the same person 430. When tracking begins with PTZ camera 2, server 30 displays camera images from both PTZ cameras 1 and 2 on screen 454. At this time, the camera image from which the face of person 430 is visible (image from camera 2) is displayed preferentially. In this example, the display size is relatively large.

[0129] In scene 5, person 430 moves out of the image capture range 411 of PTZ camera 1, so tracking is performed only by PTZ camera 2, and PTZ camera 1 returns to its initial position. Server 30 displays only the image from PTZ camera 2 on screen 455. Screen 455 displays an image from a viewpoint from a direction in which the face of person 430 can be seen.

[0130] 15, unlike scenes 1 to 5, while PTZ camera 1 is tracking person 430, PTZ camera 2 is also tracking another person 431. In this case, server 30 displays the images from both PTZ cameras 1 and 2 side by side on screen 456 of monitor 70.

[0131] In scene 7, even if person 430 moves outside the field of view 421 of PTZ camera 1, PTZ camera 2 is tracking person 431, so no handover is performed and PTZ camera 1 continues to capture the image of person 430. At that time, the pan angle and tilt angle are controlled so that person 430 remains within the field of view even if the zoom limit is exceeded. Then, images from both PTZ cameras 1 and 2 are displayed on screen 457.

[0132] The flow of the handover process will be described with reference to FIGS.

[0133] Radars 1, 2, . . . N transmit radar detection information to the server 30 (S401). The processing of this step continues while the monitoring system 100 is executing the monitoring processing.

[0134] The PTZ camera 1 starts capturing images of the person 430 (corresponding to the tracking target) who has crossed the detection line L2 (S402). The PTZ camera 1 performs PTZ control to track and capture images in accordance with the movement of the tracking target. The PTZ control uses the camera AI of the PTZ camera 1 to automatically track and capture images by the camera alone (S403).

[0135] On the other hand, the PTZ camera 2 captures an image of the preset position in the initial state (S404).

[0136] The camera determination unit 38 of the server 30 determines whether handover is necessary based on the position of the tracking target (S405). Specifically, if the tracking target is in the overlapping coverage area of ​​the PTZ camera 1 and another camera, it determines whether handover is necessary based on the moving direction of the tracking target being determined.

[0137] The camera determination unit 38 of the server 30 determines the PTZ camera to be the handover destination based on the movement direction of the tracking target (S406). For example, if the tracking target is moving away from PTZ camera 1 and approaching another camera (PTZ camera 2), the camera determination unit 38 determines the handover destination to be PTZ camera 2.

[0138] The server 30 inquires of PTZ camera 2 whether handover is possible (S407). The server 30 transmits inquiry information as to whether tracking is in progress (S408). When the server 30 receives a response from PTZ camera 2 stating "not in progress" or "handover OK" (S409), it transmits handover instruction information to PTZ cameras 1 and 2 (S410). Specifically, as handover instruction information, it transmits tracking stop instruction information to PTZ camera 1 and tracking start instruction information to orient PTZ camera 2's field of view toward the tracking target. Furthermore, the camera priorities are changed. In this example, the priority of PTZ camera 2 is changed from 2 to 1, and the priority of PTZ camera 1 is changed from 1 to 2 (S411).

[0139] When the PTZ camera 1 receives the tracking stop instruction information, it stops automatic tracking and photographing (S412) and returns to its initial position (S413).

[0140] The PTZ camera 2 performs PTZ control to capture images of the object being tracked in accordance with its movement, changes the direction of the PTZ camera 2, and starts automatic tracking and capturing (S414). Thereafter, tracking continues (S415).

[0141] Next, a case where handover is not possible will be described. As shown in Fig. 17, after the PTZ camera 2 starts capturing an image of a preset position (S404), it starts capturing an image of another tracking target (S420) and performs PTZ control in accordance with the movement of the tracking target (S421).

[0142] In this state, when the server 30 sends an inquiry to the PTZ camera 2 asking whether it is being tracked (S408), the PTZ camera 2 responds with "tracking" or "handover NG" (S422), and the server 30 determines that the PTZ camera 2 is being tracked and therefore will not be the handover destination (S423). The PTZ cameras 1 and 2 continue tracking their respective tracking targets (S424, S425).

[0143] (Method of Linking Information When Recording Radar Information) FIG. 18 is a diagram showing an example of linking information.

[0144] The linking information table 500 is an example of the data structure of radar detection information, and is not limited to a table structure. The server 30 generates the linking information table 500, and the recorder 40 stores it. The server 30 also updates the linking information table 500 at predetermined time intervals. Figure 18 shows the linking information table 500 at time "Time 1."

[0145] The linking information table 500 includes an "ID" record 501, a "coordinate" record 502, an "attribute 1 (type)" record 503, an "attribute 2 (speed)" record 504, a "tracking flag" record 505, an "alarm" record 506, a "corresponding camera 1" record 507, a "camera 1 priority" record 508, a "corresponding camera 2" record 509, and a "camera 2 priority" record 510.

[0146] In the "ID" record 501, a composite ID is recorded.

[0147] Coordinates in a map coordinate system are recorded in the "coordinates" record 502. Although three-dimensional coordinates are recorded in Fig. 18, two-dimensional coordinates may also be used.

[0148] Instead of the "attribute 1 (type)" record 503 and the "attribute 2 (speed)" record 504, an attribute other than type and speed may be recorded.

[0149] The "tracking flag" record 505 indicates whether any of the PTZ cameras is performing automatic tracking and shooting.

[0150] The "alarm" record 506 records information indicating whether an alarm has been issued for the tracking target corresponding to the composite ID. While "True" is stored in the "alarm" record 506, the server 30 displays the UI screen 204 of FIG. 4 on the monitor 70.

[0151] The "Corresponding camera 1" record 507 and the "Corresponding camera 2" record 509 store the names of the cameras capturing the tracking target corresponding to the composite ID stored in the "ID" record 501. There may be two or more corresponding cameras.

[0152] The "Priority of Camera 1" record 508 and "Priority of Camera 2" record 510 store information indicating the priority of the camera whose video the server 30 displays on the UI screen. When an object is captured by multiple cameras, the display priority is assigned according to this "Priority of Camera N." The "Priority of Camera N" is determined based on the direction of movement and zoom magnification, and assigned in order of how large the front view is likely to be. When playing back video information, the server 30 determines whether to display or size the display based on the "Priority of Camera N."

[0153] All data may be stored on the server, or all data may be recorded on the recorder 40 and not stored on the server 30. When playing back, the server 30 refers to the linking information table 500 stored in the recorder 40. Since the storage capacity of the recorder 40 is larger than that of the server 30, the linking information table 500 may be recorded on the same recorder 40 together with the camera footage and not stored on the server 30.

[0154] (Data Structure of Camera Imaging Condition Information) FIG. 19 is a diagram showing an example of the data structure of camera imaging condition information.

[0155] The image capturing condition information table 600 is an example of the data structure of the image capturing condition information of the camera, and is not limited to a table structure. The server 30 generates the image capturing condition information table 600, and the recorder 40 stores the image capturing condition information table 600. The server 30 also updates the image capturing condition information table 600 at predetermined time intervals. Fig. 19 shows the image capturing condition information table 600 at time "Time 1".

[0156] The image capturing condition information table 600 includes a “camera information” record 601 and a “viewing angle information” record 602 .

[0157] The "camera information" record 601 stores information that can uniquely identify a camera. In Fig. 19, the name of the camera is stored.

[0158] The "angle of view information" record 602 stores the pan angle, tilt angle, and zoom magnification included in the image capture condition information received by the server 30 from each camera.

[0159] The imaging condition information table 600 is used when the server 30 plays back video information and displays the field of view of the camera specified by the camera information on the map screen 290. For example, when illustrating the position 232a of the PTZ camera 1 on the map screen 290pat, the field of view 221a is displayed using the pan angle, tilt angle, and zoom magnification registered in the imaging condition information table 600.

[0160] In addition to the image capturing condition information table 600, the image information from each camera is recorded in the recorder 40.

[0161] (Data Structure of Point Cloud Trajectory Information) FIG. 20 is a diagram showing an example of the data structure of point cloud trajectory information.

[0162] The point cloud trajectory information table 700 is an example of the data structure of point cloud trajectory information and is not limited to a table structure. The server 30 extracts the "ID" records 501 and the "coordinate" records 502 from the linking information table 500 and arranges them in chronological order to create the point cloud trajectory information table 700. The server 30 also records the point cloud trajectory information table 700 in the recorder 40.

[0163] The point cloud trajectory information table 700 includes a time record 701, an "ID1" record 702 and a "coordinate 1" record 703, an "ID2" record 704 and a "coordinate 2" record 705, ..., an "IDn" record 706 and a "coordinate n" record 707, an "IDm" record 708, and a "coordinate m" record 709. Each row of the point cloud trajectory information table 700 indicates a time, Time1, Time2, Time3, ..., Timet. When coordinates 1 of the same composite ID, for example, ID1, at Time1, Time2, Time3, ..., Timet1 are plotted on one map screen 290, the movement path of the object corresponding to the composite ID, for example, the movement trajectory 242tr in FIG. 2, can be superimposed and displayed on the map screen 290. The map screen 290 in FIG. 2 illustrates the situation at time Timet, but also shows coordinate changes of the travel route going back in time from Timet to, for example, Time1.

[0164] The display control unit 37 of the server 30 creates a table for displaying the movement trajectory of the point cloud on the server, and records the same data in the recorder.

[0165] (Playback Processing) FIG. 21 is a flowchart showing the flow of video playback processing.

[0166] When the display control unit 37 of the server 30 issues a playback instruction to the recorder 40 (S501), it acquires from the recorder information related to the camera, such as video information and imaging condition information including pan angle, tilt angle, and zoom magnification, and radar-related information such as point cloud coordinates, ID, attributes, and other information, alarm information, radar tracking ID, and information about the camera tracking the object (S502).

[0167] The display control unit 37 of the server 30 generates a movement route of the point cloud and displays it superimposed on the map screen 290. Also, the recorded camera image is superimposed (S503). The processing content of this step will be described with reference to FIG. 22.

[0168] FIG. 22 is a diagram showing an image of a playback screen.

[0169] The playback screen 750 reproduces the movement of objects on the map and the field of view of the camera capturing the objects. To achieve this, the server 30 determines the field of view 721, 722, 723 of each camera based on the imaging condition information of each camera, and superimposes this on the map screen 290 (for ease of explanation, the lines on the map screen 290 are omitted in FIG. 22). Furthermore, video information 731, 732 captured by each camera is superimposed and displayed. This allows the camera images to be linked to the objects on the map and displayed. The same is possible with a model that can zoom even with a fixed camera.

[0170] Furthermore, the server 30 refers to the point cloud trajectory information table 700 and plots the coordinates of the same object “ID” at different times, for example, the coordinates (x1, y1, z1) at Time 1, the coordinates (x2, y2, z2) at Time 2, and the coordinates (x3, y3, z3) at Time 3 of ID “1” on the map screen 290, thereby superimposing the movement trajectory 741 of the object assigned ID “1”.

[0171] FIG. 23 shows an example of an operation screen for playback.

[0172] The operation screen 800 is an operation screen for playing back the video of site A within the monitored area. The operation screen 800 includes a main screen area 810, a designated camera video area 820, an alarm video area 830, a seek bar 840, and an operation button area 850.

[0173] The main screen area 810 is an area where a main screen is displayed in which the camera and radar positions on the site map, the object positions and movement trajectories, the camera's field of view area, and the camera's imageable area are superimposed on the map screen 290. In the main screen area 810, for example, the UI screens 201, 202, and 203 shown in FIGS.

[0174] The designated camera image area 820 is an area where an image captured by a camera designated by the user is displayed.

[0175] The alarm video area 830 is an area where an image captured by a camera in an alarm state is displayed, regardless of whether or not the user has specified it. For example, the camera screen 249 in FIG.

[0176] The seek bar 840 is a bar that displays the recording time, alarm occurrence time, and current playback location in chronological order. The seek bar 840 displays the event occurrence time. When the event occurrence time is specified, the video of the event occurrence object and the position of the object involved in the event occurrence (event occurrence object) are superimposed on the map screen 290. After the event occurrence time is specified, when the event occurrence object moves within its position on the map screen 290, the video information of the event occurrence object is played back in conjunction with the movement of its position on the map screen 290. The seek bar 840 is displayed together with a calendar bar 841 for specifying the video capture date.

[0177] The operation button area 850 includes various operation buttons such as "play," "stop," "fast forward," "fast rewind," and "pause" for the video.

[0178] When the time of event occurrence is specified on the seek bar 840, the main screen returns to a screen in which the positions of each object at the time of event occurrence are superimposed on the map, so that the movement of the objects on the map at the time of event occurrence can be immediately confirmed. Furthermore, an image captured at the time of event occurrence is displayed in the specified camera image area 820.

[0179] By specifying the capture date of the video you want to play on the calendar bar 841 and specifying the playback time on the seek bar 840, the video captured at the specified date and time will be displayed in the designated camera video area 820. Then, by operating the "Play" button in the operation button area 850, video from the specified date and time will be played. By switching cameras, you can quickly access images of suspicious individuals captured by another camera before the event occurred from the playback screen. When the video switches, the main screen area 810 also switches accordingly to a screen showing the location of the object at the specified date and time. This allows the camera that captured the video displayed in the designated camera video area 820, the camera's location, the field of view, and the object's location to be superimposed on the map screen 290, allowing the user to easily grasp the object's location from a bird's-eye view.

[0180] When the event occurrence time registered in the seek bar 840 is clicked, the event search screen 900 of FIG. 24 may be displayed, providing a UI screen that makes it easy to search for video before and after the event occurrence.

[0181] FIG. 24 is a diagram showing an example of an event search screen.

[0182] The event search screen 900 includes an image gallery area 910 , a pagination area 920 , and a display scale 930 .

[0183] The image gallery area 910 is an area that displays a list of alarm response histories. As the alarm history, thumbnail images 911 of camera footage at the time an alarm was triggered are displayed, along with the capture time, the cause of the alarm (e.g., "motion detection," "motion detection alarm area 1"), and the capture location. Clicking on a thumbnail image 911 displays the video recorded at the time the alarm was triggered, allowing the user to check the video before and after the event.

[0184] The pagination area 920 has at least one or more thumbnail images 911, 912, 913, and 914 arranged therein. When the images cannot all be displayed in the image gallery area 910, selecting a number in the pagination area 920 causes the number of thumbnail images displayed to jump by a predetermined number.

[0185] The display scale 930 is an operation bar for changing the display magnification of the thumbnail images 911 , 912 , 913 , and 914 .

[0186] FIG. 25 is a diagram showing an example of the event search screen 2 (event search screen 950).

[0187] The event search screen 950 of Fig. 25 displays the thumbnail images 911, 912, 913, and 914 of Fig. 24 in a list format. The event search screen 950 displays the thumbnail images 911, 912, 913, and 914 of each camera image together with the alarm response history. Furthermore, a confirmation button 960 indicating that the image represented by each thumbnail image 911, 912, 913, and 914 has been confirmed is displayed. When the user presses the confirmation button 960 after confirming the image, the display color of the confirmation button 960 changes, making it easier to distinguish between confirmed and unconfirmed images.

[0188] In the monitoring system 100 according to this embodiment, the radar device 10 transmits information (e.g., coordinates and attributes) of detected objects to the server 30, and the camera 20 transmits the captured video and the attributes of the objects captured in the video to the server 30. The server 30 registers the installation location and direction of each device on a registered map, and displays the received detection and attribute information on the map. In this case, if multiple devices detect the same object, the information is integrated on the server 30. This makes it possible to detect intruders on the premises based on the information aggregated by the server 30 and issue an alarm.

[0189] Furthermore, when the server 30 detects an intrusion of an object into the premises based on the radar detection information, it instructs the PTZ camera to change the camera's angle of view so as to capture an image of the intruding object, or instructs the fixed camera to change the zoom magnification so as to capture an image of the intruding object. At this time, the server 30 generates linking information that links the alarm information indicating the intrusion detection with the radar-detected object ID captured by the camera 20, and the recorder 40 records this information.

[0190] This linking information allows the location of the object captured in the video to be superimposed on a map when the video is played back. Also, by referencing the imaging condition information table, the field of view of the camera in the past can be displayed on the screen.

[0191] Furthermore, by specifying an object superimposed on the map in the current or past main screen area, the current or past video in which the object was captured can be played back.

[0192] Furthermore, by extracting the past coordinates of the same object from the linking information, it is possible to trace back the object's movement trajectory at that time and display it on a map, and also to play back camera images associated with the object.

[0193] Thus, with the surveillance system 100 according to this embodiment, when a large site is monitored and a suspicious person or the like intrudes into the site, the recorded information can be played back to show on a map the direction in which the person moved, allowing security guards to intuitively know where to go. Furthermore, there is no need to search for footage of the intruder among multiple past camera footage; camera footage that captures the intruder is automatically selected and displayed. Since the security guards can confirm the person's clothing and other characteristics from the camera footage, this is also useful for identifying suspicious people.

[0194] Second Embodiment FIG. 26 is a functional block diagram of a monitoring system 100a according to a second embodiment.

[0195] The monitoring system 100a according to the second embodiment differs from the monitoring system 100 according to the first embodiment in that the radar device 10 has the video monitoring playback program and map information storage unit installed on the server 30 according to the first embodiment, and does not have a server 30.

[0196] In the case where the monitoring system 100a has a plurality of radars, common map information is stored in the map information storage unit of each radar.

[0197] Then, each radar creates a conversion formula for converting the radar coordinate system of the radar-detected object into a map coordinate system, and each radar stores the radar detection information including the position coordinates, ID, and attributes of the radar-detected object converted into the map coordinate system in a recorder 40.

[0198] In addition, each radar is equipped with the functions of the server 30, such as processing to link objects detected by the radar with objects captured by the camera.

[0199] When performing video playback processing, the operation device 60 and the monitor 70 may be connected to each radar device 10 and recorder 40 via a network 90, and any one of the radar devices 10, for example, the radar 1, may execute a monitoring video playback program to display each UI screen displayed in the first embodiment on the monitor 70. Alternatively, the recorder 40 may execute a program for playing back monitoring video based on information from the radar device 10, and each UI screen displayed in the first embodiment may be displayed on the monitor 70.

[0200] Alternatively, if there are multiple radar devices 10, one may be set as a master radar device, and only the master radar device may be provided with the same functions as the server 30 (the video monitoring playback program of the server 30 may be installed). In this case, the other slave radar devices may be configured to only achieve the same functions as the radars of the first embodiment.

[0201] According to this embodiment, there is no need to install a video monitoring playback program in the radar device, thereby eliminating the need to construct a server 30. This is expected to reduce the number of steps required to install the monitoring system 100a. Furthermore, there is no need to send and receive communication data to aggregate radar detection information from each radar device in the server 30, which reduces the communication load on the network 90.

[0202] Furthermore, in the second embodiment, as in the first embodiment, by specifying recorded information, it is possible to go back in time and display the movement trajectory of the object at that time on a map, and also to play back camera footage associated with the object.

[0203] The above-described embodiment is not intended to limit the present invention, and modifications that do not deviate from the spirit of the present invention are also included in the present invention. For example, the present invention may be used in combination with the following monitoring device. In such a case, modifications may be made by deleting unnecessary components.

[0204] For example, if the server 30 has a sufficient storage capacity, the recorder 40 may not be provided.

[0205] The above embodiment includes the following inventions.

[0206] (Supplementary Note 1) A monitoring system comprising: a radar device that irradiates radio waves to an object in a monitored area, detects the object based on the radio wave reflected by the object, and outputs radar detection information; a first camera that captures an image of the object and outputs camera detection information; a memory that stores map information indicating the monitored area; a processor; and a monitor, wherein the radar detection information includes radar-detected object identification information that identifies the radar-detected object detected by the radar device, a detection time, and a position coordinate at which the radar-detected object was detected, and the camera detection information includes video information generated by the first camera and imaged object identification information that identifies the imaged object captured in the video information, the image capture time, and image capture condition information for specifying the position and field of view of the first camera, and the processor determines whether the radar-detected object and the imaged object are the same based on the position coordinate at which the radar-detected object was detected and the image capture condition information, and if it is determined that they are the same, links the radar-detected object identification information with the camera detection information, when a designation operation for a past time is accepted, the position of the radar-detected object at the designated time is superimposed on the map information and displayed on the monitor, and the video information of the imaged object determined to be identical to the radar-detected object at the designated time is displayed on the monitor.

[0207] (Supplementary Note 2) The monitoring system according to Supplementary Note 1, further comprising a storage, wherein the processor determines whether the radar-detected object and the imaged object are the same based on the position coordinates where the radar-detected object is detected and the image-capturing condition information, and if it determines that they are the same, links the radar-detected object identification information with the camera detection information and records them in the storage.

[0208] (Supplementary Note 3) The surveillance system according to Supplementary Note 1, further comprising a second camera that outputs video information of an object and imaging condition information indicating the angle of view at the time of imaging, wherein when the object imaged by the first camera moves from the imaging area of ​​the first camera to the imaging area of ​​the second camera, the processor transmits control information to the second camera to change the direction of the object so that it fits within the field of view of the second camera, and the processor links the radar-detected object identification information, the video information and imaging time generated by the first camera when the object is identical to the radar-detected object, and the video information and imaging time generated by the second camera when the object is identical to the radar-detected object.

[0209] (Supplementary Note 4) The surveillance system according to Supplementary Note 3, wherein when the processor displays on the monitor the video information of the first camera in which the same object is captured and the video information of the second camera in which the same object is captured, the processor gives a relatively high priority to the video information in which the front of the object is captured and displays it on the monitor.

[0210] (Supplementary Note 5) The monitoring system according to Supplementary Note 1, wherein the processor acquires the radar detection information in chronological order, and displays on the monitor a movement trajectory indicating a change in the position coordinates at which the radar-detected object is detected in chronological order, superimposed on the map information.

[0211] (Supplementary Note 6) The monitoring system according to Supplementary Note 5, wherein when the position of the radar-detected object on the map information is moved, the processor plays back video information of the radar-detected object in conjunction with the movement of the position on the map information.

[0212] (Supplementary Note 7) The monitoring system according to Supplementary Note 1, wherein the processor displays the time of event occurrence, and displays on the monitor an image of the event occurrence object captured at that time and the position of the event occurrence object superimposed on the map information.

[0213] (Supplementary Note 8) The monitoring system according to Supplementary Note 1, further comprising a server, wherein the memory and the processor are provided in the server.

[0214] (Supplementary Note 9) The surveillance system according to Supplementary Note 1, wherein the memory and the processor are provided in the radar device.

[0215] (Supplementary Note 10) A radar device connectable to a monitor via a network, the radar device irradiating radio waves to an object in a monitored area, detecting the object based on the radio wave reflected by the object, and outputting radar detection information; a memory storing map information indicating the monitored area; a processor; and a communication interface for communicating with a camera, wherein the radar detection information includes radar-detected object identification information identifying a radar-detected object detected by the radar device, a detection time, and a position coordinate at which the radar-detected object was detected; the radar device receives camera detection information output by the camera after capturing an object; the camera detection information includes video information generated by the camera and imaged object identification information identifying an imaged object captured in the video information, the image capture time, and image capture condition information for specifying the position and field of view of the camera; the processor determines whether the radar-detected object and the imaged object are the same based on the position coordinate at which the radar-detected object was detected and the image capture condition information, and if it is determined that they are the same, links the radar-detected object identification information with the camera detection information; When a designation operation for a past time is accepted, the position of the radar-detected object at the designated time is superimposed on the map information and displayed on the monitor, and the video information of the imaged object determined to be identical to the radar-detected object at the designated time is displayed on the monitor.

[0216] (Supplementary Note 11) A method for controlling a monitoring system, comprising the steps of: acquiring radar detection information output by a radar device after irradiating an object in a monitoring area with radio waves and detecting the object based on the reflected waves of the radio waves reflected by the object, the radar detection information including radar-detected object identification information for identifying the radar-detected object detected by the radar device, a detection time, and a position coordinate at which the radar-detected object was detected; acquiring camera detection information output by a camera after capturing an image of the object, the camera detection information including video information generated by the camera and imaged object identification information for identifying the imaged object captured in the video information, the image capture time, and image capture condition information for specifying the position and field of view of the camera; determining whether the radar-detected object and the imaged object are the same based on the position coordinate at which the radar-detected object was detected and the image capture condition information, and if they are determined to be the same, linking the radar-detected object identification information with the camera detection information; a step of, when accepting an operation to specify a past time, superimposing the position of the radar-detected object at the specified time on map information indicating a monitored area and displaying on the monitor the video information of the imaged object determined to be identical to the radar-detected object at the specified time.

[0217] (Supplementary Note 12) A control program for causing a processor to execute the above-described method for controlling the monitoring system.

[0218] 10: Radar device, 11: Radar processor, 12: Memory, 13: Radar module, 14: Communication I / F, 15: Radar attribute determination unit, 16: Radar detection information processing unit, 20: Camera, 21: Camera processor, 22: Memory, 23: Imaging unit, 24: Communication I / F, 25: First camera attribute determination unit, 26: Second camera attribute determination unit, 27: Camera drive mechanism, 28: Camera detection information processing unit, 30: Server, 31: Server processor, 32: Memory, 33: Database, 34: Communication interface, 35: Calibration unit, 36: Detected object identification unit, 37: Display control unit, 38: Camera determination unit, 39: Monitoring function unit, 391: Intrusion determination unit, 392: Notification processing unit, 40: Recorder, 60: Operation device, 70: Monitor, 80: Security guard terminal, 81: Security drone, 90: Network, 100: Monitoring system, 100a: Monitoring system, 131: Radar IC, 151: AI calculation processing unit, 152: Learning model memory, 201: UI screen, 202: UI screen, 203: UI screen, 204: UI screen, 210: Radar detection range, 210a: Radar detection range, 210b: Radar detection range, 210c: Radar detection range, 210d: Radar detection range, 220: Imageable area, 221: Viewing area, 221a: Viewing area, 221aft: Viewing area, 221alt: Viewing area, 221bef: Viewing area, 231: Installation position, 232: Installation position position, 240: intruder, 241: radar detection point icon, 242: attribute icon, 242tr: movement trajectory, 242v: movement speed, 243: attribute icon, 245: camera screen, 245at: attribute information, 249: camera screen, 249at: attribute information, 251: AI calculation processing unit, 252: learning model memory, 261: camera image, 271: rotation motor, 272: zoom motor, 290: map screen, 290pat: map screen, 401: bird's-eye view map, 411: imageable range, 412: imageable range, 421: field of view, 422: field of view, 430: person, 431: person, 451: screen, 452: screen, 453: screen, 454: screen, 455: screen, 456: screen, 457: screen,500: Linking information table, 501: Records, 502: Records, 503: Records, 504: Records, 505: Records, 506: Records, 507: Records, 508: Records, 509: Records, 510: Records, 600: Imaging condition information table, 601: Records, 602: Records, 700: Point cloud trajectory information table, 701: Time records, 702: Records, 703: Records, 704: Records, 705: Records, 706: Records, 707: Records, 708: Records, 709: Records, 721: Viewing area, 722: Viewing area, 723: Viewing area, 731: Video information, 732: Video information, 741: Movement trajectory, 750: Playback screen, 800: Operation screen, 810: Main screen area, 820: Designated camera image area, 830: Alarm image area, 840: Seek bar, 841: Calendar bar, 850: Operation button area, 900: Event search screen, 910: Image gallery area, 911: Thumbnail images, 912: Thumbnail images, 913: Thumbnail images, 914: Thumbnail images, 920: Pagination area, 930: Display scale, 950: Event search screen, 960: Confirmation button, L1: No entry line, L2: Detection line

Claims

1. A surveillance system comprising: a radar device that irradiates radio waves to an object in a monitored area, detects the object based on the radio wave reflected by the object, and outputs radar detection information; a first camera that captures an image of the object and outputs camera detection information; a memory that stores map information indicating the monitored area; a processor; and a monitor, wherein the radar detection information includes radar-detected object identification information that identifies the radar-detected object detected by the radar device, the detection time, and the position coordinates at which the radar-detected object was detected, and the camera detection information includes video information generated by the first camera and imaged object identification information that identifies the imaged object captured in the video information, the image capture time, and image capture condition information for specifying the position and field of view of the first camera, and the processor determines whether the radar-detected object and the imaged object are the same based on the position coordinates at which the radar-detected object was detected and the image capture condition information, and if it determines that they are the same, links the radar-detected object identification information with the camera detection information, when a designation operation for a past time is accepted, the position of the radar-detected object at the designated time is superimposed on the map information and displayed on the monitor, and the video information of the imaged object determined to be identical to the radar-detected object at the designated time is displayed on the monitor.

2. A monitoring system as described in claim 1, further comprising a storage, wherein the processor determines whether the radar-detected object and the imaged object are the same based on the position coordinates where the radar-detected object was detected and the image-capturing condition information, and if it determines that they are the same, links the radar-detected object identification information with the camera detection information and records them in the storage.

3. A surveillance system as described in claim 1, further comprising a second camera that outputs video information of an object and imaging condition information indicating the angle of view at the time of imaging, wherein when an object imaged by the first camera moves from the imaging range of the first camera to the imaging range of the second camera, the processor transmits control information to the second camera to change the direction of the object so that it fits within the field of view of the second camera, and the processor links the radar-detected object identification information, the video information and imaging time generated by the first camera when the object is identical to the radar-detected object, and the video information and imaging time generated by the second camera when the object is identical to the radar-detected object.

4. A surveillance system as described in claim 3, wherein when the processor displays on the monitor the video information of the first camera capturing an image of the same object and the video information of the second camera capturing an image of the same object, the processor gives a relatively high priority to the video information capturing the front of the object and displays it on the monitor.

5. A monitoring system according to claim 1, wherein the processor acquires the radar detection information in chronological order, and displays on the monitor a movement trajectory showing the change in the position coordinates of the radar-detected object over time, superimposed on the map information.

6. A monitoring system according to claim 5, wherein when the position of the radar-detected object on the map information is moved, the processor plays back image information of the radar-detected object in conjunction with the movement of the position on the map information.

7. A surveillance system according to claim 1, wherein the processor displays the time of event occurrence, and displays on the monitor an image of the event-occurring object captured at that time, and the position of the event-occurring object superimposed on the map information.

8. The monitoring system according to claim 1, further comprising a server, wherein the memory and the processor are provided in the server.

9. A surveillance system according to claim 1, wherein said memory and said processor are provided in said radar device.

10. A radar device connectable to a monitor via a network, the radar device irradiating radio waves to an object in a monitored area, detecting the object based on the radio wave reflected by the object, and outputting radar detection information; a memory for storing map information indicating the monitored area; a processor; and a communication interface for communicating with a camera, wherein the radar detection information includes radar-detected object identification information for identifying the radar-detected object detected by the radar device, the detection time, and the position coordinates at which the radar-detected object was detected; the radar device receives camera detection information output by the camera after capturing an object; the camera detection information includes video information generated by the camera and imaged object identification information for identifying the imaged object captured in the video information, the image capture time, and image capture condition information for specifying the position and field of view of the camera; the processor determines whether the radar-detected object and the imaged object are the same based on the position coordinates at which the radar-detected object was detected and the image capture condition information, and if it is determined that they are the same, links the radar-detected object identification information with the camera detection information; When a designation operation for a past time is accepted, the position of the radar-detected object at the designated time is superimposed on the map information and displayed on the monitor, and the video information of the imaged object determined to be identical to the radar-detected object at the designated time is displayed on the monitor.

11. A method for controlling a surveillance system, comprising the steps of: acquiring radar detection information output by a radar device after irradiating an object in a surveillance area with radio waves and detecting the object based on the reflected waves of the radio waves reflected by the object, the radar detection information including radar-detected object identification information that identifies the radar-detected object detected by the radar device, the detection time, and the position coordinates at which the radar-detected object was detected; acquiring camera detection information output by a camera after capturing an image of the object, the camera detection information including video information generated by the camera and imaged object identification information that identifies the imaged object captured in the video information, the image capture time, and image capture condition information for specifying the position and field of view of the camera; determining whether the radar-detected object and the imaged object are the same based on the position coordinates at which the radar-detected object was detected and the image capture condition information, and if they are determined to be the same, linking the radar-detected object identification information with the camera detection information; a step of, when accepting an operation to specify a past time, superimposing the position of the radar-detected object at the specified time on map information indicating a monitored area and displaying on the monitor the video information of the imaged object determined to be identical to the radar-detected object at the specified time.

Images