Monitoring system and monitoring method

Abstract

This monitoring system determines, on the basis of radar position information and image capture position information, whether a detected object and an image-captured object are the same object or not, and notifies a user terminal of notification information. The notification information is information associating a first identifier for identifying the detected object, detected object attribute information, a camera image, and captured image attribute information if the detected object and the image-captured object are determined to be the same object, and is information based on at least any of radar installation position information, the detected object attribute information, the image capture position information, and the captured image attribute information if the detected object and the image-captured object are determined not to be the same object.

Description

Monitoring system and monitoring method

[0001] The present disclosure relates to a monitoring system and a monitoring method.

[0002] Patent Document 1 discloses a monitoring device that determines the position of an object within a monitoring area. The monitoring device in Patent Document 1 includes an object detection means, an imaging means, an image processing means, and a control means. The object detection means includes an antenna that transmits and receives radio waves with a predetermined beam width in multiple directions, detects the presence or absence of an object in the monitoring area from reflected waves of the transmitted waves, and obtains the angular range in which the object exists. When the object detection means detects an object, the control means sets a range on the image corresponding to the angular range as an inspection area and controls the pan-tilt-zoom mechanism so that the inspection area fits within the image. The image processing means processes the inspection area on the image to identify the angle at which the object exists.

[0003] Japanese Patent Application Publication No. 2017-181099

[0004] In Patent Document 1, the detection of an object in the monitoring area by the object detection means is a trigger for setting the inspection area by the imaging means. Therefore, if the object detection means cannot detect an object present in the monitoring area, a problem arises in that an alarm cannot be issued to notify the presence of the object. For this reason, it can be said that there is room for improvement over Patent Document 1 in terms of enabling the object detection means or the imaging means to correctly detect the position of the object even if either one of them detects an object.

[0005] The present disclosure has been devised in view of the above-described conventional circumstances, and aims to improve the detection accuracy of an object present in a monitored area.

[0006] The present disclosure provides a radar including an antenna unit that transmits radio waves to a first monitoring area and receives reflected waves of the radio waves, a detection unit that performs a detection process to detect the presence or absence of an object in the first monitoring area based on the reflected waves, a control unit that generates detected object attribute information indicating attributes of a detected object detected by the detection unit based on a result of the detection process, and generates radar position information indicating the position of the detected object based on first installation information including an antenna installation position and an antenna orientation and information on an antenna viewing angle, an imaging unit that images a second monitoring area at least partially overlapping with the first monitoring area, and obtains imaging position information indicating the position of an imaged object included in an image of the second monitoring area based on second installation information including the installation position and imaging direction of the imaging unit and information on the viewing angle of the imaging unit, and obtains imaged object attribute information indicating the attribute of the imaged object based on the image. and at least one camera having a processing unit for acquiring attribute information, the monitoring system having a determination unit that performs a determination process to determine whether the detected object and the captured image object are the same object based on the radar position information and the image capture position information, and a notification control unit that causes a notification unit that performs a notification process to notify a user of notification information, wherein the notification information is information that associates at least a first identifier that identifies the detected object, the detected object attribute information, the captured image, and the captured image object attribute information when the detected object and the captured image object are the same object, and is information based on at least one of the radar position information, the detected object attribute information, the image capture position information, and the captured image object attribute information when the detected object and the captured image object are not the same object.

[0007] The present disclosure also provides a method for detecting the presence or absence of an object in a first monitoring area based on the reflected waves, generating detected object attribute information indicating an attribute of a detected object detected by a detection unit that performs the detection process based on a result of the detection process, generating radar position information indicating the position of the detected object based on first installation information including an antenna installation position and an antenna orientation and information on an antenna viewing angle, capturing an image of a second monitoring area at least partially overlapping with the first monitoring area, and acquiring imaging position information indicating the position of an imaged object included in an image of the second monitoring area based on second installation information including an installation position and an imaging direction of the imaging unit and information on the viewing angle of the imaging unit, and and acquiring image-captured object attribute information indicating attributes of the image-captured object based on the radar position information, performing a determination process to determine whether the detected object and the image-captured object are the same object based on the radar position information and the image-captured position information, and causing a notification unit that performs a notification process for a user to notify the notification information, wherein, if the detected object and the image-captured object are the same object, the notification information is information that associates at least a first identifier that identifies the detected object, the detected object attribute information, the captured image, and the image-captured object attribute information, and, if the detected object and the image-captured object are not the same object, the notification information is information based on at least any of the radar position information, the detected object attribute information, the image-captured position information, and the image-captured object attribute information.

[0008] According to the present disclosure, it is possible to improve the accuracy of detecting an object present in a monitored area.

[0009] FIG. 1 is a diagram showing an example of the system configuration of a surveillance system. FIG. 2 is a block diagram showing an example of the internal configuration of a surveillance radar. FIG. 3 is a block diagram showing an example of the internal configuration of a fixed camera. FIG. 4 is a block diagram showing an example of the internal configuration of a PTZ camera. FIG. 5 is a diagram illustrating an example of the correspondence between a radar map and a radar coordinate system, and between a camera image and an image coordinate system. FIG. 6 is a diagram illustrating an example of a coordinate conversion between a radar coordinate system, a camera coordinate system, and an image coordinate system. FIG. 6 is a diagram illustrating an example of a coordinate conversion between a radar coordinate system, a camera coordinate system, and an image coordinate system. FIG. 7 is a flowchart showing an example of an initial setting operation procedure for aligning surveillance radar coordinates with camera coordinates. FIG. 8 is a flowchart showing an example of an operation procedure for aligning surveillance radar coordinates with camera coordinates. FIG. 9 is a flowchart showing an example of an operation procedure for aligning surveillance radar coordinates with camera coordinates. FIG. 10 is a table showing an example of changes in radar ID, camera ID, and server ID according to the alignment of surveillance radar coordinates with camera coordinates. FIG. 11 is a flowchart showing an example of an operation procedure for aligning surveillance radar coordinates with PTZ camera coordinates. Fig. 12 is a flowchart showing an example of an operation procedure for initial setting for displaying a list of detected objects and imaged objects. Fig. 13 is a flowchart showing an example of an operation procedure for displaying a list of detected objects and imaged objects. Fig. 14 is a diagram showing an example of a superimposed screen in which a monitoring radar detection area and a camera image are superimposed on a two-dimensional monitoring area map. Fig. 15 is a diagram showing an example of a superimposed screen in which a camera image is superimposed on a three-dimensional monitoring area map. Fig. 16 is a flowchart showing an example of an operation procedure for initial setting for intrusion detection of detected objects and imaged objects. Fig. 17 is a diagram showing an example of an alarm rule. Fig. 18 is a flowchart showing an example of an operation procedure for intrusion detection of detected objects and imaged objects.

[0010] Hereinafter, with appropriate reference to the accompanying drawings, a detailed description of embodiments specifically disclosing a monitoring system and a monitoring method according to the present disclosure will be provided. However, unnecessary detailed description may be omitted. For example, detailed description of well-known matters or redundant description of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter recited in the claims.

[0011] FIG. 1 is a diagram illustrating an example of the system configuration of a surveillance system 100. The surveillance system 100 of this embodiment detects the presence or absence of an object in a surveillance area, tracks the object, and provides notifications related to the object. The surveillance system 100 includes a surveillance radar 10, a camera 20, a server 30, a security robot 40, and a security guard terminal 50. The camera 20 includes, for example, a fixed camera 20A and a PTZ camera 20B. Note that the camera 20 may include only one of the fixed camera 20A and the PTZ camera 20B. The surveillance radar 10, the fixed camera 20A, and the PTZ camera 20B are connected to the server 30 via a network NW so as to be able to communicate data signals. The server 30, the security robot 40, and the security guard terminal 50 are connected to each other via the network NW so as to be able to communicate data signals.

[0012] The network NW may be a wired communication network (e.g., a wired LAN (Local Area Network) or a wired WAN (Wide Area Network)). The network NW may be a wireless communication network (e.g., Bluetooth (registered trademark), wireless LAN, LTE (Long Term Evolution), or 5G (fifth generation mobile communication system)). It is preferable that communication between the server 30 and the security robot 40 via the network NW be wireless communication. It is preferable that communication between the server 30 and the security guard terminal 50 via the network NW be wireless communication.

[0013] The surveillance radar 10 transmits radio waves toward a first monitoring area AR1 (see FIG. 14 ) and receives reflected waves of the transmitted radio waves. The surveillance radar 10 then detects the presence or absence of an object in the first monitoring area AR1 based on the reflected waves. Hereinafter, an object detected by the surveillance radar 10 may be simply referred to as a "detected object." A detailed configuration example of the surveillance radar 10 will be described later. The camera 20 captures an image of a second monitoring area AR2 (see FIG. 14 ) and performs analysis processing (e.g., image analysis) on the camera image CAP1 obtained by capturing the image of the second monitoring area AR2. Hereinafter, an object captured by the camera 20 may be simply referred to as an "imaged object." The surveillance system 100 may include two or more fixed cameras 20A and / or two or more PTZ cameras 20B. A detailed configuration example of the camera 20 will be described later with reference to FIGS. 3 and 4 .

[0014] The server 30 corresponds to an information processing device (i.e., a computer) that notifies a user terminal (e.g., a security robot 40, a security guard terminal 50, a monitor 70) of notification information, which is information to be notified to the user terminal regarding a detected object and / or an imaged object.

[0015] The security robot 40 is communicably connected to the server 30 via the network NW. The security robot 40 may be equipped with, for example, a camera, a speaker, lighting, etc. The security robot 40 moves and intimidates or warns detected objects or captured images using sound or lighting. The security robot 40 may capture images of the detected objects or captured images and send the captured images to the server 30. The security robot 40 corresponds to, for example, a multicopter-type unmanned aerial vehicle (a so-called drone), a robot that can move autonomously based on a control signal, or the like.

[0016] The security guard terminal 50 is an information processing device carried by a security guard, and is communicatively connected to the server 30 via the network NW. The security guard terminal TP1 is realized by, for example, a portable information processing device such as a tablet terminal or a smartphone. Note that the monitoring system 100 may include two or more security guard terminals 50.

[0017] Next, the internal configuration of the server 30 will be described. The server 30 is realized by an information processing device such as a PC (Personal Computer), and includes a server processor 31, a memory 32, a database 33, and a communication unit 34. The server 30 is also electrically connected to the operation device 60 and the monitor 70. The database 33 may be implemented in an information processing device different from the server 30, and may be connected to the server 30 so as to enable communication of data signals therebetween.

[0018] The server processor 31 is a computing device such as a central processing unit (CPU), a graphical processing unit (GPU), or a field programmable gate array (FPGA), and functions as a controller that controls the overall operation of the server 30 .

[0019] The memory 32 includes, for example, a RAM (Random Access Memory) and a ROM (Read Only Memory). The RAM is a working memory for the server processor 31 and temporarily stores data or information generated or acquired by the server processor 31. The ROM stores programs and control data that define the operation of the server processor 31. The memory 32 may further include a storage device such as a flash memory, an SSD (Solid State Drive), or an HDD (Hard Disk Drive).

[0020] The database 33 is a storage device such as an HDD or SSD, and stores various types of information. Identification information (e.g., serial number, ID, etc.) of the surveillance radar 10 and identification information (e.g., serial number, ID, etc.) of the camera 20 may be registered (saved).

[0021] The communication unit 34 is a communication circuit that executes communication of data signals between the surveillance radar 10, the fixed camera 20A, the PTZ camera 20B, the security robot 40, and the security guard terminal 50 via the network NW.

[0022] The operation device 60 is an input device that inputs data signals to the server 30, and corresponds to a portable information processing terminal or the like operated by a user (for example, a security guard engaged in monitoring work using the monitoring system 100; the same applies below). The monitor 70 corresponds to a display device that displays the data signals output from the server 30. If the operation device 60 is a touch panel, the operation device 60 and the monitor 70 may be configured as an integrated unit.

[0023] In this embodiment, the server processor 31 (an example of a judgment unit) performs a judgment process to determine whether the detected object detected in the first monitoring area AR1 and the captured object included in the camera image of the second monitoring area AR2 are the same object.

[0024] In addition, in this embodiment, the server processor 31 (an example of a notification information control unit) generates notification information for a user terminal (e.g., a security robot 40, a security guard terminal 50, a monitor 70, etc.) and notifies the user terminal, etc. of an alarm being issued via the communication unit 34.

[0025] Next, the surveillance radar 10 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the internal configuration of the surveillance radar 10. The surveillance radar 10 (an example of a radar) includes at least a radar processor 11, a memory 12, a detection unit 13, a communication unit 14, and an antenna unit An.

[0026] The radar processor 11 is realized by, for example, a CPU, GPU, or FPGA, and functions as a controller that manages the overall operation of the surveillance radar 10. The memory 12 includes a RAM and a ROM. The RAM is a working area used by the radar processor 11 for calculations and temporarily stores data or information generated or acquired by the radar processor 11. The ROM stores programs and control data that define the operation of the radar processor 11. The memory 12 may further include a storage device such as a flash memory, SSD, or HDD.

[0027] The radar processor 11 has an AI processing unit 15 that executes processing using AI (Artificial Intelligence). The AI ​​processing unit 15 includes an AI calculation processing unit 151 and a learning model memory 152. In other words, the surveillance radar 10 can execute various types of processing using AI.

[0028] The AI ​​calculation processing unit 151 loads a trained model from the training model memory 152 and forms a neural network specialized for processing the loaded trained model. The training model memory 152 is realized by, for example, a flash memory, and stores trained models generated in advance by training processing.

[0029] The trained model in this embodiment corresponds to, for example, a model that causes AI to execute a process for determining the type of detected object, specifically, whether the detected object is a person, a car, or a motorcycle, based on the results of the detection process. Note that a trained model that causes AI to determine the movement speed of the detected object based on the results of the detection process may also be used. Information regarding the type of detected object is an example of the detected object attribute information in this embodiment.

[0030] The detection unit 13 includes n radar ICs 131, ..., 13n (n: a constant that is an integer greater than or equal to 1). In the following description, when it is not necessary to distinguish between the radar ICs 131, ..., 13n, they may be referred to as "radar ICs 13n." Each radar IC 13n is a communication circuit that controls the emission of radio waves, for example, with a wavelength of approximately 1 to 10 mm, from a transmitting antenna and the reception of radio waves from a receiving antenna, and is capable of beamforming radio waves or reflected waves into a directional area (see below) corresponding to the radar IC. Each radar IC 131, ..., 13n is configured with a different directional area. In this embodiment, the directional area is, for example, a fan-shaped area centered on the radar installation position when viewing the surveillance radar 10 from above (see FIG. 5 ). The radar IC 13n performs a detection process to detect the presence and location of an object based on the reflected waves received from the directional area. The radar IC 13n can obtain information regarding the movement speed of the detected object based on the results of the detection process. Information regarding the movement speed of the detected object is an example of detected object attribute information in this embodiment.

[0031] The antenna unit An has n (n: a constant that is an integer greater than or equal to 1) transmitting antenna units ATx1, ..., ATxn and n receiving antenna units ARx1, ..., ARxn. The antenna unit An includes n pairs of one transmitting antenna unit and one receiving antenna unit. Specifically, the first pair is made up of the transmitting antenna unit ATx1 and the receiving antenna unit ARx1, ..., and the nth pair is made up of the transmitting antenna unit ATxn and the receiving antenna unit ARxn. A radar IC is provided corresponding to each of these pairs. In other words, the radar IC 131 is connected to the transmitting antenna unit ATx1 and the receiving antenna unit ARx1 of the first pair. Furthermore, the radar IC 13n is connected to the transmitting antenna ATxn and the receiving antenna ARxn of the nth pair. The transmitting antenna units ATx1, ..., ATxn and the receiving antenna units ARx1, ..., ARxn each include one antenna or multiple antennas.

[0032] The transmitting antenna units ATx1, ..., ATxn convert analog signals from the corresponding radar ICs 131, ..., 13n into radio waves and radiate them.

[0033] The receiving antenna units ARx1, ..., ARxn each receive a reflected wave that is the radio wave emitted from the corresponding transmitting antenna and reflected by an object. The receiving antenna units ARx1, ..., ARxn convert the received reflected wave into an analog signal and send it to the corresponding radar IC 131, ..., 13n.

[0034] The communication unit 14 is a communication circuit that executes communication of data signals with the fixed camera 20A, the PTZ camera 20B, or the server 30.

[0035] With the configuration described above, in the surveillance radar 10, the radar processor 11 (an example of a judgment unit) performs a judgment process to determine whether the detected object detected in the first surveillance area AR1 and the imaged object included in the camera image of the second surveillance area AR2 are the same object.

[0036] In addition, the radar processor 11 (an example of a notification information control unit) generates notification information to be sent to a user terminal (for example, a security robot 40, a security guard terminal 50, or a monitor 70).

[0037] The memory 12 may store information related to a first monitoring area AR1 (see FIG. 14 ) indicating an area in which the monitoring radar 10 can detect the position of a detected object, first installation information including the radar installation position (coordinates) and radar direction, radar field of view angle information, and information on an alarm rule (see FIG. 17 ). The memory 12 may also store information related to the position of a detected object, detected object attribute information, information related to the position of an imaged object, imaged object attribute information, etc. The memory 12 may also store information on a calculation formula (see FIG. 6 ) used for mutual coordinate conversion between coordinates in the radar coordinate system RCS, coordinates in the camera coordinate system CCS, and coordinates in the image coordinate system ICS.

[0038] Next, the configuration of the camera 20 will be described with reference to Figures 3 and 4. First, the configuration of the fixed camera 20A will be described with reference to Figure 3. Figure 3 is a block diagram showing an example of the internal configuration of the fixed camera 20A.

[0039] The fixed camera 20A includes at least a camera processor 21A, a memory 22A, an imaging unit 23A, and a communication unit 24A. The fixed camera 20A corresponds to a camera whose angle of view cannot be changed after installation, such as a box camera or a dome camera.

[0040] The camera processor 21A is, for example, a CPU, GPU, or FPGA, and functions as a controller that controls the overall operation of the fixed camera 20A. The memory 22A includes a RAM and a ROM. The RAM is a working area used by the camera processor 21A for calculations, and temporarily stores data or information generated or acquired by the camera processor 21A. The ROM stores programs and control data that define the operation of the camera processor 21A.

[0041] The memory 22A may further include a storage device such as a flash memory, SSD, or HDD. The memory 22A stores information related to the second monitoring area AR2 (see FIG. 14), imaging position information including the camera installation position (coordinates) and camera orientation, camera field of view angle information, and alarm rules (see FIG. 17). The memory 22A also stores information such as the position of an object detected by the AI ​​processing unit 25A and imaged object attribute information. The memory 22A also stores information on a calculation formula (see FIG. 6) used for mutual coordinate conversion between coordinates in the radar coordinate system RCS, coordinates in the camera coordinate system CCS, and coordinates in the image coordinate system ICS.

[0042] The camera processor 21A has an AI processing unit 25A that executes processing using AI. The AI ​​processing unit 25A includes an AI calculation processing unit 251A and a learning model memory 252A. In other words, the fixed camera 20A can execute various types of processing using AI.

[0043] The AI ​​calculation processing unit 251A loads a trained model from the training model memory 252A and forms a neural network specialized for processing the loaded trained model. The training model memory 252A is realized by, for example, a flash memory, and stores trained models generated in advance by training processing. The trained model in the training model memory 252A may be, for example, a model for detecting objects in image data captured by the imaging unit 23A. Furthermore, the trained model in the training model memory 252A may be a model for performing attribute classification to extract imaged object attribute information related to objects in image data captured by the imaging unit 23A.

[0044] The imaging unit 23A captures an image of a subject (e.g., a person, a vehicle, or the like; the same applies below) within the second monitoring area AR2 (see FIG. 14). The imaging unit 23A includes a lens and an image sensor, and captures an optical image of the subject by receiving incident light that enters the lens with the image sensor.

[0045] The communication unit 24A is a communication circuit that executes communication of data signals with the surveillance radar 10, the PTZ camera 20B, or the server 30.

[0046] Next, the configuration of the PTZ camera 20B will be described with reference to Fig. 4. Fig. 4 is a block diagram showing an example of the internal configuration of the PTZ camera 20B.

[0047] The PTZ camera 20B includes at least a camera processor 21B, a memory 22B, an imaging unit 23B, a communication unit 24B, and a camera driving unit 26B. The PTZ camera 20B has a pan-tilt-zoom mechanism and corresponds to a camera whose angle of view can be changed after installation.

[0048] The camera processor 21B is, for example, a CPU, GPU, or FPGA. The camera processor 21B functions as a controller that manages the overall operation of the PTZ camera 20B. The memory 22B includes a RAM and a ROM. The RAM is a working area used by the camera processor 21B for calculations, and temporarily stores data or information generated or acquired by the camera processor 21B. The ROM stores programs and control data that define the operation of the camera processor 21B.

[0049] The memory 22B may further include a storage device such as a flash memory, SSD, or HDD. The memory 22B stores information related to the second monitoring area AR2 (see FIG. 14), imaging position information including the camera installation position (coordinates) and camera orientation, camera field of view angle information, and alarm rules (see FIG. 17). The memory 22B also stores information such as the position of an object detected by the AI ​​processing unit 25B and imaged object attribute information. The memory 22B also stores information on a calculation formula (see FIG. 6) used for mutual coordinate conversion between coordinates in the radar coordinate system RCS, coordinates in the camera coordinate system CCS, and coordinates in the image coordinate system ICS.

[0050] The camera processor 21B has an AI processing unit 25B that can perform predetermined signal processing using AI. The AI ​​processing unit 25B includes an AI calculation processing unit 251B and a learning model memory 252B. In other words, the PTZ camera 20B can perform various types of processing using AI.

[0051] The AI ​​calculation processing unit 251B loads a trained model from the training model memory 252B and forms a neural network specialized for processing the loaded trained model. The training model memory 252B is realized, for example, by a flash memory and stores a trained model generated in advance by a training process. The training model memory 252B is realized, for example, by a flash memory and stores a trained model generated in advance by a training process. The trained model in the training model memory 252B may be, for example, a model for detecting an object in image data captured by the imaging unit 23B. Furthermore, the trained model in the training model memory 252B may be a model for performing attribute classification to extract imaged object attribute information regarding an object in image data captured by the imaging unit 23B.

[0052] The imaging unit 23B captures an image of a subject (e.g., a person, a vehicle, or other detected object; the same applies below) within the second monitoring area AR2 (see FIG. 14). The imaging unit 23B includes a lens and an image sensor, and captures an optical image of the subject by the image sensor receiving incident light that enters the lens.

[0053] The communication unit 24B is a communication circuit that executes communication of data signals with the surveillance radar 10, the fixed camera 20A, or the server 30.

[0054] The camera driver 26B includes a rotation motor 261 and a zoom motor 262. The rotation motor 261 is a driver for panning and / or tilting the housing of the PTZ camera 20B. The rotation motor 261 performs panning and tilting by driving in accordance with the motor position calculated by the camera processor 21B. The zoom motor 262 is a driver for driving the zoom lens included in the imaging unit 23. The zoom motor 262 changes the optical magnification of the lens included in the imaging unit 23 by driving in accordance with the motor position calculated by the camera processor 21B. In this way, the camera driver 26B controls the panning and tilting of the housing of the PTZ camera 20B and also controls zoom processing using the zoom lens included in the imaging unit 23B.

[0055] The camera processor 21B calculates a target position of the rotary motor 261 for the imaging unit 23 to capture an image of a detected object located at a first position (e.g., a position where a moving object is detected) indicated by the radar position information detected by the surveillance radar 10. At this time, the camera processor 21B may calculate, along with the target position of the rotary motor 261, a target position of the zoom motor 262 suitable for the imaging unit 23B to capture an image of the detected object located at the first position. Specifically, the camera processor 21B calculates the target position of the rotary motor 261 based on the relationship between the position (e.g., current position) of the rotary motor 261 held by the PTZ camera 20B and the imaging range in real space (e.g., the second monitoring area AR2), and the second installation information (e.g., the camera installation position and camera imaging direction of the PTZ camera 20B). In addition, the camera processor 21B calculates the target position of the zoom motor 262 based on the relationship between the position (e.g., current position) of the zoom motor 262 held by the PTZ camera 20B and the imaging range in real space (e.g., second monitoring area AR2), and the second installation information (e.g., the camera installation position and camera imaging direction of the PTZ camera 20B).

[0056] Next, the correspondence between the radar map RMP1 and the camera image IMG1 in this embodiment will be described with reference to Fig. 5. Fig. 5 is a diagram illustrating the correspondence between the radar map RMP1 and the radar coordinate system RCS, and the correspondence between the camera image IMG1 and the image coordinate system ICS. Fig. 5 illustrates the camera image IMG1 when one person Ps1 and two vehicles Vc1 and Vc2 are captured by the camera 20 in the second monitoring area AR2.

[0057] The camera image IMG1 is an image captured by the camera 20. FIG. 5 illustrates an example of an image coordinate system ICS when the upper left corner of the camera image IMG1 is the origin Oics. In the camera image IMG1, the position of the person Ps1 and the positions of the vehicles Vc1 and Vc2 are represented as coordinates in the image coordinate system ICS. For example, the position of the person Ps1 is represented by coordinates (u1, v1) in the image coordinate system ICS. The image coordinate system ICS is a two-dimensional coordinate system. In FIG. 5, the horizontal axis of the camera image IMG1 corresponds to the u axis of the image coordinate system ICS, and the vertical axis of the camera image IMG1 corresponds to the v axis of the image coordinate system ICS.

[0058] The radar map RMP1 shows the detection results of the surveillance radar 10 in the first monitoring area AR1 when the installation position of the surveillance radar 10 is set as the origin Orcs of the radar coordinate system RCS. The radar map RMP1 visually represents the detection results of the surveillance radar 10. FIG. 5 illustrates an example of the radar map RMP1 when the installation position of the surveillance radar 10 is set as the origin Orcs of the radar coordinate system RCS. In the radar map RMP1, the position of a person Ps1 and the positions of vehicles Vc1 and Vc2 are represented as coordinates in the radar coordinate system RCS. For example, the position of person Ps1 is represented by coordinates (x1, y1, z1) in the radar coordinate system RCS. The radar coordinate system RCS is a three-dimensional coordinate system. 5, the depth direction of the radar map RMP1 is represented as the z-axis of the radar coordinate system RCS (the vertical axis in FIG. 5), one of the planes perpendicular to the depth direction of the radar map RMP1 (the horizontal axis in FIG. 5) is represented as the x-axis of the radar coordinate system RCS, and the other plane perpendicular to the z-axis of the radar map RMP1 is represented as the y-axis of the radar coordinate system RCS. Note that the radar map RMP1 in FIG. 5 illustrates the (z-axis, x-axis) direction components of the radar coordinate system RCS as the detection results of the surveillance radar 10.

[0059] Here, with reference to Figures 6A and 6B, an overview of the coordinate conversion between the radar coordinate system RCS, the camera coordinate system CCS, and the image coordinate system ICS in this embodiment will be described. Figure 6A is a diagram showing an overview example of the coordinate conversion between the radar coordinate system RCS and the camera coordinate system CCS. Figure 6B is a diagram showing an overview example of the coordinate conversion between the camera coordinate system CCS and the image coordinate system ICS. First, with reference to Figure 6A, an overview example of the coordinate conversion between the radar coordinate system RCS and the camera coordinate system CCS will be described. For example, as shown in Figure 6A, coordinate conversion between the radar coordinate system RCS and the camera coordinate system CCS can be performed using external parameters of the camera (see equation (1)).

[0060]

[0061] In formula (1), [Xc, Yc, Zc] are camera coordinates in the camera coordinate system CCS. That is, [Xc, Yc, Zc] correspond to a position on the camera coordinate system CCS as viewed from the origin of the camera coordinate system CCS (i.e., the installation position of the camera 20). [Xw, Yw, Zw] are radar coordinates in the radar coordinate system RCS. That is, [Xw, Yw, Zw] correspond to a position on the radar coordinate system RCS as viewed from the origin of the radar coordinate system RCS (i.e., the installation position of the radar for the surveillance radar 10). From equation (1), therefore, [Xc, Yc, Zc] corresponds to the sum of the product of [Xw, Yw, Zw] and an external parameter indicating a rotation matrix for rotating [Xw, Yw, Zw], which indicates the radar coordinates in the radar coordinate system RCS, and the external parameter [t1, t2, t3] indicating the movement (translation) of the origin of the radar coordinate system RCS to the origin of the camera coordinate system CCS. As shown in equation (1), any position (coordinate) on the radar coordinate system RCS can be converted to a position (coordinate) on the camera coordinate system CCS. Furthermore, any position (coordinate) on the camera coordinate system CCS can be converted to a position (coordinate) on the radar coordinate system RCS by transforming equation (1) to find the inverse matrix of the rotation matrix.

[0062] Next, a summary example of the coordinate transformation between the camera coordinate system CCS and the image coordinate system ICS will be described with reference to Fig. 6B. As shown in Fig. 6B, by using the internal parameters of the camera, the coordinate transformation between the camera coordinate system CCS and the image coordinate system ICS can be performed (see Equation (2)).

[0063]

[0064] As described above, the image coordinate system ICS is a two-dimensional coordinate system. The following description will be made assuming that the image coordinate system ICS is defined by an image plane projected from the origin of the camera coordinate system CCS (i.e., the installation position of the camera 20) to a position at a depth s. In this case, the position (u, v) on the image coordinate system ICS as viewed from the origin Oics of the image coordinate system ICS (see FIG. 5) is expressed as s[u, v, 1] (see equation (2)). In other words, in equation (2), s[u, v, 1] corresponds to an element obtained by introducing the coordinates on the image of the image coordinate system ICS into a coordinate system of the same dimension as the camera coordinate system CCS. [Xc, Yc, Zc] are coordinates of the camera coordinate system CCS. That is, [Xc, Yc, Zc] corresponds to a position on the camera coordinate system CCS as viewed from the origin of the camera coordinate system CCS (i.e., the installation position of the camera 20). Furthermore, fx and fy are internal parameters indicating the focal length of the camera 20. cx and cy are internal parameters indicating the optical center of the camera 20. Therefore, from equation (2), the depth s corresponds to the Z-axis element (Zc) of the camera coordinate system CCS. In this way, as shown in equation (2), coordinates on the camera coordinate system CCS, which is a three-dimensional coordinate system, can be transformed into coordinates on the image coordinate system ICS, which is a two-dimensional coordinate system.

[0065] As described above with reference to Figures 6A and 6B, the radar coordinate system RCS can be coordinate-transformed into the camera coordinate system CCS. Furthermore, the camera coordinate system CCS can be coordinate-transformed into the image coordinate system ICS. In other words, the coordinates of the radar coordinate system RCS can be expressed as coordinates of the image coordinate system ICS.

[0066] Returning to Figure 5, we will continue to explain the correspondence between the radar map RMP1 and the camera image IMG1. First, we will explain the correspondence between the radar map RMP1 and the camera image IMG1 in the surveillance radar 10. In this embodiment, the radar processor 11 (surveillance radar 10) can convert coordinates in the radar coordinate system RCS, which indicate the positions of person Ps1 and vehicles Vc1 and Vc2, into coordinates in the image coordinate system ICS. Specifically, the surveillance radar 10 converts the coordinates (x1, y1, z1) in the radar coordinate system RCS for person Ps1 into coordinates (u1, v1) in the image coordinate system ICS using equations (1) and (2).

[0067] The surveillance radar 10 sends to the server 30 the coordinates of person Ps1 and the coordinates of vehicles Vc1 and Vc2 in the image coordinate system ICS after transformation processing from the radar coordinate system RCS to the image coordinate system ICS. The radar processor 11 (surveillance radar 10) may superimpose information indicating person Ps1 (such as a frame Fp1) on the coordinates of person Ps1 in the image coordinate system ICS (see FIG. 5). The radar processor 11 (surveillance radar 10) may also superimpose information indicating vehicles Vc1 and V (such as a frame Fv1 and a frame Fv2) on the coordinates of vehicles Vc1 and Vc2 in the image coordinate system ICS (see FIG. 5). The coordinates of the image coordinate system ICS on which the information such as the frame Fp1, frame Fv1, and frame Fv2 is superimposed are sent from the surveillance radar 10 to the server 30. The server 30 may display on the monitor 70 or the like a superimposed image in which information such as the frame Fp1, the frame Fv1, the frame Fv2, etc. is superimposed on the radar map RMP1 and the camera image IMG1.

[0068] It is also possible to store data of a three-dimensional radar map RMP1 corresponding to the radar coordinate system RCS in the camera 20, and use equations (1) and (2) in the camera 20 to convert coordinates (x1, y1, z1) on the radar coordinate system RCS into coordinates (u1, v1) on the image coordinate system ICS.

[0069] Next, the correspondence between the radar map RMP1 and the camera image IMG1 in the camera 20 will be described. The camera 20 uses equations (1) and (2) to convert the coordinates in the image coordinate system ICS related to the person Ps1 and the vehicles Vc1 and Vc2 into positions (coordinates) in the radar coordinate system RCS. Specifically, the camera 20 converts the coordinates (u1, v1) of the person Ps1 in the image coordinate system ICS into coordinates (x1, y1, z1) in the radar coordinate system RCS.

[0070] The camera 20 transmits to the surveillance radar 10 the coordinates of the person Ps1 and the coordinates of the vehicles Vc1 and Vc2 in the image coordinate system ICS after transformation processing from the radar coordinate system RCS to the image coordinate system ICS. The surveillance radar 10 stores data of a three-dimensional radar map RMP1 corresponding to the radar coordinate system RCS in the memory 12 or the like. The radar processor 11 (surveillance radar 10) may superimpose information indicating the person Ps1 (such as a frame Fp1) on the coordinates of the person Ps1 in the radar coordinate system RCS (see FIG. 5). The radar processor 11 (surveillance radar 10) may also superimpose information indicating the vehicles Vc1 and Vc2 (such as a frame Fv1 and a frame Fv2) on the coordinates of the vehicles Vc1 and Vc2 in the radar coordinate system RCS (see FIG. 5). The coordinates of the image coordinate system ICS on which the information such as the frame Fp1, frame Fv1, and frame Fv2 is superimposed are transmitted from the surveillance radar 10 to the server 30. The server 30 may display on the monitor 70 or the like a superimposed image in which information such as the frame Fp1, the frame Fv1, the frame Fv2, etc. is superimposed on the radar map RMP1 and the camera image IMG1.

[0071] In addition, the external parameters and internal parameters of the camera 20 may be stored in the surveillance radar 10, and the surveillance radar 10 may use equations (1) and (2) to convert coordinates (u1, v1) on the image coordinate system ICS into coordinates (x1, y1, z1) on the radar coordinate system RCS.

[0072] The database 33 may also store information indicating coordinates indicating a predetermined range, position, etc. in the radar coordinate system RCS, and / or coordinates indicating a predetermined range, position, etc. in the camera coordinate system CCS.

[0073] In this embodiment, it is possible to accurately convert coordinates in the image coordinate system ICS to coordinates in the camera coordinate system CCS based on the distance from the origin of the camera coordinate system CCS to the positions (coordinates) of any two points on the projection surface (in other words, the camera image CAP1) that are a focal length f away, the focal length f, and the actual distance (measured value, etc.) between any two points on the camera image CAP1. For example, the any two points on the camera image CAP1 are the points corresponding to the top of the person's head and the points corresponding to the feet of a person based on their height (average). In addition, in the case of a vehicle, the any two points on the camera image CAP1 are the points corresponding to the top of the vehicle and the points corresponding to the bottom of the wheels. In the case of a vehicle, it is also possible to estimate the vehicle type based on the vehicle height.

[0074] Next, an operational procedure for associating the coordinates of a detected object in the radar coordinate system with the coordinates of an imaged object in the camera coordinate system in the monitoring system 100 will be described. FIG. 7 is a flowchart showing an example of an initial setting operational procedure for aligning the coordinates of the monitoring radar 10 with the coordinates of the camera 20. FIGS. 8 and 9 are flowcharts showing an example of an operational procedure for aligning the coordinates of the monitoring radar 10 with the coordinates of the camera 20. FIG. 10 is a table illustrating an example of changes in the radar ID, camera ID, and server ID according to the alignment of the coordinates of the monitoring radar 10 with the coordinates of the camera 20. Note that in FIG. 10 , the values ​​and number of digits of the radar ID, camera ID, and server ID are not limited to those shown. In the explanations of FIGS. 7 to 10 , an example will be described in which a fixed camera 20A is used as the camera 20.

[0075] 7 , a user operates the operation device 60 to input information (an example of first installation information) including information on the radar installation position and radar orientation on a map of the surveillance radar 10 (e.g., a site map of the monitoring area) to the server processor 31 (step St1). The radar viewing angle information is information stored in the surveillance radar 10, but the server 30 may also hold the radar viewing angle information of the surveillance radar 10 in advance. Note that a GPS (Global Positioning System) receiver may be provided in the surveillance radar 10, and the position information of the surveillance radar 10 measured by the GPS may be shared by sending it from the surveillance radar 10 to the server 30 as radar installation position information.

[0076] By the user operating the operation device 60, information (an example of second installation information) including information on the camera installation position of the fixed camera 20A on a map (e.g., a site map of the monitoring area) and information on the camera imaging direction is input to the server processor 31 (step St2). Although the information on the camera viewing angle is information stored in the camera 20, the server 30 may previously store information on the camera viewing angle of the fixed camera 20A. Note that a GPS receiver may be provided in the fixed camera 20A, and the position information of the fixed camera 20A measured by GPS may be shared by sending it to the server 30 as information on the camera installation position.

[0077] The server processor 31 generates and sets a calculation formula using the information acquired in steps St1 and St2 (step St3). In step St3, the server processor 31 generates and sets calculation formulas (formulas (1) and (2): see FIGS. 6A and 6B ) for converting the coordinates of the surveillance radar 10 and the coordinates of the fixed camera 20A into coordinates in an arbitrary coordinate system (e.g., a radar coordinate system RCS). Therefore, in step St3, the server processor 31 generates the calculation formula using information on the radar installation position, radar direction, and radar viewing angle of the surveillance radar 10, and the camera installation position, camera imaging direction, and camera viewing angle of the fixed camera 20A. The server 30 sends the information on the calculation formula generated in step St3 to the fixed camera 20A and the surveillance radar 10. Step St3 completes the series of initialization processes shown in FIG. 7 . The arbitrary coordinate system may be not only a three-dimensional coordinate system (radar coordinate system RCS or camera coordinate system CCS) but also a two-dimensional image coordinate system ICS.

[0078] Next, the operational procedure for aligning the coordinates of the surveillance radar 10 with the coordinates of the camera 20 will be described with reference to Fig. 8. As shown in Fig. 8, the surveillance system 100 may execute a series of processes (steps StR11 to StR15) performed by the surveillance radar 10 and a series of processes (steps StC11 to StC15) performed by the fixed camera 20A in parallel.

[0079] First, a series of processes performed by the surveillance radar 10 will be described. The surveillance radar 10 emits radio waves in the first surveillance area AR1 (step StR11). The surveillance radar 10 receives reflected waves of the radio waves emitted in step StR11 (step StR12). The reflected waves received by the surveillance radar 10 in step StR12 correspond to the waves reflected when the radio waves emitted by the surveillance radar 10 in step StR11 are reflected by an object present in the first surveillance area AR1.

[0080] The surveillance radar 10 inputs the reflected waves received in step StR12 to the radar IC 13n. The radar IC 13n performs signal processing using the reflected waves to detect objects present within the first monitoring area AR1. Of the objects present within the first monitoring area AR1, those detected by the radar IC 13n correspond to the detected objects. The results of the signal processing by the radar IC 13n are sent to the radar processor 11. The radar processor 11 acquires coordinates in a radar coordinate system RCS indicating the position of the detected object based on the results of the signal processing by the radar IC 13n. The radar processor 11 may also acquire the movement speed and type of the detected object as detected object attribute information based on the signal processing results of the radar IC 13n. The radar processor 11 may also acquire the movement speed of the detected object calculated by the radar IC 13n. Then, the radar processor 11 assigns a radar ID that identifies the detected object to the object detected by the radar IC 13n among the objects present in the first monitoring area AR1 (objects 1, 2, 4, and 5 in table TBL0, see Figure 10), and associates the detected object attribute information with the radar ID (step StR13).

[0081] In step StR13, object 1 is assigned a radar ID of "1001." Object 2 is assigned a radar ID of "1002." Object 4 is assigned a radar ID of "1003." Object 5 is assigned a radar ID of "1001" (see FIG. 10).

[0082] The radar processor 11 uses the calculation formula generated in step St3 (see FIG. 7 ) to convert the coordinates in the radar coordinate system RCS, which indicate the position of the detected object assigned the radar ID, into coordinates in an arbitrary coordinate system (step StR14). In the following description, the coordinates after converting the coordinates in the radar coordinate system RCS, which indicate the position of the detected object, into an arbitrary coordinate system may be referred to as "converted detected object coordinates." Note that if the radar coordinate system RCS is an arbitrary coordinate system, the processing of step StR14 can be omitted. The radar processor 11 sends the converted detected object coordinates and detected object attribute information for each detected object assigned a radar ID to the server 30 (step StR15). Next, the monitoring system 100 performs the operation of step StS11 (see FIG. 9 ). The coordinates in the radar coordinate system RCS, which indicate the position of the detected object, are an example of radar position information in this embodiment. Furthermore, the converted detected object coordinates are an example of radar position information in this embodiment.

[0083] Next, a series of processes performed by the fixed camera 20A will be described. The fixed camera 20A captures an image of the second monitoring area AR2 (step StC11). In step StC11, the image of the second monitoring area AR2 captured by the fixed camera 20A (camera image CAP1) is input from the imaging unit 23A to the camera processor 21A. Next, the camera processor 21A causes the AI ​​processing unit 25A to perform image analysis of the camera image CAP1. The camera processor 21A then acquires the position (coordinates in the image coordinate system ICS) of an object included in the camera image CAP1 based on the analysis results of the camera image CAP1 by the AI ​​processing unit 25A (step StC12). The AI ​​calculation processing unit 251A performs, for example, a process to determine the presence or absence of an object included in the camera image CAP1. Of the objects present in the second monitoring area AR2, an object recognized as an object included in the camera image CAP1 through image analysis by the AI ​​processing unit 25A corresponds to the captured object. That is, in step StC12, the camera processor 21A acquires the position of the captured object. Then, the camera processor 21A assigns a camera ID for identifying the captured object to the object included in the camera image CAP1 among the objects present in the second monitoring area AR2 (step StC12). The camera ID is an example of the second identifier in this embodiment.

[0084] Next, the camera processor 21A causes the AI ​​processing unit 25A to perform attribute classification to extract image-captured object attribute information. The camera processor 21A associates the image-captured object attribute information obtained as a result of the attribute classification process by the AI ​​processing unit 25A with the image-captured object position and camera ID (objects 1, 3, and 5 in table TBL0, see FIG. 10, step StC13).

[0085] In this embodiment, the attributes of the imaged object refer to characteristic elements of the imaged object. For example, the attributes of the imaged object are at least one of the type of imaged object, gender, age group, height, clothing color, vehicle model, vehicle color, license plate, a score indicating the accuracy (attribute-likeness) when classifying the imaged object attribute information, and the object's movement speed. The type of imaged object indicates whether the imaged object is a person, a vehicle, a motorcycle, an animal, etc. The vehicle model indicates the type of vehicle, such as a sedan, a wagon, a minivan, etc. The AI ​​calculation processing unit 251A performs processing such as determining whether the imaged object is a person, a car, or a motorcycle using the position information of the imaged object and the imaged object attribute information.

[0086] In step StC13, object 1 is assigned a camera ID of "2001", object 3 is assigned a camera ID of "2002", and object 5 is assigned a camera ID of "2003" (see FIG. 10).

[0087] The camera processor 21A uses the calculation formula of step St3 (see FIG. 7 ) to convert the coordinates of the image coordinate system ICS, which indicates the position of the imaged object assigned the camera ID, into coordinates of an arbitrary coordinate system (step StC14). As described above, the arbitrary coordinate system in this embodiment corresponds to the radar coordinate system RCS, i.e., the coordinate system of the surveillance radar 10. In the following description, the coordinates of the radar coordinate system RCS, which indicate the position of the imaged object after conversion to the arbitrary coordinate system, may be referred to as "converted imaged object coordinates." In step StC14, the camera processor 21A converts the coordinates of the image coordinate system ICS into coordinates of the radar coordinate system RCS. For each imaged object assigned a camera ID, the camera processor 21A sends the converted imaged object coordinates and imaged object attribute information to the server 30 (step StC15). Next, the surveillance system 100 performs the operation of step StS11 (see FIG. 9 ). The coordinates in the image coordinate system ICS indicating the position of the imaged object (or the coordinates converted from the image coordinate system ICS to the camera coordinate system CCS) are an example of image capture position information in this embodiment. Also, the converted imaged object coordinates are an example of image capture position information in this embodiment.

[0088] Next, a series of processes performed by the surveillance radar 10 in FIG. 9 will be described. In FIG. 9, the server processor 31 determines whether the detected object and the captured object are the same object based on the transformed detected object coordinates and the transformed captured object coordinates (step StS11). Specifically, the server processor 31 determines whether the transformed detected object coordinates of the detected object corresponding to the radar ID sent from the surveillance radar 10 are the same as the transformed captured object coordinates of the captured object corresponding to the camera ID sent from the fixed camera 20A. Note that if the processing of step StR14 (see FIG. 8) is omitted, the server processor 31 executes the processing of step StS11 based on the coordinates of the radar coordinate system RCS indicating the position of the detected object corresponding to the radar ID sent from the surveillance radar 10 and the transformed captured object coordinates of the captured object corresponding to the camera ID sent from the fixed camera 20A. Hereinafter, the description will be continued assuming that the processing of step StR14 is not omitted.

[0089] As shown in table TBL0, at the time of step StS11, radar IDs have been assigned to objects 1, 2, 4, and 5, and camera IDs have been assigned to objects 1, 3, and 5. In the example shown in FIG. 10 , the server processor 31 performs the processing of step StS11 based on the transformed detected object coordinates (or coordinates in the radar coordinate system RCS) of each of objects 1, 2, 4, and 5, and the transformed captured object coordinates of each of objects 1, 3, and 5.

[0090] There is a high possibility that the transformed detected object coordinates (or coordinates in the radar coordinate system RCS) and the transformed captured object coordinates for the same object are the same. That is, the server processor 31 compares the transformed detected object coordinates (or coordinates in the radar coordinate system RCS) corresponding to radar ID "1001" with the transformed captured object coordinates corresponding to each of camera IDs "2001," "2002," and "2003." As shown in FIG. 10 , the radar ID "1001" and the camera ID "2001" are values ​​assigned to object 1. In this case, the transformed detected object coordinates (or coordinates in the radar coordinate system RCS) corresponding to radar ID "1001" and the transformed captured object coordinates corresponding to camera ID "2001" match. That is, in this case, the server processor 31 determines that the detected object corresponding to radar ID "1001" and the captured object corresponding to camera ID "2001" are the same object. Furthermore, the server processor 31 determines that the detected object corresponding to the radar ID "1004" and the imaged object corresponding to the camera ID "2003" are the same object.

[0091] If the server processor 31 determines that the detected object and the captured image object are the same object (step StS11, YES), it assigns a server ID (an example of a third identifier) ​​that identifies the detected object determined to be the same object to the radar ID of the detected object determined to be the same object and the camera ID of the captured image object determined to be the same object (see objects 1 and 5 in table TBL1, FIG. 10) (step StS12). The server processor 31 then sends the captured image object attribute information corresponding to the camera ID associated with the detected object to which the server ID has been assigned and the radar ID associated with the detected object to which the server ID has been assigned to the surveillance radar 10 (step StS12). That is, in step StS12, the captured image object attribute information is sent from the server 30 to the surveillance radar 10.

[0092] In step StS12, the server processor 31 assigns the server ID "3001" to the radar ID "1001" and the camera ID "2001", and the server ID "3004" to the radar ID "1004" and the camera ID "2003" (see Figure 10).

[0093] The radar processor 11 associates the detected object attribute information acquired by the surveillance radar 10 with the imaged object attribute information associated with the radar ID transmitted from the server 30 in step StS12 (step StR16). Note that the imaged object attribute information of the imaged object associated with the radar ID corresponds to the imaged object attribute information acquired by the fixed camera 20A with respect to the imaged object determined in step StS11 to be the same object as the detected object associated with the radar ID.

[0094] That is, in step StR16, the radar processor 11 associates the radar ID "1001" with the imaged object attribute information related to the camera ID "2001" corresponding to the server ID "3001." Also, in step StR16, the radar processor 11 associates the radar ID "1004" with the imaged object attribute information related to the camera ID "2003" corresponding to the server ID "3004."

[0095] In this way, the surveillance radar 10 can acquire the detected object attribute information acquired by the surveillance radar 10 regarding the detected object to which the server ID is assigned, and the imaged object attribute information acquired by the fixed camera 20A. In the surveillance system 100, after step StR16, the processes of Figures 8 and 9 are repeated.

[0096] 8 and 9 , the monitoring system 100 can notify information associating a camera image and imaged object attribute information with a radar ID that identifies a detected object and detected object attribute information of the object corresponding to the radar ID. For example, the server processor 31 and the radar processor 11 display, on a user terminal such as a monitor 70, a superimposed image in which information such as a frame Fp1, a frame Fv1, and a frame Fv2 is superimposed on a radar map RMP1 and a camera image IMG1 as information associating a camera image and imaged object attribute information with a radar ID that identifies a detected object. The superimposed image in which information such as a frame Fp1, a frame Fv1, and a frame Fv2 is superimposed on the radar map RMP1 and the camera image IMG1 is an example of notification information in this embodiment.

[0097] 9 , if the server processor 31 determines that the detected object and the captured image object are the same object, it generates notification information by associating the camera image and the captured image object attribute information with the radar ID that identifies the detected object and the detected object attribute information of the object corresponding to the radar ID. Also, in the series of steps in Fig. 9 , if the radar processor 11 determines that the detected object and the captured image object are the same object, it generates notification information by associating the camera image and the captured image object attribute information with the radar ID that identifies the detected object and the detected object attribute information of the object corresponding to the radar ID.

[0098] On the other hand, if it is determined in step StS11 that the detected object and the captured object are not the same object (step StS11, NO), the server processor 31 performs processing in step StS13. In step StS13, the server processor 31 determines, based on the transformed detected object coordinates and the transformed captured object image, whether or not an object that has not been detected by the surveillance radar 10 has been captured by the fixed camera 20A (step StS13).

[0099] As shown in table TBL0, at the time of step StS11, radar IDs are assigned to objects 1, 2, 4, and 5, and camera IDs are assigned to objects 1, 3, and 5. Note that since objects 1 and 5 are determined as YES in step StS11, the server processor 31 performs the processing of step StS13 based on the transformed captured object coordinates of object 3 in the example shown in FIG.

[0100] That is, the server processor 31 determines whether there are any transformed detected object coordinates (or coordinates in the radar coordinate system RCS) that are identical to the transformed captured object coordinates corresponding to the camera ID "2002." As shown in FIG. 10 , the camera ID "2002" is a value assigned to the object 3. In this case, since the object 3 has not been detected by the surveillance radar 10, there are no transformed detected object coordinates (or coordinates in the radar coordinate system RCS) that are identical to the transformed captured object coordinates corresponding to the camera ID "2002." That is, in this case, in step StS13, the server processor 31 determines that the captured object corresponding to the camera ID "2002" is an object that has not been detected by the surveillance radar 10.

[0101] If the server processor 31 determines that the fixed camera 20A has captured an image of an object that the surveillance radar 10 has not detected (YES in step StS13), it assigns a server ID and a radar ID for the surveillance radar 10 to the imaged object of the fixed camera 20A (object 3 in table TBL0, see FIG. 10) (step StS14).The server processor 31 sends the imaged object attribute information for which the result of the determination in step StS13 is YES, the server ID, and the radar ID to the surveillance radar 10 (step StS14).

[0102] In step StS14, the server processor 31 assigns the server ID "3005" and the radar ID "3005" to the camera ID "2002" (see FIG. 10).

[0103] The radar processor 11 uses the information sent from the server 30 in step StS14 to generate information on a new detected object (specifically, an object captured by the fixed camera 20A but not detected by the surveillance radar 10) (step StR17). In the following description, an object captured by the camera 20 but not detected by the surveillance radar 10 may be referred to as a "detection target object." For example, the radar processor 11 assigns a radar ID to the detection target object and associates attribute information of the detection target object with the radar ID. The attribute information of the detection target object corresponds to the captured object attribute information acquired by the fixed camera 20A regarding the captured object corresponding to the detection target object. In this way, the surveillance radar 10 can associate the captured object attribute information of an object captured by the fixed camera 20A but not detected by the surveillance radar 10 with a radar ID that enables tracking by the surveillance radar 10. In the surveillance system 100, after step StR17, the processes of FIGS. 8 and 9 are repeated.

[0104] If the server processor 31 determines in step StS13 that the object detected by the surveillance radar 10 has not been captured by the fixed camera 20A (step StS13, NO), it assigns a server ID to the object detected by the surveillance radar 10 (objects 2 and 4 in table TBL0, see FIG. 10) (step StS15). After step StS15, the surveillance system 100 repeats the processes in FIGS. 8 and 9.

[0105] As shown in table TBL0, at the time of step StS11, radar IDs are assigned to objects 1, 2, 4, and 5, and camera IDs are assigned to objects 1, 3, and 5. Note that objects 1 and 5 are determined as YES in step StS11, and object 3 is determined as YES in step StS13. Therefore, in the example shown in FIG. 10 , the server processor 31 performs the processing of step StS15 based on the transformed detected object coordinates of objects 2 and 4.

[0106] That is, the server processor 31 performs the processing of step StS15 using the transformed detected object coordinates (or coordinates in the radar coordinate system RCS) corresponding to the radar ID "1002" and the transformed detected object coordinates (or coordinates in the radar coordinate system RCS) corresponding to the radar ID "1004." As shown in FIG. 10 , the radar ID "1002" is a value assigned to the object 2. In this case, since the object 2 has not been imaged by the camera 20, there are no transformed imaged object coordinates that are identical to the transformed detected object coordinates (or coordinates in the radar coordinate system RCS) corresponding to the radar ID "1002." The same applies to the radar ID "1004." That is, in this case, in step StS13, the server processor 31 determines that the detected object corresponding to the radar ID "1002" and the detected object corresponding to the radar ID "1004" are objects that have not been imaged by the camera 20. Then, in step StS15, the server processor 31 assigns the server ID "3002" to the radar ID "1002" and the server ID "3003" to the radar ID "1004" (see FIG. 10).

[0107] 8 and 9 , the server processor 31 (or radar processor 11) displays a radar map RMP1 and a camera image IMG1 on a user terminal such as a monitor 70 based on at least one of the radar position information, the detected object attribute information, the image capture position information, and the image capture object attribute information. In other words, if the detected object and the image capture object are not the same object, the server processor 31 generates notification information based on at least one of the radar position information, the detected object attribute information, the image capture position information, and the image capture object attribute information. In other words, if the detected object and the image capture object are not the same object, the radar processor 11 generates notification information based on at least one of the radar position information, the detected object attribute information, the image capture position information, and the image capture object attribute information.

[0108] The object 6 is not detected by the surveillance radar 10, and is also not detected by the camera 20. Therefore, the server 30 is not aware of the existence of the object 6 and is therefore unable to assign a server ID (see FIG. 10).

[0109] Next, an operational procedure for associating coordinates in a radar coordinate system indicating the position of a detected object with coordinates in a camera coordinate system indicating the position of an imaged object in the surveillance system 100 will be described with reference to FIG. 11 . FIG. 11 is a flowchart showing an example of an operational procedure for aligning the coordinates of the surveillance radar 10 with the coordinates of the PTZ camera 20B. The initial setting operational procedure performed before starting the process of FIG. 11 is the same as that of FIG. 7 using the fixed camera 20A. Therefore, when starting the process of FIG. 11 , the camera processor 21B performs processing similar to that performed by the camera processor 21A in FIG. 7 . Therefore, in the description of FIG. 11 , the reference numerals for the camera processor 21A in the description of FIG. 7 will be replaced with the reference numerals for the camera processor 21B, and further description will be omitted. In the description of FIG. 11 , the PTZ camera 20B will be used as the camera 20. Furthermore, in the description of FIG. 11 , the same processes as those described in FIG. 8 will be assigned the same step numbers, and their descriptions will be simplified or omitted, and differences will be described.

[0110] The PTZ camera 20B pans and / or tilts the camera lens to perform adjustments such as zooming to increase or decrease the imaging magnification. Therefore, in the explanation of Fig. 11, the series of processes performed by the PTZ camera 20B (steps StC21 to StC26) are performed subsequent to the series of processes performed by the surveillance radar 10 (steps StR11 to StR14).

[0111] In the process of step StR14 in FIG. 11, the surveillance radar 10 sends the converted detected object coordinates and detected object attribute information to the PTZ camera 20B for each detected object to which a radar ID has been assigned (step StR14).

[0112] The camera processor 21B calculates the positions of the rotation motor 261 and the zoom motor 262 (step StC21). In step StC21, the camera processor 21B uses the calculation formula generated in step St3 (see FIG. 7 ) to convert the coordinates (which may be converted detected object coordinates) in the radar coordinate system RCS indicating the position of the detected object sent from the surveillance radar 10 into coordinates in the camera coordinate system CCS. Specifically, the camera processor 21B calculates the position of the rotation motor 261 for directing the camera imaging direction toward the coordinate point in the camera coordinate system CCS indicating the position of the detected object, and the position of the zoom motor 262 for performing zoom processing toward the coordinate point in the camera coordinate system CCS indicating the position of the detected object. The camera processor 21B controls the driving of the camera driving unit 26B to drive to the position calculated in step StC21, and causes the imaging unit 23B to capture the surroundings including the detected object (step StC22). In step StC22, the camera driver 26B may be driven to capture an image of the imageable range of the PTZ camera 20B. The image captured by the imaging unit 23B in step StC22 (camera image CAP1) is input from the imaging unit 23B to the camera processor 21B.

[0113] The camera processor 21B causes the AI ​​processing unit 25B to perform image analysis using the camera image CAP1 as input in step StC22. Then, based on the analysis results of the camera image CAP1 by the AI ​​processing unit 25B, the camera processor 21B acquires the position (coordinates in the image coordinate system ICS) of the captured object in the camera image CAP1 (step StC23). The object recognized as an object included in the camera image CAP1 through the image analysis by the AI ​​processing unit 25B corresponds to the captured object. The camera processor 21B assigns a camera ID to the object included in the camera image CAP1 to identify the captured object (step StC23). The PTZ camera 20B captures an image of the captured object to which the camera ID has been assigned, thereby tracking the movement path or presence of the object detected by the surveillance radar 10.

[0114] Next, the camera processor 21B causes the AI ​​processing unit 25B to perform attribute classification to extract image-captured object attribute information. The camera processor 21B associates the image-captured object attribute information obtained as a result of the attribute classification process by the AI ​​processing unit 25B with the image-captured object position and the camera ID (step StC24).

[0115] The camera processor 21B converts the coordinates in the image coordinate system ICS, which indicate the position of the imaged object assigned the camera ID, into coordinates in the radar coordinate system RCS (step StC25) using the calculation formula generated in step St3 (see FIG. 7). For each imaged object assigned the camera ID, the camera processor 21B sends the imaged object coordinates (coordinates in the radar coordinate system RCS) after conversion in step StC25 and imaged object attribute information to the surveillance radar 10 (step StC26).

[0116] Based on the signal processing in step StR13, the radar processor 11 determines whether the surveillance radar 10 has detected the imaged object corresponding to the camera ID sent from the PTZ camera 20B in step StC26 (step StR21).

[0117] If the radar processor 11 determines that the monitoring radar 10 has detected an object captured by the PTZ camera 20B (step StR21, YES), it performs the processing of step StR22. That is, for a detected object assigned a radar ID, the radar processor 11 associates the attribute information acquired by the monitoring radar 10 with the captured object attribute information corresponding to the camera ID sent from the PTZ camera 20B (step StR22). A detected object assigned a radar ID corresponds to an object present in the first monitoring area AR1 that has been detected by the monitoring radar 10. In this way, the monitoring radar 10 can acquire detected object attribute information and captured object attribute information for the same object detected by the monitoring radar 10 and captured by the PTZ camera 20B. After step StR22, the monitoring system 100 repeatedly performs the processing of FIG. 11 .

[0118] On the other hand, if the radar processor 11 determines in step StR21 that the surveillance radar 10 has not detected the object captured by the PTZ camera 20B (NO in step StR21), it performs the processing of step StR23. That is, the radar processor 11 uses the information sent from the PTZ camera 20B in step StC26 to generate information on the detection target object (specifically, the object captured by the PTZ camera 20B but not detected by the surveillance radar 10) (step StR23). For example, the radar processor 11 assigns a new radar ID to the detection target object and associates the radar ID with the attribute information of the detection target object (i.e., the captured object attribute information acquired by the PTZ camera 20B for the captured object corresponding to the detection target object). In this way, the surveillance radar 10 can track the object by associating the radar ID with the captured object attribute information of the object not detected by the surveillance radar 10 but captured by the PTZ camera 20B. After step StR23, the monitoring system 100 repeats the process of FIG.

[0119] Next, a processing example using the result of associating the coordinates of the radar coordinate system of a detected object with the coordinate transformation-processed coordinates of the captured object in the monitoring system 100 will be described. Figures 12 and 13 are flowcharts showing an example of the initial setting procedure for displaying a list of detected objects and captured objects. Figure 14 is a diagram showing an example of a superimposed screen in which a first monitoring area AR1 and a camera image CAP1 are superimposed on a map MP1. Figure 15 is a diagram showing an example of a superimposed screen in which a camera image CAP1 is superimposed on a map MP2.

[0120] 7, the same step numbers are assigned in FIG. 12, the explanation is simplified or omitted, and differences are explained. The same step numbers are assigned in FIG. 13, the same process as in FIG. 8, the explanation is simplified or omitted, and differences are explained. In the explanations of FIGS. 12 to 15, the camera 20 may be, for example, a fixed camera 20A or a PTZ camera 20B. In the following explanation, the camera 20 is assumed to be a fixed camera 20A, and a case where multiple cameras 20 are installed will also be described.

[0121] Referring to FIG. 12 , the initial setup procedure for displaying a list of detected objects and captured objects will be described. A user operates the operation device 60 to input two-dimensional map data of the location where the monitoring system 100 is installed to the server 30 (step St11). The two-dimensional map of the location where the monitoring system 100 is installed may be, for example, an aerial map if the location is outdoors, or a floor map of a shopping mall or the like if the location is indoors, but is not limited to these. FIG. 14 shows aerial map data including map MP1 as the two-dimensional map data of the location where the monitoring system 100 is installed. In FIG. 12 , the processing from step St11 onward is the same as the processing from step St1 onward in FIG. 7 , and therefore description thereof will be omitted.

[0122] 13, a series of processes (steps StR11 to StR15, see FIG. 8) are performed by the surveillance radar 10, and a series of processes (steps StC11 to StC15, see FIG. 8) are performed by the camera 20. Next, the server processor 31 superimposes a marker indicating the detected object on the two-dimensional map input in step St11 (see FIG. 12) (step StS21). In step StS21, the server processor 31 may superimpose a superimposed image IMP1 (see FIG. 14) on the map MP1. After step StS21, the server processor 31 proceeds to step StS11.

[0123] If the server processor 31 determines that the detected object and the captured object are the same object (step StS11, YES), it determines whether or not there is an imaged object with the same camera ID in the camera images of the multiple cameras 20 (step StS22).If there is no imaged object with the same camera ID in the camera images of the multiple cameras 20 (step StS22, NO), it proceeds to step StS25.

[0124] On the other hand, if the server processor 31 determines that a captured object with the same camera ID exists in the camera images captured by the multiple cameras 20 (step StS22, YES), it selects the camera image CAP1 with the highest attribute classification score from the multiple captured camera images (step StS23). After step StS23, the server processor 31 proceeds to step StS25. A camera image with a high attribute classification score corresponds to, for example, an image in which the captured object, such as a person, is facing forward (i.e., toward the camera 20) and the face of the person is so clear that the area of ​​the person's face exceeds a predetermined number of pixels. In step StS23, the server processor 31 may select the camera image with the highest attribute classification score as the camera image CAP1. It goes without saying that specific examples of camera images with high attribute classification scores are not limited to the images described above. By doing so, when multiple cameras 20 are capturing the same object, the appearance of the captured object can be more accurately understood.

[0125] If the server processor 31 determines that the detected object and the captured object are not the same object (step StS11, NO), it performs the processing of step StS13. That is, the server processor 31 determines whether an object that has not been detected by the surveillance radar 10 has been captured by the camera 20 based on the transformed detected object coordinates and the transformed captured object coordinates (step StS13). If it determines that an object that has not been detected by the surveillance radar 10 has not been captured by the camera 20 either (step StS13, NO), the server processor 31 ends the processing of FIG.

[0126] On the other hand, if the server processor 31 determines that an object that has not been detected by the surveillance radar 10 has been captured by the camera 20 (step StS13, YES), it performs the process of step StS24. That is, the server processor 31 superimposes a marker (e.g., a circular image) indicating the captured object (e.g., person Ps1) on the position of the captured object on the two-dimensional map input in step St11 of Fig. 12 (step StS24). After step StS24, the server processor 31 proceeds to step StS25.

[0127] The server processor 31 generates a superimposed image IMP1 showing a camera image CAP1 of the captured object (e.g., person Ps1) and attribute information related to at least one of the detected object and the captured object (step StS25). The server 30 generates a superimposed screen WD1 (see FIG. 14) including the superimposed image IMP1, and outputs it to the monitor 70 or the like for display.

[0128] 14 is a diagram showing a superimposed screen WD1 as an example of notification information. The superimposed screen WD1 has a map MP1 and a superimposed image IMP1 superimposed on the map MP1. The map MP1 corresponds to two-dimensional data such as a two-dimensional aerial map or a two-dimensional survey map. The superimposed image IMP1 corresponds to, for example, an image showing the first monitoring area AR1 (the dotted area in FIG. 14 ), an image showing the second monitoring area AR2 (the shaded area in FIG. 14 ), an image showing the radar installation position PS0, a camera image CAP1, etc.

[0129] FIG. 14 illustrates an example of the superimposed screen WD1 when a person Ps1 is detected by the surveillance radar 10 and captured by the camera 20. The size relationship between the first and second monitoring areas AR1 and AR2 may be reversed, and the shapes of the first and second monitoring areas AR1 and AR2 are not limited to those shown in FIG. 14 . The superimposed image IMP1 may also display attribute information (detected object attribute information or captured object attribute information) of the object (e.g., person Ps1) along with a camera image CAP1 of the object. This allows a user of the terminal (e.g., the security guard terminal 50, the monitor 70, etc.) to receive the notification of the superimposed screen WD1 regarding the person's characteristic features and their location in the first or second monitoring area AR1 or AR2.

[0130] FIG. 15 illustrates a superimposed screen WD2 as an example of notification information. The superimposed screen WD2 includes a map MP2 and a superimposed image IMP2. The map MP2 corresponds to a three-dimensional map such as a three-dimensional aerial map or a perspective projection. The superimposed image IMP2 is a camera image CAP1 of a detected object, for example. The superimposed image IMP2 may also display attribute information (detected object attribute information or captured object attribute information) of the object (e.g., person Ps1) along with the camera image CAP1 of the detected object. This allows a user viewing the notification destination (e.g., the security guard terminal 50 or the monitor 70) on which the superimposed screen WD2 is displayed to be notified with high visibility about the location of a person with characteristic features in the first monitoring area AR1 or the second monitoring area AR2.

[0131] Next, with reference to FIGS. 16 , 17 , and 18 , another processing example using the association result between the coordinates of a detected object in the radar coordinate system RCS and the coordinates of an imaged object in the radar coordinate system RCS after coordinate transformation in the monitoring system 100 will be described. FIG. 16 is a flowchart showing an example of an operational procedure for performing intrusion detection with respect to a detected object and an imaged object. FIG. 17 is a diagram showing an example of an alarm rule. FIG. 18 is a flowchart showing an operational procedure for performing intrusion detection with respect to a detected object and an imaged object. In the description of FIG. 16 , the same steps as those in FIG. 7 or 12 are assigned the same step numbers, and their description will be simplified or omitted. In the description of FIG. 18 , the same steps as those in FIG. 8 or 9 are assigned the same step numbers, and their description will be simplified or omitted. Although FIGS. 16 to 18 describe the case where a fixed camera 20A is used, a PTZ camera 20B may also be used as the camera 20.

[0132] 16 , after step St3, the server processor 31 receives input of information indicating the entry-restricted area AR10 (step St21). As shown in FIG. 14 , the entry-restricted area AR10 is an area into which an object is prohibited and can be arbitrarily designated by the user of the monitoring system 100. The entry-restricted area AR10 is also an area into which an alarm is issued (notification is sent to the user, people around the monitoring system 100, etc.) when an object is detected entering the area. The server 30 also receives input of information indicating the alarm rule TBL2 (see FIG. 17 ) specified by the user through operation of the operation device 60 (step St21). The server 30 sends the information indicating the entry-restricted area and the information in the alarm rule TBL2 to the monitoring radar 10 (step St21). The monitoring radar 10 stores and sets the information indicating the entry-restricted area and the information in the alarm rule TBL2 sent from the server 30 in the memory 12 (step St21).

[0133] The user of the monitoring system 100 can operate the operating device 60 to specify the entry-restricted area AR10. For example, the entry-restricted area AR10 may be specified as a shape other than a rectangle. For example, as an example of the entry-restricted area AR10, an entry-restricted line BNL1 for detecting the presence or absence of an object at a position a predetermined distance away from a reference position (in FIG. 14, the installation position of the fixed camera 20A) may be specified (see FIG. 14). Furthermore, the user of the monitoring system 100 can specify any area included in the first monitoring area AR1 and the second monitoring area AR2 as the entry-restricted area AR10.

[0134] 17 , the alarm rule TBL2 includes an item L11 indicating the location and type (line or area) of the no-entry area AR10, an item L12 indicating the object movement conditions, an item L13 indicating the object type, and an item L14 indicating the object attributes. Item L13 and item L14 may include information that can be identified only by the surveillance radar 10 or information that can be identified only by the camera 20.

[0135] No. 1 in the alarm rule TBL2 shows an example in which an alarm is issued when an object crosses the no-entry line BNL1 (line crossing), the object is a person, and the object's attribute is red. An example of an object's attribute being red would be, but is not limited to, a case in which the object is a person wearing red clothing. The alarm rule TBL2 for no-entry areas can be edited by the user using the operation device 60, and the edited alarm rule may be stored in the memory 32.

[0136] 18, a series of processes is performed by the surveillance radar 10 (steps StR11 to StR15, see FIG. 8), and a series of processes is performed by the camera 20 (steps StC11 to StC15, see FIG. 8). Subsequently, a series of processes is performed by the server 30 (steps StS11 to StS15, see FIG. 9), and a series of processes is performed by the surveillance radar 10 (steps StR16 to StR17, see FIG. 9). Note that the series of processes shown in FIG. 18 may be performed mainly by the server processor 31 in the server 30.

[0137] The radar processor 11 performs a tracking process to track the object using the result of the detection process by the detection unit 13 (step StR31). The radar processor 11 determines whether the attribute information related to the object to be tracked (at least one of the detected object attribute information and the captured object attribute information) corresponds to the alarm rule TBL2 (step StR32). If it is determined that the attribute information related to the object to be tracked (i.e., at least one of the detected object attribute information and the captured object attribute information related to the object) does not correspond to the alarm rule TBL2 (step StR32, NO), the processing of the surveillance system 100 shown in FIG. 18 ends.

[0138] On the other hand, if the radar processor 11 determines that the attribute information related to the tracked object (at least one of the detected object attribute information and the captured object attribute information) corresponds to the alarm activation rule TBL2 (step StR32, YES), it performs the processing of step StR33. That is, the radar processor 11 generates notification information for alarm activation and sends it to the server 30 (step StR33). The server processor 31 notifies a predetermined notification destination, such as a user terminal, of the alarm activation. Here, the notification information for alarm activation includes at least information such as attribute information of the tracked object (at least one of the detected object attribute information and the captured object attribute information), the position of the tracked object (e.g., the position in the radar coordinate system RCS), and the current time. The server processor 31 may also generate control information for moving the security robot 40, information for intimidating or warning the object, and notify the security robot 40 of the generated information as an alarm activation.

[0139] As described above, the monitoring system 100 of the present disclosure not only notifies the user terminal or the like of the presence or absence of an object, but also of notification information including various information about the object acquired by the monitoring radar 10 and / or the camera 20. Therefore, the monitoring system 100 can improve the detection accuracy of objects present in the monitoring area.

[0140] In addition, the monitoring system 100 is capable of collecting various information regarding objects detected by the monitoring radar 10, thereby improving the detection accuracy of objects present within the monitoring area (e.g., the first monitoring area AR1, the second monitoring area AR2).

[0141] In addition, the monitoring system 100 can store alarm rules for alerting the presence of an object that has entered a prohibited area, and can accurately alert the intrusion of the relevant object by performing intrusion detection processing in accordance with the alarm rules.

[0142] Furthermore, the monitoring system 100 can accurately detect only objects that have engaged in specified abnormal behavior in an arbitrarily specified area.

[0143] Furthermore, the monitoring system 100 can accurately detect only objects that meet specified conditions as attribute information of the objects.

[0144] Furthermore, the monitoring system 100 displays the position of an object superimposed on the map MP1, thereby enabling a user viewing the map MP1 to easily grasp the position of the object.

[0145] Furthermore, the monitoring system 100 can display not only the location of an object on the map MP1, but also the camera image at the time the object was captured and the object's attribute information, thereby allowing the user to grasp detailed information about the object.

[0146] Furthermore, the monitoring system 100 allows the user to easily visually grasp the location of a person and the characteristics of that person.

[0147] Furthermore, by comprehensively using the surveillance radar 10 and multiple cameras 20, the surveillance system 100 can not only further improve the accuracy of object detection, but also extract more detailed information, such as attribute information of the object.

[0148] Furthermore, the monitoring system 100 can accurately present the user with highly reliable information about an object by displaying a camera image that is more suitable for extracting attribute information about the object and the attribute information obtained from the camera image.

[0149] Furthermore, the monitoring system 100 can output highly reliable information about an object using either a score indicating the accuracy of the attribute information contained in the attribute information, the size of the object, or both the score and the size of the object.

[0150] Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications, alterations, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure. Furthermore, the components of the various embodiments described above may be combined in any manner without departing from the spirit of the invention.

[0151] This application is based on a Japanese patent application (Patent Application No. 2022-052014) filed on March 28, 2022, the contents of which are incorporated by reference into this application.

[0152] The present disclosure is useful as a monitoring system and a monitoring method that improve the accuracy of detecting an object present in a monitoring area.

[0153] DESCRIPTION OF SYMBOLS 10 Surveillance radar 11 Radar processor 12, 22A, 22B, 32 Memory 13 Detection unit 14, 24A, 24B, 34 Communication unit 15, 25A, 25B AI processing unit 20 Camera 20A Fixed camera 20B PTZ camera 21A, 21B Camera processor 23A, 23B Imaging unit 26B Camera driving unit 30 Server 31 Server processor 33 Database 40 Security robot 50 Security guard terminal 60 Operation device 70 Monitor 100 Surveillance system 131, 13n Radar IC 151, 251A, 251B AI calculation processing unit 152, 252A, 252B Learning model memory 261 Rotation motor 262 Zoom motor An Antenna unit ARx1, ARxn Receiving antenna unit ATx1, ATxn Transmitting antenna section NW Network

Claims

1. a radar having an antenna unit that transmits radio waves to a first monitoring area and receives reflected waves of the radio waves; a detection unit that performs a detection process to detect the presence or absence of an object in the first monitoring area based on the reflected waves; and a control unit that generates detected object attribute information that indicates attributes of the detected object detected by the detection unit based on a result of the detection process, and generates radar position information that indicates the position of the detected object based on first installation information that includes an antenna installation position and an antenna orientation and information on an antenna viewing angle; a processing unit that acquires imaging position information indicating a position of an imaged object included in an image of the second monitoring area based on second installation information including an installation position of the imaging unit and an imaging direction of the imaging unit and information on a viewing angle of the imaging unit, and acquires imaged object attribute information indicating attributes of the imaged object based on the image, The monitoring system includes: a determination unit that performs a determination process to determine whether the detected object and the captured image object are the same object based on the radar position information and the captured image position information; a notification control unit that causes a notification unit that performs notification processing for a user to notify the user of notification information; The notification information includes: When the detected object and the captured image object are the same object, the information includes at least a first identifier for identifying the detected object, the detected object attribute information, the captured image, and information associating the captured image object attribute information, When the detected object and the captured image object are not the same object, the information is based on at least one of the radar position information, the detected object attribute information, the captured image position information, and the captured image object attribute information. Surveillance system.

2. The camera is a motor for adjusting the imaging direction; a motor control unit that drives the motor, the control unit drives the motor so that the imaging unit captures an image of an object included in the second monitoring area among the detected objects, based on a relationship between a position of the motor, an imaging range of the imaging unit in real space, and the second installation information; When the detected object and the captured object are the same object, the notification information is information in which a monitoring position captured image of the object included in the second monitoring area captured by the imaging unit and captured object attribute information of the object included in the monitoring position captured image are associated with the first identifier corresponding to the detected object. The monitoring system of claim 1 .

3. The monitoring system includes: a tracking processing unit that performs a tracking process to track an object based on a result of the detection process, The determination unit executes an abnormality determination process for determining whether the detected object or the imaged object is an object to be warned based on an abnormality determination condition including at least one of the imaged object attribute information and a movement state of the detected object or the imaged object; The notification control unit issuing an alarm as the notification process when it is determined that the image-captured object attribute information corresponding to the object that is the target of the tracking process, or the detected object that is the target of the tracking process, or the moving state of the image-captured object that is the target of the tracking process, satisfies the abnormality determination condition; 3. A monitoring system according to claim 1 or 2.

4. The abnormality determination condition is At least one of the following is detected: the detected object or the imaged object enters a forbidden area where objects are forbidden to enter; the detected object or the imaged object stays in the forbidden area; a specific behavior pattern related to the detected object or the imaged object; the detected object or the imaged object leaving an object behind; or the detected object or the imaged object taking away an object. The monitoring system of claim 3 .

5. The alert is a threat to an object, a warning, or information for causing a drone or a robot to execute the threat or the warning. The monitoring system of claim 3 .

6. The image capture object attribute information is The information includes at least one of the type of the imaged object, its gender, its age group, its height, its clothing color, its car model, its car color, a score indicating the accuracy of the attribute, and its moving speed. The monitoring system of claim 1 .

7. an information processing device having the notification unit, The notification unit As the notification process, the position of the detected object and the position of the captured object are displayed superimposed on a map image including the first monitoring area and the second monitoring area. The monitoring system of claim 1 .

8. The information processing device includes: When the notification information includes the image capture object attribute information, the image capture image and the image capture object attribute information are displayed superimposed on the map image. The monitoring system of claim 7.

9. The map image is a two-dimensional image or a three-dimensional image of an area including the first monitoring area and the second monitoring area; 9. A monitoring system according to claim 7 or 8.

10. the surveillance system comprises a plurality of the cameras; The notification information includes: When there are a plurality of captured images of the image capturing object that are determined to be identical to the detected object in the determination process, the first identifier corresponding to the detected object, the plurality of captured images of the image capturing object that are determined to be identical to the detected object, and image capturing object attribute information of the image capturing object that are determined to be identical to the detected object are associated with each other. The monitoring system of claim 1 .

11. The notification information includes: The information includes a captured image that meets a predetermined criterion regarding the captured object that is determined to be the same as the detected object, and the captured object attribute information. The monitoring system of claim 10.

12. The predetermined criteria are: The attribute is determined based on at least one of a score indicating the accuracy of the attribute and a size of the object. The monitoring system of claim 11.

13. transmitting radio waves to a first monitoring area and receiving reflected waves of the radio waves; performing a detection process for detecting the presence or absence of an object in the first monitoring area based on the reflected wave; generating detected object attribute information indicating attributes of the detected object detected by the detection unit that performs the detection process based on a result of the detection process, and generating radar position information indicating the position of the detected object based on first installation information including an antenna installation position and an antenna orientation and information on an antenna viewing angle; capturing an image of a second monitoring area, at least a portion of which overlaps with the first monitoring area, by an imaging unit; acquiring image capture position information indicating a position of an image capture object included in an image captured in the second monitoring area based on second installation information including an installation position of the image capture unit and an image capture direction of the image capture unit and information on a viewing angle of the image capture unit, and acquiring image capture object attribute information indicating an attribute of the image capture object based on the image capture image; performing a determination process to determine whether the detected object and the captured image object are the same object based on the radar position information and the captured image position information; causing a notification unit that performs notification processing for the user to notify the notification information; The notification information includes: When the detected object and the captured image object are the same object, the information includes at least a first identifier for identifying the detected object, the detected object attribute information, the captured image, and information associating the captured image object attribute information, When the detected object and the captured image object are not the same object, the information is based on at least one of the radar position information, the detected object attribute information, the captured image position information, and the captured image object attribute information. Monitoring method.