surveillance system

Abstract

To obtain a clear color image obtained by photographing an intruder that has invaded a monitored area. [Solution] Surveillance system 1 comprises intrusion monitoring sensor 11 that uses millimeter waves to detect an intruder into a monitored area and identify its position, visible light night vision camera 12 that is capable of capturing color images even in weak ambient light, and data processing device 13. When intrusion monitoring sensor 11 detects an intruder, data processing device 13 generates a control signal to instruct visible light night vision camera 12 to focus on the position of the intruder identified by intrusion monitoring sensor 11, based on the position of the intruder, and outputs the signal to visible light night vision camera 12. Visible light night vision camera 12 focuses in accordance with the control signal received from data processing device 13.

Description

[Technical Field]

[0001] The present invention relates to a technique for monitoring intrusions in a particular area. [Background technology]

[0002] Traditionally, infrared beam sensors, motion sensors (PIR), or surveillance cameras have been used to monitor intrusions outdoors or at critical facilities.

[0003] For example, Patent Document 1 is a patent document relating to a system that uses an infrared beam sensor to monitor intrusions outdoors or at important facilities. Patent Document 1 describes a technology that increases detection accuracy while suppressing malfunctions by providing a sub-detection area separate from a main detection area and improving the detection sensitivity of the main detection area when an object is detected in the sub-detection area. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-16015 Summary of the Invention [Problem to be solved by the invention]

[0005] Intrusion monitoring systems that use PIR sensors or beam sensors can detect whether an intrusion has occurred, but they have the drawback of not being able to obtain detailed information to identify the intruder, and they often produce false alarms due to animals or environmental factors.

[0006] Intrusion monitoring systems that use surveillance cameras have the advantage of being able to obtain visual information. However, when a single surveillance camera is used to monitor a wide area, it is difficult to obtain a clear image of an intruder that has entered the monitored area.

[0007] In view of the above circumstances, an object of the present invention is to make it possible to acquire a clear color image of an intruder that has entered a monitored area. [Means for solving the problem]

[0008] In one aspect, the present invention provides a surveillance system comprising an intrusion monitoring sensor that uses millimeter waves or terahertz waves to detect an intruder that has entered a surveillance area and identify the position of the intruder in the surveillance area, and a visible light camera that focuses and captures a color image based on the position of the intruder identified by the intrusion monitoring sensor. [Effects of the Invention]

[0009] According to the present invention, a clear color image of an intruder that has entered a monitored area can be obtained. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing the configuration of a monitoring system according to an embodiment. [Figure 2] FIG. 1 is a flowchart showing the operation of a monitoring system according to an embodiment. [Figure 3] FIG. 10 is a diagram showing the configuration of a monitoring system according to a modified example. [Figure 4] FIG. 10 is a diagram showing the configuration of a monitoring system according to a modified example. [Figure 5] FIG. 10 is a diagram showing the configuration of a monitoring system according to a modified example. [Figure 6] FIG. 10 is a diagram showing the configuration of a monitoring system according to a modified example. [Figure 7] FIG. 10 is a diagram showing the configuration of a monitoring system according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0011] 1 is a diagram showing the configuration of a monitoring system 1 according to one embodiment of the present invention. The monitoring system 1 includes an intrusion monitoring sensor 11, a visible light night vision camera 12 (an example of a fixed visible light camera), and a data processing device 13. The data processing device 13 is connected to the intrusion monitoring sensor 11 and the visible light night vision camera 12.

[0012] The intrusion monitoring sensor 11 detects an intrusion into the monitoring area X and identifies the position of the intruder in the monitoring area X, more specifically, the position in the longitudinal direction of the elongated monitoring area X. In this embodiment, the intrusion monitoring sensor 11 uses millimeter waves to detect the intruder and identify the position of the intruder.

[0013] The intrusion monitoring sensor 11 includes a wave transmitting unit 111 and a wave receiving unit 112 arranged in close proximity, and a reflector 113 arranged in a position facing the wave transmitting unit 111 and the wave receiving unit 112 across the monitoring area X.

[0014] The wave receiving unit 112 receives the millimeter waves transmitted from the wave transmitting unit 111 and reflected by the reflector 113. The intrusion monitoring sensor 11 detects an intruder by detecting changes in the intensity, timing, phase, etc. of the millimeter waves received by the wave receiving unit 112 using a control unit (not shown).

[0015] Furthermore, when the intrusion monitoring sensor 11 detects an intrusion, the control unit determines the distance from the wave transmitting unit 111 and the wave receiving unit 112 to the intrusion in the longitudinal direction of the monitoring area X, for example, according to the principle of time-of-flight, based on the timing at which the millimeter waves reflected by the intrusion are received by the wave receiving unit 112. The distance determined in this way indicates the position of the intrusion.

[0016] The intrusion monitoring sensor 11 may use any of an FMCW (frequency modulated continuous wave) method, a pulse radar method, or the like to detect an intrusion.

[0017] Visible light night vision camera 12 is a visible light camera that functions as a surveillance camera that continuously captures images of surveillance area X. Visible light night vision camera 12 is equipped with a highly sensitive image sensor, a large-diameter lens, and a dedicated image processing engine that removes noise from weak signals and reconstructs images, and is therefore able to generate color images that clearly show the color and shape of objects even under weak ambient light that would be perceived as darkness by the naked eye.

[0018] As of the time of filing, the high-sensitivity image sensor included in the visible light night vision camera 12 may be a BSI CMOS (Back-Side Illuminated Complementary Metal-Oxide-Semiconductor) sensor, a stacked CMOS (Complementary Metal-Oxide-Semiconductor) sensor, a SPAD (Single-Photon Avalanche Diode) sensor, an EMCCD (Electron-Multiplying Charge-Coupled Device) sensor, an ICCD (Intensified Charge-Coupled Device) sensor, or an ICMOS (Intensified Complementary Metal-Oxide-Semiconductor) sensor. Of these, the SPAD sensor, EMCCD sensor, ICCD sensor, and ICMOS sensor are particularly called ultra-high-sensitivity image sensors and are desirable. If sensors with sensitivity equal to or greater than these sensors become available after the filing of this application, these sensors may be used as the high-sensitivity image sensor included in the visible light night vision camera 12.

[0019] The data processing device 13 is a device that controls the monitoring system 1. The hardware of the data processing device 13 is a computer that includes a memory that stores various data, a processor that processes various data in accordance with programs stored in the memory, an input / output interface that exchanges data with the intrusion monitoring sensor 11 and the visible light night vision camera 12, and a communication interface that exchanges various data with external devices such as terminal devices used by users of the monitoring system 1 and higher-level systems via a network.

[0020] The data processing device 13 that performs the operations described below is realized by the processor of a computer, which is the hardware of the data processing device 13, performing data processing in accordance with the program according to this embodiment.

[0021] After the monitoring system 1 is started up, the intrusion monitoring sensor 11 continuously detects an intrusion into the monitoring area X and outputs the detection result to the data processing device 13. The detection result output by the intrusion monitoring sensor 11 indicates the presence or absence of an intrusion and the position of the detected intrusion (if an intrusion is present).

[0022] After the monitoring system 1 is started, the visible light night vision camera 12 continuously captures images of the monitoring area X and outputs the captured images to the data processing device 13. While the intrusion monitoring sensor 11 is not detecting an intruder, the visible light night vision camera 12 captures images with its focus set at the center position in the longitudinal direction of the monitoring area X as the default focus position.

[0023] The data processing device 13 stores the detection results continuously input from the intrusion monitoring sensor 11 and the images input from the visible light night vision camera 12 .

[0024] As described above, in a situation where the detection results of the intrusion monitoring sensor 11 and the images captured by the visible light night vision camera 12 are sequentially stored in the data processing device 13, the monitoring system 1 performs operations according to the flow shown in Figure 2 each time a new detection result of the intrusion monitoring sensor 11 is stored.

[0025] The data processing device 13 determines whether the newly stored detection result of the intrusion monitoring sensor 11 indicates the presence of an intrusion (step S1). If the detection result of the intrusion monitoring sensor 11 indicates the absence of an intrusion (step S1; No), the monitoring system 1 does not perform the processes from step S2 onwards and ends the process according to the flow of FIG.

[0026] On the other hand, if the detection result of the intrusion monitoring sensor 11 indicates the presence of an intrusion (step S1; Yes), the data processing device 13 generates a control signal instructing the camera to focus on the position of the intrusion indicated by the newly stored detection result, and outputs the control signal to the visible light night vision camera 12 (step S2).

[0027] The visible light night vision camera 12 receives the control signal output from the data processing device 13 in step S2 and adjusts the focus in accordance with the control signal (step S3). As a result, a clear image of the intruding object is recorded in the data processing device 13.

[0028] If the detection result of the intrusion monitoring sensor 11 indicates the presence of an intruder (step S1; Yes), in parallel with the processing of step S2, the data processing device 13 performs image recognition processing on the image received from the visible light night vision camera 12 using artificial intelligence technology such as deep learning, and determines the type of intruder (for example, whether the intruder is a human, animal, automobile, etc.) (step S4).

[0029] When the data processing device 13 determines the type of intruding object in step S4, it determines whether or not a notification is necessary based on the determined type of intruding object (step S5). For example, if the type of intruding object is a predetermined type (e.g., a human or a moving object such as a vehicle), the data processing device 13 determines that a notification is necessary (step S5; Yes). In that case, the data processing device 13 transmits a notification regarding the detection of the intruding object to a predetermined external device (e.g., a terminal device used by the user or a higher-level system) (step S6). This notification includes a video including the image used to recognize the intruding object.

[0030] If the data processing device 13 determines that notification is not necessary (step S5; No), it does not perform the process of step S6.

[0031] The external device that receives the notification transmitted from the data processing device 13 in step S6 performs processing in accordance with the notification. For example, when a terminal device used by a user receives the notification transmitted from the data processing device 13, the terminal device notifies the user that an intruding object has been detected in the monitored area X by sounding an alarm or displaying an alarm message. When the user performs an operation on the terminal device in response to the notification, for example, to instruct the terminal device to play a video, the terminal device plays the video included in the notification received from the data processing device 13. The user can watch the video and confirm the intruding object.

[0032] [Variations] The above-described monitoring system 1 is one embodiment of the system according to the present invention, and may be modified in various ways within the scope of the technical concept of the present invention. Examples of such modifications are shown below. Two or more of the following modifications may be combined as appropriate.

[0033] (1) The visible light night vision camera 12 may have an optical zoom function, and the zoom magnification may be changed in accordance with a control signal output from the data processing device 13.

[0034] In this modification, in step S2, the data processing device 13 generates a control signal to instruct the camera to focus on the position of the intruder indicated by the detection result of the intrusion monitoring sensor 11 and to zoom at a zoom factor corresponding to the distance to that position, and outputs this control signal to the visible light night vision camera 12. In this case, the zoom factor instructed by the control signal is a factor that causes an intruder of average size (such as a person or a car) to appear large within the shooting range but not protrude beyond the shooting range. Furthermore, the zoom factor increases as the distance from the visible light night vision camera 12 to the intruder increases.

[0035] Upon receiving the control signal output from the data processing device 13, the visible light night vision camera 12 adjusts its focus and zooms at the specified zoom ratio in accordance with the control signal. As a result, a large, clear image of the intruding object within the shooting range is recorded in the data processing device 13. This makes it possible to perform face recognition and other functions.

[0036] (2) The surveillance system 1 may include a pan head that changes the shooting direction of the visible light night vision camera 12. FIG. 3 shows the configuration of a surveillance system 1 that includes such a pan head 14. The pan head 14 is a controlled pan head that can rotate (pan) the camera mount at least around a vertical axis from a reference position by a specified angle in accordance with a control signal output from the data processing device 13. The pan head 14 may be a two-axis controlled pan head that can rotate (tilt) the camera mount around a left-right axis from a reference position by a specified angle in addition to panning. Alternatively, the pan head 14 may be a three-axis controlled pan head that can rotate (roll) the camera mount around a front-to-back axis from a reference position by a specified angle in addition to panning and tilting.

[0037] The visible light night vision camera 12 is attached to a camera mount on a pan head 14 .

[0038] In this variant, when the data processing device 13 determines the type of intruding object in step S4, it instructs the pan head 14 to rotate the camera mount so that the intruding object is within the shooting range of the visible light night vision camera 12 based on the position at which the intruding object appears in the image used for the determination.

[0039] More specifically, the data processing device 13 identifies the position where the intruder is captured in each of the consecutive images in the time series, identifies the movement vector of the intruder, and, based on the identified movement vector and the positional relationship between the monitoring area X and the visible light night vision camera 12, identifies the attitude of the visible light night vision camera 12 that captures the intruder approximately in the center of the imaging range. The data processing device 13 identifies the rotation angle of the camera mount of the pan head 14 (the rotation angle of each of the two axes if the pan head 14 is a two-axis control pan head, or the rotation angle of each of the three axes if the pan head 14 is a three-axis control pan head) so that the attitude of the visible light night vision camera 12 becomes the identified attitude, and generates a control signal instructing the pan head 14 to rotate the camera mount to the identified rotation angle. The data processing device 13 outputs the generated control signal to the pan head 14, and the pan head 14 changes the attitude of the camera mount in accordance with the control signal received from the data processing device 13.

[0040] According to this modification, automatic tracking is performed so that the intruder is always captured approximately in the center of the camera, and the intruder can be tracked within the movable range of the camera platform 14 even after it leaves the monitoring area X. As a result, if the intruder is a human, for example, personal authentication based on gait (walking habits) becomes possible.

[0041] (3) The monitoring system 1 may include one or more autonomous mobile units (so-called drones) equipped with a visible light night vision camera that tracks and photographs an intruder detected by the intrusion monitoring sensor 11. FIG. 4 is a diagram showing the configuration of a monitoring system 1 that includes such an autonomous mobile unit 15. Note that while the autonomous mobile unit 15 shown in FIG. 4 is an airborne autonomous mobile unit, the autonomous mobile unit 15 is not limited to an airborne autonomous mobile unit (UAV) and may be a ground-running autonomous mobile unit (UGV), a waterborne autonomous mobile unit (USV), or the like, depending on the usage environment of the monitoring system 1.

[0042] In this modification, under normal circumstances, the autonomous moving body 15 waits at a base near the monitored area X. If the data processing device 13 determines in step S5 that the intrusion is of a type that requires notification, it wirelessly transmits a notification to the autonomous moving body 15. This notification includes the position of the intrusion in the monitored area X.

[0043] When the autonomous mobile body 15 receives the notification sent from the data processing device 13, it moves to a position within the shooting range of the visible light night vision camera 16 (an example of a mobile visible light camera) mounted on the autonomous mobile body 15, where the position of the intruder included in the notification is included, and recognizes the intruder from the image captured by the visible light night vision camera 16.If the intruder is recognized, it moves so that the intruder is approximately in the center of the shooting range.

[0044] According to this modification, even after the intruder moves out of the range that can be photographed by the visible light night vision camera 12, the visible light night vision camera 16 continues to photograph the intruder as the autonomous moving body 15 tracks the intruder. As a result, the intruder can be tracked over a wide area.

[0045] (4) The monitoring system 1 may be equipped with multiple intrusion monitoring sensors 11 to monitor a monitoring area that extends horizontally. In this case, the monitoring system 1 may be equipped with multiple visible light night vision cameras 12 so that the entire monitoring area that extends horizontally can be photographed.

[0046] 5 is a diagram (viewed from above) showing the configuration of a monitoring system 1 including multiple intrusion monitoring sensors 11, i.e., intrusion monitoring sensors 11(1) to 11(6), and multiple visible light night vision cameras 12, i.e., visible light night vision cameras 12(1) to 12(6). Note that in the example of FIG. 5, there is a one-to-one correspondence between the intrusion monitoring sensors 11 and the visible light night vision cameras 12; however, if the imaging range of one visible light night vision camera 12 is wider than the monitoring area of ​​one intrusion monitoring sensor 11, there may not be a one-to-one correspondence between the intrusion monitoring sensors 11 and the visible light night vision cameras 12, and the number of visible light night vision cameras 12 may be less than the number of intrusion monitoring sensors 11. Also, in the example of FIG. 5, there are six intrusion monitoring sensors 11, but the number is not limited to six.

[0047] As shown in Fig. 5, each of the intrusion monitoring sensors 11(1)-11(6) is arranged so that their monitoring areas, that is, monitoring areas X(1)-X(6), are equally spaced in the y-axis direction, where the longitudinal direction is the x-axis direction and the horizontal direction perpendicular to the x-axis direction is the y-axis. As a result, the monitoring system 1 according to this modification can monitor an area that extends on the xy plane (monitoring area XY shown in Fig. 5).

[0048] In this modification, the data processing device 13 aligns the images captured by each of the visible light night vision cameras 12(1)-12(6) and stitches them together to generate a single image. The data processing device 13 recognizes an intruder from the composite image. Furthermore, by continuously recognizing the intruder from the composite image, the data processing device 13 identifies the movement path and movement vector of the intruder in the monitoring area XY. The data processing device 13 predicts the future position of the intruder from the identified movement path and movement vector of the intruder, and outputs a control signal to the visible light night vision camera 12 corresponding to the monitoring area X including the predicted position, instructing it to focus on that position. As a result, when a new intruder intrudes into the monitoring area X corresponding to the device, the visible light night vision camera 12(1)-12(6) can capture an image with its focus already adjusted to the intrusion position.

[0049] Furthermore, when this modification is combined with modification (1), data processing device 13 may output a control signal to visible light night vision cameras 12(1)-12(6) corresponding to monitoring area X into which the intruder will newly enter, instructing them to zoom at a zoom magnification corresponding to the predicted future position of the intruder. In this case, when the intruder newly enters monitoring area X corresponding to the camera itself, visible light night vision cameras 12(1)-12(6) can take an image in a state where the camera has zoomed in at a zoom magnification corresponding to the intrusion position in advance.

[0050] (5) In the above-described embodiment, some or all of the processing performed by the data processing device 13 may be performed by another data processing device that can communicate with the data processing device 13 via a network. Fig. 6 is a diagram showing the configuration of a monitoring system 1 according to this modification.

[0051] In this modification, the data processing device 13 sequentially transmits the detection results output from the intrusion monitoring sensor 11 and the images output from the visible light night vision camera 12 to a server device 17 (an example of a data processing device different from the data processing device 13). The server device 17 uses the detection results and images received from the data processing device 13 to perform the data processing that the data processing device 13 performs in the above-described embodiment.

[0052] According to this modification, it is possible to centrally manage the detection results and images output from the intrusion monitoring sensors 11 and visible light night vision cameras 12 installed at multiple different locations. Furthermore, according to this modification, it is possible to keep the artificial intelligence models used for image recognition and the like up to date at all times.

[0053] (6) In the above-described embodiment, part of the processing that was supposed to be performed by the data processing device 13 may be performed by either the intrusion monitoring sensor 11 or the visible light night vision camera 12. For example, the image recognition of an intruding object that was supposed to be performed by the data processing device 13 may be performed by the control unit of the visible light night vision camera 12. Also, for example, the determination of the rotation angle of the camera mount of the pan head 14 based on the position of the intruding object that was supposed to be performed by the data processing device 13 may be performed by the control unit of the pan head 14.

[0054] (7) The surveillance system 1 may be equipped with a normal visible light camera instead of the visible light night vision camera 12. In this case, images cannot be captured under weak ambient light, but sufficient surveillance can be performed, for example, when surveillance is required only during the daytime or when there is lighting that illuminates the surveillance area X with visible light at night.

[0055] The same applies to the visible light night vision camera 16 that is provided in the monitoring system 1 in the modification (3). That is, instead of the visible light night vision camera 16, a normal visible light camera may be mounted on the autonomous moving body 15.

[0056] (8) One monitoring area X may be covered by the imaging ranges of multiple visible light night vision cameras 12. FIG. 7 is a diagram showing the configuration of a monitoring system 1 according to this modification. The monitoring system 1 of FIG. 7 includes two visible light night vision cameras 12, namely, visible light night vision cameras 12(1) and 12(2). Visible light night vision camera 12(1) images approximately half of the monitoring area X on the side of the wave transmitting unit 111 and wave receiving unit 112 of the intrusion monitoring sensor 11, and visible light night vision camera 12(2) images approximately half of the monitoring area X on the side of the reflector 113 of the intrusion monitoring sensor 11.

[0057] (9) Intrusion monitoring sensor 11 may use terahertz waves instead of millimeter waves to detect intruders. Intrusion monitoring sensors using terahertz waves can monitor only a narrower range than those using millimeter waves, but can pinpoint the location of an intruder with high accuracy. Therefore, when the monitoring area is narrow, providing monitoring system 1 with intrusion monitoring sensor 11 using terahertz waves improves the focusing accuracy on intruders and enables the acquisition of clearer images.

[0058] (10) The intrusion monitoring sensor 11 may perform monitoring by simultaneously using radio waves in a plurality of frequency bands with different characteristics.

[0059] In this modification, the intrusion monitoring sensor 11 has the function of substantially simultaneously transmitting and receiving radio waves in the Sub6 band (6 GHz or less), millimeter wave band (30-300 GHz), and sub-terahertz band (100-300 GHz), for example.

[0060] The data processing device 13 comprehensively processes information obtained from radio waves in these frequency bands. Specifically, it performs rough intrusion detection based on information from the Sub6 band, which has excellent wide-area monitoring and penetration through obstacles, identifies the detailed position and shape of the intruder based on information from the millimeter wave band, which is suitable for high-resolution detection, and identifies the material of the intruder based on information from the sub-terahertz band, which has different reflection and transmission characteristics depending on the material.

[0061] In this way, by using information from a plurality of frequency bands with different characteristics in a complementary manner, it is possible to achieve extremely high-precision intrusion detection that is less affected by weather and other environmental factors.

[0062] (11) When the intrusion monitoring sensor 11 uses terahertz waves as in the modified example (9), it may be provided with a function to identify the material of an object carried by an intruder by utilizing the reflection and transmission characteristics of the terahertz waves.

[0063] Terahertz waves exhibit unique frequency spectra (reflection and transmission characteristics) depending on the material, such as metal, plastic, ceramic, fabric, liquid, or powder. For this reason, the data processing device 13 or the control unit of the intrusion monitoring sensor 11 may be configured to have a material identification function that performs spectroscopic analysis of the radio waves received by the wave receiving unit 112, for example, in the range of 0.1 THz to 1 THz. This material identification function identifies the object possessed by the intruder by comparing the analyzed frequency spectrum with pre-stored spectral data of dangerous objects, and determines whether the identified object is a dangerous object such as a metal weapon or explosive.

[0064] According to this modified example, it is possible to identify non-metallic dangerous objects (such as ceramic knives and plastic bombs) that are difficult to detect with conventional metal detectors, thereby significantly improving the security level.

[0065] (12) The data processing device 13 may have a function of using artificial intelligence technology such as deep learning to learn intrusion patterns from accumulated log data and predict and analyze the future behavior of newly detected intruders. This modification enables the monitoring system to go beyond simple detection and recording to take preventive measures based on predictions.

[0066] Specifically, this monitoring system uses the previously recorded detection logs (intrusion location, intrusion time) of the intrusion monitoring sensor 11 and the video logs (movement path, movement speed, behavior, etc. of the intruder) captured by the visible light night vision camera 12 as training data. From this massive amount of data, machine learning is performed using a recurrent neural network such as an LSTM (Long Short-Term Memory) network, and a trained model for predicting intrusion behavior is constructed.

[0067] During system operation, when a new intruder is detected, the initial data of the intruder (location, time of intrusion, and initial behavior obtained from captured video) is input into the pre-built trained model. Based on the input initial behavior, the model outputs a predicted behavior pattern that the intruder is likely to take in the future.

[0068] When constructing this trained model, the explanatory variables (inputs to the model) and objective variables (outputs from the model) of the training data can include information such as the following:

[0069] Explanatory variables (inputs to the model): Features that indicate the initial state of an intrusion during a certain period after the intrusion is detected. Specifically, they include, for example, the following:

[0070] Initial state: The location and time when the intrusion was detected. Environmental information such as the day of the week and weather may also be included. Type of intrusion: The type of intrusion determined by image recognition (human, vehicle, animal, etc.). Time series of initial behavior: Time series data of movement path, speed, and acceleration. Data that categorizes human postures (bending, running, etc.) and behaviors that show vigilance toward surroundings. Information about the presence and type of personal belongings (bags, tools, etc.).

[0071] Objective variable (output from the model): The future behavior of the intruder that the model should predict. Specifically, it includes, for example, the following:

[0072] Future movement path: Coordinate data of the movement path for the next certain period. This will predict the intruder's destination and route. Future behavior category: A label for the behavior that is predicted to be ultimately taken (e.g., "attempt to enter building," "approach facility," "leave premises"). This determines the intent and risk of the intrusion. Risk Score: A numerical score that quantifies the likelihood that an intrusion will lead to a serious security incident, based on past cases of actual damage.

[0073] In this way, by predicting future behavior using a trained model that has learned from past events, more advanced and efficient security responses can be realized, such as proactive deployment of security guards, instructions for focused surveillance using specific cameras, and automatic decisions on whether to report to the police. [Explanation of symbols]

[0074] 1...surveillance system, 11...intrusion monitoring sensor, 12...visible light night vision camera, 13...data processing device, 14...platform head, 15...autonomous mobile body, 16...visible light night vision camera, 17...server device, 111...transmitting unit, 112...receiving unit, 113...reflector.

Claims

1. an intrusion monitoring sensor that detects an intruder that has intruded into a monitoring area using millimeter waves or terahertz waves and identifies the position of the intruder in the monitoring area; a visible light camera that continuously captures an image of the monitoring area and generates a color image both while the intrusion monitoring sensor is not detecting an intrusion and while the intrusion monitoring sensor is detecting an intrusion; Equipped with When an intrusion is detected by the intrusion monitoring sensor, the visible light camera focuses on the position of the intrusion identified by the intrusion monitoring sensor; The visible light camera is a visible light night vision camera equipped with a BSI CMOS sensor, a stacked CMOS sensor, a SPAD sensor, an EMCCD sensor, an ICCD sensor, an ICMOS sensor, or an image sensor with higher sensitivity than these sensors. Surveillance system.

2. The visible light camera has an optical zoom function and takes pictures by zooming at a zoom magnification according to the position of an intruder identified by the intrusion monitoring sensor. The monitoring system of claim 1 .

3. A data processing device is provided that determines the type of intruder by image recognition from the image captured by the visible light camera. The monitoring system of claim 1 .

4. The data processing device issues a notification when the determined type of intrusion is a predetermined type. The monitoring system of claim 3 .

5. a control platform having a camera mount rotatable around a vertical axis; the visible light camera is attached to the camera mount; The data processing device instructs the control platform to rotate the camera mount based on a position in an image of an intruder whose type has been determined by image recognition from an image captured by the visible light camera, so that the intruder is within the imaging range of the visible light camera. The monitoring system of claim 3 .

6. An autonomous moving body and a visible light camera mounted on the autonomous moving body, When the determined type of the intrusion object is a predetermined type, the data processing device notifies the autonomous moving body of the position of the intrusion object; In response to a notification from the data processing device, the autonomous moving body moves to a position where an intruder enters the imaging range of the visible light camera mounted on the autonomous moving body. The monitoring system of claim 3 .

7. a plurality of the intrusion monitoring sensors, a plurality of the visible light cameras, and a data processing device; Equipped with When any of the plurality of intrusion monitoring sensors identifies the position of the intrusion object, the data processing device predicts the future position of the intrusion object based on the change in the position of the intrusion object over time, and instructs the visible light cameras among the plurality of visible light cameras whose imaging range includes the future position of the intrusion object to focus on the future position of the intrusion object. The monitoring system of claim 1 .

8. a plurality of the intrusion monitoring sensors; The plurality of intrusion monitoring sensors are arranged such that the longitudinal directions of the monitoring areas of the plurality of intrusion monitoring sensors are parallel to each other and the monitoring areas of the plurality of intrusion monitoring sensors are equally spaced. The monitoring system of claim 1 .

9. The intrusion monitoring sensor simultaneously transmits and receives radio waves in a plurality of different frequency bands, and a data processing device that detects an intruding object, identifies the position of the intruding object, and identifies at least one of the shape and material of the intruding object based on information obtained from each of the radio waves in the plurality of frequency bands. The monitoring system of claim 1 .

10. the intrusion monitoring sensor detects an intrusion using terahertz waves; The intrusion monitoring sensor includes a data processing device that analyzes the frequency spectrum of the terahertz waves received by the sensor and compares the analyzed frequency spectrum with pre-stored spectral data of a specific object to identify the material of the object associated with the detected intrusion. The monitoring system of claim 1 .

11. A data processing device that stores a trained model constructed by machine learning using training data generated using previously accumulated positions and times of intruders identified by an intrusion monitoring sensor and videos of the intruders captured by a visible light camera, and that predicts at least one of the future movement path, behavior category, and danger level of a newly detected intruder by inputting the positions and times of the intruder identified by the intrusion monitoring sensor and the videos of the intruder captured by the visible light camera into the trained model. The monitoring system of claim 1 .

12. On the computer, a process of acquiring the position of an intruder from an intrusion monitoring sensor that detects an intruder that has intruded into a monitoring area using millimeter waves or terahertz waves and identifies the position of the intruder in the monitoring area; a process of outputting a control signal to a visible light night vision camera that is a visible light camera that continuously captures an image of the monitoring area and generates a color image both while the intrusion monitoring sensor is not detecting an intrusion and while the intrusion monitoring sensor is detecting an intrusion, the visible light night vision camera having any of a BSI CMOS sensor, a stacked CMOS sensor, a SPAD sensor, an EMCCD sensor, an ICCD sensor, an ICMOS sensor, and an image sensor with higher sensitivity than these sensors, to instruct the camera to focus on the position of the intrusion identified by the intrusion monitoring sensor; A program to execute.

Images