Surveillance equipment and surveillance systems

Abstract

To provide a monitoring device which excludes a camera image with possible false detection from an object of abnormality detection processing in advance to prevent the false detection in the abnormality detection processing.SOLUTION: A monitoring server which detects abnormality around the track on the basis of a camera image captured by a camera installed on the railway line sets, on the camera image, an object area of abnormality detection processing on the basis of an operation input of a user, performs the abnormality detection processing which detects abnormality of the object area on the basis of the real-time camera image output by the camera, outputs, upon detecting the abnormality of the object area, notification information for notifying a watcher of the occurrence of the abnormality, and furthermore, prior to the abnormality detection processing, performs exclusion determination processing for determining whether or not the real-time camera image is excluded from the abnormality detection processing.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a monitoring device and a monitoring system that use a processor to execute processing to detect abnormalities around a track based on camera images output from a camera. [Background technology]

[0002] On railways, abnormalities around the tracks, such as falling rocks, landslides, avalanches, and other obstacles on the tracks, can sometimes hinder the safe passage of trains. For this reason, in order to respond quickly to abnormalities around the tracks, wayside monitoring systems have become widespread, which enable remote monitoring of abnormalities around the tracks by taking pictures of the area around the tracks with cameras installed along the railway line.

[0003] In such a wayside monitoring system, monitors can visually check for abnormalities around the track by viewing camera images, but if a monitoring device is installed that detects abnormalities around the track and issues an alert to alert the monitor when an abnormality around the track is detected, the burden on the monitor can be reduced.

[0004] A conventional monitoring device for detecting such abnormalities around the track is a technology that compares real-time camera images of the monitored area with camera images of normal conditions, and detects abnormalities by detecting large changes that appear in the real-time camera images (see Patent Document 1). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-291935 Summary of the Invention [Problem to be solved by the invention]

[0006] Now, there are many events that cause significant changes in camera images even when there are no abnormalities around the tracks, i.e., obstacles on the tracks such as falling rocks, landslides, avalanches, etc. that are interfering with the safe passage of trains. For example, significant changes appear in camera images when the camera rotates or performs PTZ operations (pan, tilt, zoom), when a train passes on the tracks, during heavy rainfall or snowfall, when train or automobile headlights shine on the camera, or when an object such as an insect flies near the camera lens.

[0007] However, with conventional anomaly detection methods, when an event like the one described above occurs and a large change appears in the camera image, a false detection occurs, and a false alarm is sent to the monitor.If these false alarms occur frequently, the task of checking them becomes a heavy burden for the monitor.

[0008] Therefore, the main object of the present invention is to provide a monitoring device and a monitoring system that can suppress false detections in anomaly detection processing by excluding camera images that are likely to result in false detections in anomaly detection processing from the targets of the anomaly detection processing in advance. [Means for solving the problem]

[0009] The monitoring device of the present invention is a monitoring device in which a processor executes an anomaly detection process that detects an abnormality in a target area based on a camera image captured by a camera, wherein the processor sets the target area on the camera image based on a user's operation input, and performs the anomaly detection process that detects an abnormality in the target area based on the real-time camera image output from the camera, and further, prior to the anomaly detection process, acquires movement information of a subject based on the real-time camera image, and performs an exclusion determination process that determines whether or not the camera image should be excluded from the target of the anomaly detection process based on the movement information, and when an abnormality in the target area is detected, outputs notification information to notify a monitor of the occurrence of the abnormality, and, depending on the detection result of the anomaly detection process and the determination result of the exclusion determination process, A monitoring camera image is obtained by determining whether the target area is in a normal state, an abnormal state, or an exclusion state, and synthesizing a state image representing the normal state, the abnormal state, or the exclusion state with the camera image.The configuration is to generate and output the following.

[0010] Furthermore, a monitoring device of the present invention is a monitoring device in which a processor executes an anomaly detection process that detects anomalies in a target area based on a camera image captured by a camera, wherein the processor sets the target area on the camera image based on a user's operation input, and performs the anomaly detection process that detects anomalies in the target area based on the real-time camera image output from the camera, and further, prior to the anomaly detection process, acquires motion information of a subject based on the real-time camera image, and performs an exclusion determination process that determines whether or not the camera image should be excluded from the anomaly detection process based on the motion information, and depending on the detection result of the anomaly detection process and the determination result of the exclusion determination process, A monitoring camera image is obtained by determining whether the target area is in a normal state, an abnormal state, or an exclusion state, and synthesizing a state image representing the normal state, the abnormal state, or the exclusion state with the camera image. The configuration is to generate and output the following.

[0011] Furthermore, a monitoring system of the present invention is a monitoring system comprising a camera that photographs a target area, and an information processing device that executes anomaly detection processing that detects abnormalities in the target area based on camera images photographed by the camera, wherein the information processing device sets the target area on the camera image based on a user's operation input, and performs the anomaly detection processing that detects abnormalities in the target area based on the real-time camera image output from the camera, and further, prior to the anomaly detection processing, acquires movement information of a subject based on the real-time camera image, and performs exclusion determination processing that determines whether or not the camera image should be excluded from the target of the anomaly detection processing based on the movement information, and when an abnormality in the target area is detected, outputs notification information to notify a monitor of the occurrence of the abnormality, and, depending on the detection result of the anomaly detection processing and the determination result of the exclusion determination processing, A monitoring camera image is obtained by determining whether the target area is in a normal state, an abnormal state, or an exclusion state, and synthesizing a state image representing the normal state, the abnormal state, or the exclusion state with the camera image. The configuration is to generate and output the following.

[0012] Furthermore, a monitoring system of the present invention is a monitoring system comprising a camera that photographs a target area, and an information processing device that executes anomaly detection processing that detects abnormalities in the target area based on camera images photographed by the camera, wherein the information processing device sets the target area on the camera image based on a user's operation input, and executes the anomaly detection processing that detects abnormalities in the target area based on the real-time camera image output from the camera, and further, prior to the anomaly detection processing, acquires motion information of a subject based on the real-time camera image, and executes exclusion determination processing that determines whether or not the camera image should be excluded from the target of the anomaly detection processing based on the motion information, and depending on the detection result of the anomaly detection processing and the determination result of the exclusion determination processing, A monitoring camera image is obtained by determining whether the target area is in a normal state, an abnormal state, or an exclusion state, and synthesizing a state image representing the normal state, the abnormal state, or the exclusion state with the camera image. The configuration is to generate and output the following. [Effects of the Invention]

[0013] According to the present invention, Depending on the detection results of the anomaly detection process and the judgment results of the exclusion judgment process, the target area is judged to be in a normal state, an abnormal state, or an exclusion state, and a monitoring camera image is output in which a status image indicating whether the target area is in a normal state, an abnormal state, or an exclusion state is synthesized with the camera image, allowing the monitor to immediately understand the state of the target area. It is possible. [Brief explanation of the drawings]

[0014] [Figure 1] Overall configuration of a railway line monitoring system according to the present embodiment. [Figure 2] FIG. 10 is an explanatory diagram showing a monitoring screen displayed on the monitoring terminal 4. [Figure 3] Block diagram showing the general configuration of the monitoring server 2 [Figure 4] An explanatory diagram showing an overview of the processing performed by monitoring server 2. [Figure 5] FIG. 10 is an explanatory diagram showing the timing of display processing and notification processing according to the detection result of the abnormality detection processing and the determination result of the exclusion determination processing performed by the monitoring server 2. [Figure 6] FIG. 10 is an explanatory diagram showing the area setting screen displayed on the display unit of the monitoring server 2. [Figure 7] FIG. 10 is an explanatory diagram showing an overview of the exclusion determination process performed by the monitoring server 2. [Figure 8] An explanatory diagram showing an overview of the abnormality detection process performed by the monitoring server 2. [Figure 9]Flow diagram showing the operation procedure of monitoring server 2 DETAILED DESCRIPTION OF THE INVENTION

[0015] A first invention made to solve the above problems is a monitoring device that uses a processor to execute an anomaly detection process that detects an abnormality in a target area based on a camera image captured by a camera, wherein the processor sets the target area on the camera image based on a user's operation input, and performs the anomaly detection process that detects an abnormality in the target area based on the real-time camera image output from the camera, and further, prior to the anomaly detection process, acquires movement information of a subject based on the real-time camera image, and performs an exclusion determination process that determines whether or not the camera image should be excluded from the target of the anomaly detection process based on the movement information, and when an abnormality in the target area is detected, outputs notification information to notify a monitor of the occurrence of the abnormality, and, depending on the detection result of the anomaly detection process and the determination result of the exclusion determination process, A monitoring camera image is obtained by determining whether the target area is in a normal state, an abnormal state, or an exclusion state, and synthesizing a state image representing the normal state, the abnormal state, or the exclusion state with the camera image. The configuration is to generate and output the following.

[0016] According to this, Depending on the detection results of the anomaly detection process and the judgment results of the exclusion judgment process, the target area is judged to be in a normal state, an abnormal state, or an exclusion state, and a monitoring camera image is output in which a status image indicating whether the target area is in a normal state, an abnormal state, or an exclusion state is synthesized with the camera image, allowing the monitor to immediately understand the state of the target area. It is possible.

[0017] In addition, in a second invention, the processor is configured to detect abnormalities in the target area using a classifier that has been trained in advance using camera images in an abnormal state and camera images in a normal state as the abnormality detection process.

[0018] This allows the abnormality detection process to be performed with high accuracy.

[0019] In addition, in a third invention, the processor is configured to perform the exclusion determination process using information indicating the direction and magnitude of a motion vector as the motion information to determine whether or not to exclude the camera image from the target of the anomaly detection process.

[0020] This makes it possible to exclude camera images in a state where there is a high possibility of false positives occurring in the anomaly detection process from the target of the anomaly detection process. For example, false positives may occur when the camera is rotating or performing a PTZ operation, when a train is passing on the track, when poor image quality occurs due to heavy rainfall or snowfall, when poor image quality occurs due to illumination by train or automobile headlights, or when poor image quality occurs due to an object such as an insect floating near the camera, and camera images capturing such events can be excluded from the target of the anomaly detection process.

[0021] In addition, in a fourth invention, when an exclusion state in which a judgment result is obtained that the camera image is excluded from the target of anomaly detection processing continues for more than a predetermined time, the processor sets the state to a continuous exclusion state and outputs the notification information indicating this.

[0022] This allows the monitor to know that the exclusion state, that is, the state in which the anomaly detection process is not being carried out, has continued for a long period of time.

[0023] In addition, in a fifth invention, the processor is configured to output the notification information indicating that the continuous exclusion state is lifted when a determination result is obtained that the camera image should not be excluded from the target of the abnormality detection processing and the continuous exclusion state is lifted.

[0024] This allows the monitor to understand that the exclusion state has been lifted and the state has returned to one in which the abnormality detection process can be performed.

[0025] A sixth invention is a monitoring device that uses a processor to execute an anomaly detection process that detects an anomaly in a target area based on a camera image captured by a camera, wherein the processor sets the target area on the camera image based on a user's operation input, and performs the anomaly detection process that detects an anomaly in the target area based on the real-time camera image output from the camera, and further, prior to the anomaly detection process, acquires motion information of a subject based on the real-time camera image, and performs exclusion determination process that determines whether or not the camera image should be excluded from the anomaly detection process based on the motion information, and depending on the detection result of the anomaly detection process and the determination result of the exclusion determination process, A monitoring camera image is obtained by determining whether the target area is in a normal state, an abnormal state, or an exclusion state, and synthesizing a state image representing the normal state, the abnormal state, or the exclusion state with the camera image. The configuration is to generate and output the following.

[0026] According to this, Depending on the detection results of the anomaly detection process and the judgment results of the exclusion judgment process, the target area is judged to be in a normal state, an abnormal state, or an exclusion state, and a monitoring camera image is output in which a status image indicating whether the target area is in a normal state, an abnormal state, or an exclusion state is synthesized with the camera image, allowing the monitor to immediately understand the state of the target area. It is possible.

[0027] In addition, in a seventh invention, the processor is configured to output, as the display information, information that superimposes a frame image indicating the target area on the camera image in a display form corresponding to the detection result of the anomaly detection process and the judgment result of the exclusion judgment process.

[0028] This allows the observer to immediately grasp the detection results of the anomaly detection process and the determination results of the exclusion determination process based on the display format of the frame image superimposed on the camera image.

[0029] An eighth invention is a surveillance system comprising a camera that photographs a target area, and an information processing device that executes anomaly detection processing to detect anomalies in the target area based on camera images photographed by the camera, wherein the information processing device sets the target area on the camera image based on a user's operation input, and performs the anomaly detection processing to detect anomalies in the target area based on the real-time camera image output from the camera, and further, prior to the anomaly detection processing, acquires motion information of a subject based on the real-time camera image, and performs exclusion determination processing to determine whether or not the camera image should be excluded from the target of the anomaly detection processing based on the motion information, and when an anomaly in the target area is detected, outputs notification information to notify a monitor of the occurrence of the anomaly, and, depending on the detection result of the anomaly detection processing and the determination result of the exclusion determination processing, A monitoring camera image is obtained by determining whether the target area is in a normal state, an abnormal state, or an exclusion state, and synthesizing a state image representing the normal state, the abnormal state, or the exclusion state with the camera image. The configuration is to generate and output the following.

[0030] According to this, as with the first invention, Monitors can instantly understand whether the target area is in a normal, abnormal, or excluded state. It is possible.

[0031] A ninth invention is a monitoring system comprising a camera that photographs a target area, and an information processing device that executes anomaly detection processing to detect anomalies in the target area based on a camera image photographed by the camera, wherein the information processing device sets the target area on the camera image based on a user's operation input, and performs the anomaly detection processing to detect anomalies in the target area based on the real-time camera image output from the camera, and further, prior to the anomaly detection processing, acquires motion information of a subject based on the real-time camera image, and performs exclusion determination processing to determine whether or not the camera image should be excluded from the target of the anomaly detection processing based on the motion information, and depending on the detection result of the anomaly detection processing and the determination result of the exclusion determination processing, A monitoring camera image is obtained by determining whether the target area is in a normal state, an abnormal state, or an exclusion state, and synthesizing a state image representing the normal state, the abnormal state, or the exclusion state with the camera image. The configuration is to generate and output the following.

[0032] According to this, as with the sixth invention, Monitors can instantly understand whether the target area is in a normal, abnormal, or excluded state. It is possible.

[0033] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0034] (First embodiment) FIG. 1 is a diagram showing the overall configuration of a railway line monitoring system according to this embodiment.

[0035] This wayside monitoring system comprises a camera 1, a monitoring server 2 (wayside monitoring device, information processing device), a recorder 3 (camera image storage device), a monitoring terminal 4 (camera image viewing device), a warning light 5 (alarm device), and a user terminal 6. The camera 1, monitoring server 2, recorder 3, monitoring terminal 4, warning light 5, and user terminal 6 are connected via a network.

[0036] The camera 1 is installed along the railway line and captures images of the monitoring area around the track. The camera 1 also transmits the camera images to the monitoring server 2.

[0037] The monitoring server 2 performs image analysis on camera images of the monitoring area to detect abnormalities around the track, i.e., conditions that hinder the safe passage of trains due to obstacles on the track such as falling rocks, landslides, avalanches, etc. (anomaly detection process). Prior to the anomaly detection process, the monitoring server 2 also performs image analysis on the camera images to determine whether or not to exclude the camera images from the anomaly detection process (exclusion determination process).

[0038] Furthermore, when the monitoring server 2 detects an abnormality around the track, it generates and outputs notification information to notify the monitor of the abnormality. In this embodiment, the notification information is sent to the user terminal 6 as an email containing a message about the abnormality detection. Here, the notification information is not limited to email, and may be sent to a pre-designated user terminal 6 using a push function. Furthermore, as the notification information, control information is sent to the warning light 5 to instruct the warning light 5 to perform a notification operation (lighting up or flashing).

[0039] Furthermore, the monitoring server 2 sets a target area on a camera image capturing the monitored area in response to a specified operation by the administrator (user). The monitoring server 2 then generates a monitoring camera image in which the frame line of the target area is superimposed on the camera image, and transmits the generated image to the recorder 3. At this time, the frame line of the target area is drawn in a display format according to the detection result of the anomaly detection process and the determination result of the exclusion determination process. The monitoring server 2 also transmits the time of alert and the identification information of the camera 1 as alert information to the recorder 3.

[0040] The recorder 3 stores the surveillance camera images received from the surveillance server 2. The recorder 3 also stores the alert information (alert time, identification information of camera 1, etc.) received from the surveillance server 2 in association with the surveillance camera images. In the above example, the camera images are acquired from the surveillance server 2 into the recorder 3, but the camera images may also be stored in the recorder 3 directly from the camera 1.

[0041] The monitoring terminal 4 is installed in the monitoring room. The monitoring terminal 4 displays the monitoring camera images acquired from the recorder 3 on the monitoring screen. This allows the monitor to specifically check the status of the monitored area. At this time, the monitoring terminal 4 acquires the real-time monitoring camera images generated by the monitoring server 2 via the recorder 3, allowing the monitor to view the real-time monitoring camera images. In addition, the monitoring terminal 4 acquires the past monitoring camera images stored in the recorder 3 from the recorder 3, allowing the monitor to view the past monitoring camera images.

[0042] The warning light 5 is installed in the monitoring room. This warning light 5 performs a predetermined notification operation (lighting up or blinking) based on control information transmitted from the monitoring server 2. This allows the monitor in the monitoring room to recognize that an abnormality has been detected around the track in the monitoring area. Note that an alarm device that outputs a sound to notify of an abnormality around the track may be used together with or instead of the warning light 5.

[0043] The user terminal 6 is composed of a tablet, smartphone, or the like carried by the monitor. This user terminal 6 receives emails sent from the monitoring server 2. As a result, even if the monitor is not in the monitoring room, he or she can recognize that an abnormality has been detected around the track by viewing the anomaly detection message contained in the email. After viewing the anomaly detection message, the monitor can also view the monitoring camera images on the monitoring terminal 4 to confirm the specific circumstances of the abnormality.

[0044] The surveillance camera image at the time of the alarm may be attached to the e-mail. Also, link information for acquiring the surveillance camera image from the recorder 3 may be attached to the e-mail. This allows the monitor to view the surveillance camera image on the user terminal 6.

[0045] Next, we will explain the monitoring screen displayed on the monitoring terminal 4. Fig. 2 is an explanatory diagram showing the monitoring screen. When the monitor specifies multiple cameras 1, camera images corresponding to the specified multiple cameras 1 are displayed side by side on the monitoring screen.

[0046] Figure 2(A) is a monitoring screen in a state where no abnormalities are detected in the target area of ​​the camera image (normal state). Figure 2(B) is a monitoring screen in a state where an abnormality is detected in the target area of ​​the camera image (abnormal state). Figure 2(C) is a monitoring screen in a state where the camera image is excluded from the target of the abnormality detection process (exclusion state).

[0047] As shown in FIG. 2(A), when no abnormality is detected in the target area in the camera image, the frame line (frame image) representing the target area is drawn in black.

[0048] On the other hand, as shown in FIG. 2(B), when an abnormality is detected in the target area in the camera image, the border of the target area is highlighted, specifically, the border of the target area changes from black to red. Furthermore, the border is drawn thicker for a predetermined time from the time when an alarm is issued, i.e., when the message indicating the abnormality detection and the control information (alert information) for the warning light 5 are output. In the example shown in FIG. 2(B), an abnormality is detected because an obstacle (such as a falling rock) is present on the track within the target area.

[0049] Furthermore, as shown in Figure 2(C), when a camera image is excluded from the target of anomaly detection processing, the frame line representing the target area changes from black to gray. Note that in the example shown in Figure 2(C), a train is passing through the target area, which is an event to be excluded, so the camera image is excluded from the target of anomaly detection processing.

[0050] In this manner, in this embodiment, the frame lines representing the target areas are drawn in different colors (black, red, gray) depending on the detection results of the anomaly detection process and the determination results of the exclusion determination process, allowing the monitor to immediately grasp the detection results of the anomaly detection and the determination results of the exclusion determination.

[0051] In this embodiment, the color of the border line is changed as an example of changing the display form of the border line (border image), but other display forms, such as the thickness or type (dotted line, etc.) of the border line, may also be changed, or the border line may be displayed in a flashing manner.

[0052] Furthermore, the display information representing the detection results of the anomaly detection process and the determination results of the exclusion determination process is not limited to a frame line (frame image) as in the present embodiment. That is, the detection results of the anomaly detection process and the determination results of the exclusion determination process may be represented by other forms of display information, such as a display frame of a camera image or a balloon (speech bubble).

[0053] In the example shown in FIG. 2(B), the presence of an obstacle (such as a fallen rock) on the track is detected as an abnormality around the track, but other equipment such as overhead lines may also be detected in the abnormality detection process.

[0054] Next, we will explain the general configuration of the monitoring server 2. Fig. 3 is a block diagram showing the general configuration of the monitoring server 2. Fig. 4 is an explanatory diagram showing an overview of the processing performed by the monitoring server 2. Fig. 5 is an explanatory diagram showing the timing of the display processing and notification processing according to the detection result of the anomaly detection processing and the determination result of the exclusion determination processing performed by the monitoring server 2.

[0055] As shown in FIG. 3, the monitoring server 2 includes a communication unit 21, a display unit 22, an operation input unit 23, a memory 24, and a processor 25.

[0056] The communication unit 21 communicates with the camera 1, recorder 3, monitoring terminal 4, warning light 5, and user terminal 6 via the network. Specifically, the communication unit 21 receives camera images transmitted from the camera 1. The communication unit 21 also transmits display information for the monitoring screen to the monitoring terminal 4 and receives operation information from the monitoring terminal 4. The communication unit 21 also transmits control information to the warning light 5. The communication unit 21 also transmits e-mails to the user terminal 6.

[0057] The display unit 22 displays an operation screen relating to the operating conditions of the device itself.

[0058] The operation input unit 23 allows the administrator to operate the screen displayed on the display unit 22.

[0059] The memory 24 stores programs executed by the processor 25. The memory 24 also temporarily stores camera images acquired from the camera 1.

[0060] The processor 25 performs various processes related to wayside monitoring by executing the programs stored in the memory 24. In this embodiment, the processor 25 performs area setting process, exclusion determination process, abnormality detection process, display process, notification process, and the like.

[0061] In the area setting process, processor 25 sets a target area for the anomaly detection process on the camera image based on an operation input by the administrator. In this embodiment, the range of the target area is set by the administrator performing an operation to specify the range of the target area on the area setting screen (see FIG. 6).

[0062] In the exclusion determination process, processor 25 determines whether or not to exclude a real-time camera image from the target of the anomaly detection process. Specifically, the processor 25 detects the movement of a subject by comparing camera images (frames) sequentially output from camera 1 at each time, obtains movement information regarding the movement of the subject, and determines whether or not to exclude the camera image from the target of the anomaly detection process based on the movement information.

[0063] In the anomaly detection process, processor 25 detects anomalies in the target area based on real-time camera images. Specifically, an anomaly in the target area is detected using a classifier (machine learning model) generated by machine learning (e.g., deep learning). This anomaly detection process can detect anomalies around the track, i.e., a state in which safe train passage is hindered by obstacles on the track caused by falling rocks, landslides, avalanches, etc.

[0064] In the display process, processor 25 generates a surveillance camera image (display information) by combining a frame line (frame image) of the target area with the camera image, and outputs the surveillance camera image to recorder 3 (see FIG. 4). This allows the surveillance camera image to be displayed by monitoring terminal 4 or user terminal 6 accessing recorder 3. In this display process, the frame line of the target area is drawn in a display form (black, red, gray) according to the detection result of the anomaly detection process and the determination result of the exclusion determination process.

[0065] In this embodiment, in a normal state where the exclusion determination process determines not to exclude the camera image from the anomaly detection process and no abnormality is detected in the target area in the anomaly detection process, the border of the target area is drawn in black (see FIGS. 2(A), 5(A), and (B)). On the other hand, when the anomaly detection process detects an abnormality in the target area and the state transitions to an abnormal state, the border of the target area changes to red (see FIGS. 2(B) and 5(A)). Furthermore, when the exclusion determination process determines to exclude the camera image from the anomaly detection process and the state transitions to the excluded state, the border of the target area changes to gray (see FIGS. 2(C) and 5(B)).

[0066] In the notification process, when an abnormality in the target area is detected in the abnormality detection process, the processor 25 generates and outputs notification information to notify the monitor that an abnormality has been detected. This notification information is output when a predetermined notification condition is met. In this embodiment, the notification condition is that the notification information is output when a state in which an abnormality has been detected in sequentially input camera images (an abnormal state) continues for a predetermined period of time or more. The notification information includes the time of notification, identification information of the camera 1, etc.

[0067] In this embodiment, an email containing a message about the abnormality detection is sent to the user terminal 6 as notification information for notifying the supervisor that an abnormality has been detected (see FIGS. 4 and 5(A)). Also, as notification information, control information instructing the warning light 5 to perform a notification operation (such as lighting up or blinking) is sent to the warning light 5 (see FIGS. 4 and 5(A)).

[0068] Furthermore, in the notification process, the processor 25 determines in the exclusion determination process to exclude the camera image from the anomaly detection process, and if this state continues for a predetermined time or longer, an email containing a message about the continuous exclusion state is sent to the user terminal 6 as notification information to notify the monitor that the exclusion state, i.e., the state in which anomaly detection is not being performed, has continued for a long time (see Figures 4 and 5(B)).

[0069] Furthermore, in this embodiment, after transitioning to the continuous exclusion state, if the exclusion determination process determines that the camera image should not be excluded from the anomaly detection process, an email containing a message indicating that the continuous exclusion state has been released is sent to the user terminal 6 as notification information to notify the monitor that the continuous exclusion state has been released, i.e., that the state has returned to one in which anomaly detection can be performed (see Figures 4 and 5(B)).

[0070] In this embodiment, if the state in which the camera image is excluded from the anomaly detection process (exclusion state) continues for a predetermined time or more, a notification is sent to the monitor. This allows the monitor to understand the exclusion state, i.e., that the anomaly detection process is not being performed. Furthermore, in the case of an exclusion event that continues for a long time, such as rainfall or snowfall, a notification is sent to the monitor. However, in the case of an exclusion event that does not require a notification and is resolved in a short time, such as the passing of a train, a notification is not sent to the monitor, thereby preventing unnecessary notifications.

[0071] In this embodiment, the monitoring server 2 generates a monitoring camera image by superimposing the frame line of the target area onto the camera image and transmits it to the recorder 3, but the camera image, position information of the target area on the camera image, and information on the display format (color, etc.) of the frame line of the target area may also be transmitted as display information to the recorder 3. In this case, the monitoring terminal 4 or the user terminal 6 performs a process of superimposing the frame line of the target area onto the camera image based on the display information accumulated in the recorder 3, thereby displaying the monitoring camera image.

[0072] Next, a description will be given of the area setting screen displayed on the display unit 22 of the monitoring server 2. Fig. 6 is an explanatory diagram showing the area setting screen.

[0073] In the monitoring server 2, an operation screen relating to the operating conditions of the device itself is displayed on the display unit 22. In this embodiment, an area setting screen for setting a target area for anomaly detection processing is displayed on the display unit 22 as one of the operation screens.

[0074] This area setting screen displays a camera image of the monitoring area taken by camera 1, and allows the administrator to specify the range of the target area on the camera image. In the example shown in FIG. 6, the range of the target area is specified by inputting the positions of the vertices of a polygon that represents the range of the target area. This causes a frame line (frame image) representing the range of the target area to be superimposed on the camera image. Note that if camera 1 uses the PTZ function to capture images of multiple monitoring areas, it is possible to set a target area for anomaly detection processing for each monitoring area after PTZ operation.

[0075] Here, the abnormality detection process detects abnormalities around the track, i.e., conditions that hinder the safe passage of trains due to obstacles on the track caused by falling rocks, landslides, avalanches, etc., so the target area for the abnormality detection process is set to surround the required range of track (rails).

[0076] In this embodiment, the administrator directly operates the monitoring server 2 to specify the range of the target area, but it is also possible for a management terminal (not shown) to access the monitoring server 2 and for the administrator to operate the management terminal to specify the range of the target area.

[0077] Next, a description will be given of the exclusion determination process carried out by the monitoring server 2. Fig. 7 is an explanatory diagram showing an outline of the exclusion determination process.

[0078] The monitoring server 2 performs an exclusion determination process by analyzing the real-time camera image transmitted from the camera 1 to determine whether or not to exclude the real-time camera image from the anomaly detection process. This exclusion determination process detects the movement of a subject by comparing the real-time camera image with the immediately preceding camera image, acquires motion information related to the subject's movement, and determines whether or not to exclude the real-time camera image from the anomaly detection process based on the motion information. Note that this exclusion determination process can use, for example, a method using block matching. Furthermore, the exclusion determination process can make a determination based on the direction and magnitude of the motion (vector) acquired as the motion information.

[0079] By using this exclusion determination process, if an event that should be excluded from the anomaly detection process (an excluded event) occurs in a real-time camera image as an event that may cause a false detection in the anomaly detection process, the camera image can be excluded from the anomaly detection process.

[0080] Here, camera images in which the following exclusion target events occur can be excluded from the anomaly detection process by the exclusion determination process. Note that in this embodiment, each exclusion target event is not identified, but it is also possible to identify the type of each event using a classifier or the like equipped with learning data for each event.

[0081] First, the first example of an event to be excluded is when camera 1 is rotating or performing PTZ operations (pan, tilt, zoom), as shown in FIG. 7(A). Camera 1 installed along a railway line may monitor both the tracks on the uphill and downhill sides of the installation point. In this case, camera 1 automatically rotates periodically to switch the monitoring area. Also, an observer may manually operate camera 1 to check the situation in an area different from the normal monitoring area. In this case, camera 1 performs PTZ operations in response to the observer's operation. In such a case, when camera 1 is rotating or performing PTZ operations, the image changes drastically overall, which increases the likelihood of false detections occurring in the anomaly detection process.

[0082] The second example of an event to be excluded is when a train is passing over the track, as shown in Figure 7(B). In this case, the train body covers the track, so any abnormalities around the track cannot be detected. Also, since the train is passing through normally, it is assumed that there are no abnormalities.

[0083] The third example of an event to be excluded is poor image quality caused by heavy rainfall or snowfall, as shown in Figure 7(C). When there is significant rainfall or snowfall, the camera image is covered with a large number of raindrops or snowflakes, making the condition around the track in the camera image unclear and making it difficult to properly detect abnormalities.

[0084] A fourth example of an event to be excluded is when poor image quality occurs due to light from train or automobile headlights, as shown in Figure 7(D). Camera 1 is illuminated by the light from the train headlights traveling on the track, and also by the light from automobile headlights traveling along the road. When light from such train or automobile headlights enters Camera 1, significant whiteout occurs in the camera image due to the movement of the train or automobile, making it difficult to properly detect abnormalities around the track in the camera image. Note that if a train or automobile is stopped, this is not an event to be excluded, and abnormality detection processing is performed.

[0085] A fifth example of an event to be excluded is when a poor image is caused by an object such as an insect floating near the lens of camera 1, as shown in Figure 7(E). An animal such as an insect may fly near the lens of camera 1, or a plastic bag blown up by the wind may pass by. In this case, these floating objects appear large in the image, obscuring the state around their trajectory in the camera image, making it difficult to properly detect anomalies. Note that a state in which a floating object such as an insect adheres to the lens of camera 1 is not an event to be excluded, and anomaly detection processing is still performed.

[0086] In this embodiment, the exclusion determination process is performed based on the camera image, but the exclusion determination process may be performed based on operation information regarding the rotation or PTZ operation of the camera 1 obtained from the camera 1. This makes it possible to exclude camera images captured during the rotation or PTZ operation of the camera 1 from the targets of the anomaly detection process (see FIG. 7(A)).

[0087] However, while these excluded events occur temporarily and resolve naturally, abnormalities around the track, such as falling rocks, landslides, and avalanches that impede the safe passage of trains, are not temporary and will continue until workers complete restoration work.

[0088] Next, we will explain the abnormality detection process performed by the monitoring server 2. Figure 8 is an explanatory diagram showing an overview of the abnormality detection process. Figure 8(A) shows the situation during learning, and Figure 8(B) shows the situation during operation.

[0089] As shown in Fig. 8(B), the monitoring server 2 performs an anomaly detection process to detect abnormalities around the track in the target area based on the input real-time camera images. In this embodiment, the anomaly detection process is performed using a classifier (machine learning model) generated by machine learning (e.g., deep learning). This anomaly detection process can detect abnormalities around the track, i.e., conditions that hinder the safe passage of trains due to obstacles on the track caused by falling rocks, landslides, avalanches, etc.

[0090] Here, as shown in Fig. 8(A), the classifier is trained in advance using camera images in an abnormal state and camera images in a normal state. As a result, when real-time camera images are input to the classifier, as shown in Fig. 8(B), a detection result indicating whether or not an abnormality has been detected in the real-time camera images is output from the classifier. Note that, since it is difficult to obtain camera images in an actual abnormal state, it is also possible to generate pseudo camera images in an abnormal state from camera images in a normal state.

[0091] Next, we will explain the operation procedure of the monitoring server 2. Fig. 9 is a flow diagram showing the operation procedure of the monitoring server 2. Note that the monitoring server 2 receives camera images (frames) from the camera 1 at each time in sequence.

[0092] In the monitoring server 2, first, the processor 25 acquires a real-time camera image input from the camera 1 (ST101). Next, the processor 25 performs an exclusion determination process to determine whether or not to exclude the input real-time camera image from the target of the anomaly detection process (ST102).

[0093] Here, if the real-time camera images are not excluded from the anomaly detection process (No in ST103), then processor 25 performs anomaly detection process to detect anomalies in the target area based on the real-time camera images (ST104).

[0094] If an abnormality is detected in the target area (Yes in ST105), processor 25 then determines whether the state in which the abnormality is detected (abnormal state) has continued for a predetermined time or more (ST106).

[0095] Here, if the abnormal state continues for a predetermined time or longer (Yes in ST106), the processor 25 sends an email containing a message of abnormality detection to the user terminal 6 as notification information for notifying the supervisor that an abnormality has been detected (ST107).The processor 25 also sends control information (notification information) to the warning light 5 as notification information instructing it to perform a notification operation (ST108).

[0096] Next, processor 25 determines whether or not the input of camera images from camera 1 has finished (ST115). If the input of camera images from camera 1 has not finished (No in ST115), the process returns to ST101 and proceeds to processing the next camera image (frame).

[0097] Furthermore, if the real-time camera image is to be excluded from the anomaly detection processing (Yes in ST103), processor 25 then determines whether the state in which the camera image is excluded from the anomaly detection processing (exclusion state) has continued for a predetermined time or more (ST109).

[0098] If the excluded state continues for a predetermined time or longer (Yes in ST109), the processor 25 sets the state to a continuous excluded state (ST110). Next, the processor 25 sends an email to the user terminal 6 including a message about the excluded state, i.e., a continuous excluded state, indicating that a state in which anomaly detection has not been performed has continued for a long time, as notification information for notifying the monitor that the excluded state has continued for a long time (ST111). Then, the process proceeds to ST115.

[0099] If no abnormality is detected in the target area (No in ST105), processor 25 then determines whether or not the continuous exclusion state is set (ST112).

[0100] Here, if the continuous exclusion state is set (Yes in ST112), the processor 25 cancels the continuous exclusion state (ST113). Next, the processor 25 sends an email to the user terminal 6 as notification information for notifying the monitor that the continuous exclusion state has been canceled, including a continuous exclusion state cancellation message indicating that the continuous exclusion state has been canceled, i.e., that the state has returned to one in which the abnormality detection process can be performed (ST114). Then, the process proceeds to ST115.

[0101] In this embodiment, if the exclusion state continues for a predetermined time or longer, an email containing a message indicating a continuous exclusion state, stating that the exclusion state has continued for a long time, is sent to the user terminal 6 (ST111). At this time, however, in the surveillance camera image displayed on the surveillance terminal 4, the border line of the target area may be drawn in a display format (for example, a thick gray line) different from the display format of the exclusion state (gray line).

[0102] In addition, it is preferable that the administrator can specify an appropriate time (e.g., a few minutes, several tens of minutes, one hour, etc.) as the threshold time when determining whether the excluded state has continued for a predetermined time or more (ST109) on the operation screen of the monitoring server 2.

[0103] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is also possible to combine the components described in the above embodiments to create new embodiments. [Industrial Applicability]

[0104] The monitoring device and monitoring system according to the present invention have the effect of suppressing false detections in the anomaly detection process by excluding, in advance, camera images that are likely to be falsely detected in the anomaly detection process from the targets of the anomaly detection process, and are useful as monitoring devices and monitoring systems that use a processor to execute a process to detect abnormalities around the track based on the camera images output from the camera. [Explanation of symbols]

[0105] 1 camera 2. Monitoring server (trackside monitoring device, information processing device) 3 Recorder 4 Monitoring terminal 5 warning light 6 User terminals 21 Communications Department 22 Display section 23 Operation input section 24 memory 25 processors

Claims

1. A monitoring device that executes an anomaly detection process by a processor to detect an anomaly in a target area based on a camera image captured by a camera, The processor: setting the target area on the camera image based on a user's operation input; performing the anomaly detection process to detect an anomaly in the target area based on the real-time camera image output from the camera; Furthermore, prior to the abnormality detection process, acquiring motion information of a subject based on the real-time camera image, and performing an exclusion determination process to determine whether or not to exclude the camera image from the anomaly detection process based on the motion information; When an abnormality is detected in the target area, notification information is output to notify the occurrence of the abnormality to a monitor, A monitoring device characterized by determining whether the target area is in a normal state, an abnormal state, or an exclusion state depending on the detection result of the abnormality detection process and the judgment result of the exclusion judgment process, and generating and outputting a monitoring camera image in which a state image indicating whether the target area is in the normal state, the abnormal state, or the exclusion state is combined with the camera image.

2. The processor: The monitoring device according to claim 1, wherein the abnormality detection process detects abnormalities in the target area using a classifier that has been trained in advance using the camera images in an abnormal state and the camera images in a normal state.

3. The processor: The monitoring device according to claim 1, characterized in that the exclusion determination process is performed by using information indicating the direction and magnitude of a motion vector as the motion information to determine whether or not the camera image should be excluded from the anomaly detection process.

4. The processor: The monitoring device described in claim 1, characterized in that if an exclusion state in which a judgment result has been obtained that excludes the camera image from being subject to abnormality detection processing continues for a predetermined time or more, the monitoring device sets the camera image to a continuous exclusion state and outputs the notification information indicating this.

5. The processor: The monitoring device according to claim 4, characterized in that when a determination result is obtained that the camera image does not exclude it from the abnormality detection processing and the continuous exclusion state is lifted, the notification information indicating this is output.

6. A monitoring device that executes an anomaly detection process by a processor to detect an anomaly in a target area based on a camera image captured by a camera, The processor: setting the target area on the camera image based on a user's operation input; performing the anomaly detection process to detect an anomaly in the target area based on the real-time camera image output from the camera; Furthermore, prior to the abnormality detection process, acquiring motion information of a subject based on the real-time camera image, and performing an exclusion determination process to determine whether or not to exclude the camera image from the anomaly detection process based on the motion information; A monitoring device characterized by determining whether the target area is in a normal state, an abnormal state, or an exclusion state depending on the detection result of the abnormality detection process and the judgment result of the exclusion judgment process, and generating and outputting a monitoring camera image in which a state image indicating whether the target area is in the normal state, the abnormal state, or the exclusion state is combined with the camera image.

7. The processor: The monitoring device according to claim 6, wherein the display form of the frame image showing the target area is changed as the status image.

8. A monitoring system comprising a camera that photographs a target area and an information processing device that executes an abnormality detection process that detects an abnormality in the target area based on a camera image photographed by the camera, The information processing device includes: setting the target area on the camera image based on a user's operation input; performing the anomaly detection process to detect an anomaly in the target area based on the real-time camera image output from the camera; Furthermore, prior to the abnormality detection process, acquiring motion information of a subject based on the real-time camera image, and performing an exclusion determination process to determine whether or not to exclude the camera image from the anomaly detection process based on the motion information; When an abnormality is detected in the target area, notification information is output to notify the occurrence of the abnormality to a monitor, A monitoring system characterized by determining whether the target area is in a normal state, an abnormal state, or an exclusion state depending on the detection results of the abnormality detection process and the judgment results of the exclusion judgment process, and generating and outputting a monitoring camera image in which a status image indicating whether the target area is in the normal state, the abnormal state, or the exclusion state is combined with the camera image.

9. A monitoring system comprising a camera that photographs a target area and an information processing device that executes an abnormality detection process that detects an abnormality in the target area based on a camera image photographed by the camera, The information processing device includes: setting the target area on the camera image based on a user's operation input; performing the anomaly detection process to detect an anomaly in the target area based on the real-time camera image output from the camera; Furthermore, prior to the abnormality detection process, acquiring motion information of a subject based on the real-time camera image, and performing an exclusion determination process to determine whether or not to exclude the camera image from the anomaly detection process based on the motion information; A monitoring system characterized by determining whether the target area is in a normal state, an abnormal state, or an exclusion state depending on the detection results of the abnormality detection process and the judgment results of the exclusion judgment process, and generating and outputting a monitoring camera image in which a status image indicating whether the target area is in the normal state, the abnormal state, or the exclusion state is combined with the camera image.

Images