Monitoring system, monitoring method, and monitoring program

Abstract

Depending on the situation, instead of pre-registering a large number of normal images, anomalies are detected by using a suitable image from past images as a reference image. [Solution] The monitoring system of the present invention comprises an imaging unit, a reference image determination unit that uses at least two past images stored in a storage unit, each with a different imaging time on the time axis, to determine which image is appropriate as a normal image of the monitored object, and sequentially updates the reference image based on the determination result, an abnormality determination unit that uses the normal image determined by the reference image determination unit as the reference image, analyzes the difference between the current image and the reference image, and determines whether there is an abnormality on the current screen based on the presence or absence of the difference, an abnormality verification unit that, if an abnormality is determined by the abnormality determination unit, analyzes the difference between the current image and the latest image to verify the abnormality on the current image, and a monitoring unit that outputs the abnormality detection result.

Description

Technical Field

[0001] The present invention relates to a monitoring system, a monitoring method, and a monitoring program, and can be applied to, for example, a monitoring system that monitors whether an abnormal event has occurred on a railway line, a road, or the like.

Background Art

[0002] When an abnormal event such as a landslide, a rockfall, or an avalanche occurs on a railway line, a road, or the like, for example, a train cannot operate safely, and prompt restoration work must be carried out.

[0003] To monitor the occurrence of such abnormal events, there are various technologies, and one of them is, for example, the technology described in Patent Document 1. The technology described in Patent Document 1 compares a certain frame image in the video of the road taken by an imaging device with a reference image of the same shooting area to obtain the similarity of feature amounts, and determines whether the similarity satisfies a predetermined criterion, and determines that it is a frame to be analyzed where an abnormality has occurred.

[0004] As described above, a reference image used as an image in a normal state where no abnormal event has occurred is compared with the current image to determine whether an abnormal event has occurred.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, depending on what kind of image is used as the normal-time image to be compared with the captured image, it may happen that even if it is originally normal, it is misdetected as abnormal.

[0007] For example, differences in time of day, such as morning, noon, and night, can cause the brightness of an image to change due to sunlight, the entire image to take on a reddish tint, or the shape or presence of shadows to be misinterpreted as an anomaly. Furthermore, changes in weather, such as shadows caused by clouds, heavy rain, or snowfall, can also lead to false detections. In addition, the presence of headlights from vehicles passing along the road in the image can also cause false detections.

[0008] While it might be possible to pre-register images of normal operation depending on the situation, obtaining images for all possible situations would be too burdensome and impractical.

[0009] Therefore, in light of the above-mentioned issues, we aim to provide a monitoring system, monitoring method, and monitoring program that, depending on the situation, automatically updates past images from a predetermined time prior to the present as reference images, without pre-registering a large number of normal images, and uses the sequentially updated reference images to detect whether or not an abnormality has occurred in the captured image. [Means for solving the problem]

[0010] To solve these problems, the first monitoring system of the present invention is: A monitoring system that does not register reference images for anomaly detection in advance, but instead refers to images of the monitored target in a normal state from images captured in a time series to monitor for anomalies in the monitored target, (1) an imaging unit that captures images of the monitored object, and (2) the imaging unit In chronological order A memory unit that stores the captured images, and (3) Without registering reference images for anomaly detection in advance, Among the images stored in the memory unit, the one from 1 hour before the current time. It is an image. The first image and the second time period earlier, which is less than the first time period. It is an image. If there is no difference when comparing it with the second image, the second image is determined to be a suitable image representing the normal state. The second image, which was determined to be an image suitable for a normal state, Reference image chase The next reference image determination unit to be updated, and (4) the third time before the current time, which is less than the second time. ImagesThe third image is defined as an abnormality determination unit which determines whether or not there is an abnormality in the third image by performing a difference analysis between the second image and the third image, which are used as reference images, according to the determination result by the reference image determination unit, and (5) if the abnormality determination unit determines that there is an abnormality in the third image, the abnormality verification unit which verifies the abnormality in the third image by performing a difference analysis between the third image and the latest fourth image at the current time.

[0011] The second monitoring method of the present invention is: A monitoring method that monitors for abnormalities in a monitored target by referring to images of the monitored target in a normal state from images captured in a time series, without registering reference images for anomaly detection in advance. (1) The imaging unit captures an image of the object being monitored, and (2) the storage unit is stored by the imaging unit. In chronological order (3) The captured image is stored, and the reference image determination unit determines the image stored in the storage unit that is one time earlier than the current time. It is an image. The first image and the second time period earlier, which is less than the first time period. It is an image. If there is no difference when comparing it with the second image, the second image is determined to be a suitable image representing the normal state. The second image, which was determined to be an image suitable for a normal state, Reference image chase Next update, (4) the abnormality detection unit determines that the time is less than the second time and less than the third time before the current time. Images The third image is determined by the reference image determination unit, and at least by difference analysis between the second image and the third image as reference images, the presence or absence of an abnormality in the third image is determined according to the determination result by the reference image determination unit. (5) If the abnormality verification unit determines that there is an abnormality in the third image, it verifies the abnormality in the third image by difference analysis between the third image and the latest fourth image at the current time.

[0012] The third monitoring program of the present invention is: A monitoring program that does not pre-register reference images for anomaly detection, but instead refers to images of the monitored target in a normal state from images captured in a time series to monitor for anomalies in the monitored target. (1) Without registering reference images for anomaly detection in advance, The monitored items that are remembered chronological Of the images, the one from one hour before the current time. It is an image. The first image and the second time period earlier, which is less than the first time period. It is an image. If there is no difference when comparing it with the second image, the second image is determined to be a suitable image representing the normal state. The second image was determined to be a suitable image for a normal state. Reference image chaseA reference image determination unit that updates next, and (2) before the third time, which is smaller than the second time, from the current time Images As the third image, according to the determination result by the reference image determination unit, at least by differential analysis of the second image and the third image as the reference image, an abnormality determination unit that determines the presence or absence of an abnormality on the third image, and (3) when it is determined by the abnormality determination unit that there is an abnormality on the third image, it functions as an abnormality verification unit that verifies the abnormality on the third image by differential analysis of the third image and the latest fourth image at the current time.

Advantages of the Invention

[0013] According to the present invention, according to the situation, without pre-registering a large number of normal-time images, the past images before a predetermined time from the present are sequentially and automatically updated as reference images, and it is possible to detect whether an abnormality has occurred on the captured image using the sequentially updated reference images.

Brief Description of the Drawings

[0014] [Figure 1] It is a configuration diagram showing the configuration of a monitoring system according to the first embodiment. [Figure 2] It is a configuration diagram showing the configuration of the image analysis process of an information processing device according to the first embodiment. [Figure 3] It is a flowchart showing the process of a monitoring method for detecting an abnormality in a monitoring system according to the first embodiment. [Figure 4] It is an explanatory diagram (part 1) for explaining the abnormality detection process according to the first embodiment. [Figure 5] It is an explanatory diagram (part 2) for explaining the abnormality detection process according to the first embodiment. [Figure 6] It is an explanatory diagram (part 3) for explaining the abnormality detection process according to the first embodiment. [Figure 7] It is a diagram showing an example of an image that is not an abnormality according to the first embodiment (part 1). [Figure 8] It is a diagram showing an example of an image that is not an abnormality according to the first embodiment (part 2). [Figure 9] This figure (1) shows an example of an image used to determine that an abnormality has occurred according to the first embodiment. [Figure 10] This figure (part 2) shows an example of an image used to determine that an abnormality has occurred according to the first embodiment. [Figure 11] This is an explanatory diagram illustrating the anomaly detection process according to the second embodiment. [Modes for carrying out the invention]

[0015] (A) First Embodiment In the following, a first embodiment of the monitoring system, monitoring method, and monitoring program according to the present invention will be described in detail with reference to the drawings.

[0016] (A-1) Configuration of the first embodiment Figure 1 is a configuration diagram showing the configuration of the monitoring system according to the first embodiment.

[0017] In Figure 1, the monitoring system 10 according to the first embodiment includes an information processing device 1, a plurality of monitoring cameras 2 (2-1 to 2-m; m is a positive integer) as imaging units, and a monitoring terminal 3 as a monitoring unit. The information processing device 1, monitoring cameras 2, and monitoring terminal 3 are each connectable to the network NT and can exchange information.

[0018] The monitoring system 10 is a system that places monitoring cameras 2 along railway tracks and analyzes images captured by the monitoring cameras 2 to monitor whether any abnormal situations such as landslides, rockfalls, or avalanches are occurring.

[0019] Traditionally, railway operators have identified areas with weak ground that are prone to landslides and other problems, installed surveillance cameras in those locations, and displayed the images on displays in a remote center for monitoring by personnel. The center maintains a system that allows for 24 / 7 monitoring of images from multiple surveillance cameras. However, problems include overlooking anomalies and delays between anomalies occurring and monitoring personnel recognizing them. Furthermore, even when monitoring personnel view the images, their memory of normal conditions may be vague, making it difficult to determine if a change in the image truly indicates an abnormal event. In cases where a judgment cannot be made based on images alone, workers are dispatched to the site to conduct on-site verification.

[0020] To address these challenges, the monitoring system 10 according to the first embodiment includes a monitoring camera 2 that captures images of railway tracks, and uses multiple images captured at predetermined intervals to determine whether they are suitable as normal images. The system then automatically updates the appropriate images as reference images. Furthermore, the monitoring system 10 compares the reference images with the current images and uses images taken at predetermined intervals to determine whether an abnormality has actually occurred in the image where an abnormality has occurred.

[0021] Surveillance camera 2 captures images of the railway tracks and provides the image data of the surveillance images to the information processing device 1. By providing the image data of the surveillance images to the information processing device 1 in real time, surveillance camera 2 can quickly detect abnormalities.

[0022] Furthermore, by incorporating the image analysis unit 11 of the information processing device 1 into the surveillance camera 2, it is also possible to perform real-time image analysis on the surveillance camera 2 to detect abnormalities in the railway tracks.

[0023] The information processing device 1 acquires image data from the surveillance camera 2, performs image analysis processing, and determines whether or not an anomaly has occurred in the currently monitored image. When an anomaly is detected, it notifies the monitoring terminal 3 accordingly. In addition to the anomaly detection result, the information processing device 1 can also provide past monitoring images acquired from the surveillance camera 2 upon request from the monitoring terminal 3.

[0024] The information processing device 1 can be a server, personal computer, or the like, and is a device equipped with a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc. The CPU can perform its functions as the information processing device 1 by loading various computer programs (for example, monitoring programs, image analysis programs, etc.) stored in the built-in ROM and memory into the RAM and executing them.

[0025] The monitoring terminal 3 is operated by the user and acquires anomaly detection results from the information processing device 1 and monitoring images from the monitoring camera 2. It displays the anomaly detection results and monitoring images on the display unit 31 and allows the user to give instructions through the operation unit 32.

[0026] The monitoring terminal 3 can be a server, personal computer, or the like, and is a device having a CPU, ROM, RAM, display unit 31, operation unit 32, etc. The CPU can perform its function as the monitoring terminal 3 by loading various computer programs (for example, including monitoring programs) stored in the built-in ROM and memory into the RAM and executing them.

[0027] Figure 2 is a configuration diagram showing the image analysis processing configuration of the information processing device 1 according to the first embodiment.

[0028] In Figure 2, the information processing device 1 includes an image analysis unit 11, a storage unit 12, and a communication unit 13.

[0029] The memory unit 12 stores images from the surveillance camera 2, and the communication unit 13 transmits the anomaly detection results from the image analysis unit 11 to the monitoring terminal 3.

[0030] The image analysis unit 11 performs anomaly detection by image analysis using four images that were captured at different times on the time axis. In this embodiment, the use of four images is illustrated, but more images may be used.

[0031] The image analysis unit 11 includes an image acquisition unit 111, a reference image determination unit 112, an anomaly determination unit 113, and an anomaly verification unit 114.

[0032] The image acquisition unit 111 acquires image data of the surveillance image from the surveillance camera 2 and stores it in the storage unit 12.

[0033] The reference image determination unit 112 reads out an image from one time period prior to the current time (first image) and an image from two time periods prior to the current time (second image) from the images stored in the storage unit 12, compares the first image and the second image to determine if there is a difference, and if there is no difference, it determines the second image to be the normal image and uses the second image as the reference image.

[0034] Furthermore, if there is a difference between the first image and the second image, the second image may not be a normal image, so the reference image determination unit 112 will not use the second image as a reference image. In this case, the previous reference image is discarded and no monitoring analysis is performed at this point. Details of the processing of the reference image determination unit 112 will be explained in detail in the operation section.

[0035] The anomaly detection unit 113 reads an image from the memory unit 12 that was three hours prior to the current time (the third image), compares the third image with the second image which is used as a reference image, and determines whether there is a difference. If there is a difference, it determines that an anomaly may have occurred at the time of the third image (i.e., three hours prior to the current time). If there is no difference between the third image and the second image, it determines that no anomaly has occurred.

[0036] When the anomaly detection unit 113 determines that there is a possibility of an anomaly in the third image, the anomaly verification unit 114 reads the latest image (the fourth image) from the current time, compares the third image and the fourth image, and determines whether there is a difference. If there is no difference, the anomaly that occurred in the third image is also present in the latest image, so the anomaly verification unit 114 determines that the anomaly occurred at the time of the third image and notifies the monitoring terminal 3 of the result that an anomaly has occurred via the communication unit 13. If there is a difference, it can be suspected that no anomaly occurred at the time of the third image. Furthermore, if the anomaly verification unit 114 compares the fourth image with a reference image and finds no difference, it can also be suspected that the anomaly in the third image has been resolved and no anomaly has occurred.

[0037] (A-2) Operation of the first embodiment (A-2-1) Monitoring methods for detecting abnormal events Next, the processing operation of the monitoring method for detecting abnormal events on railway tracks using the monitoring system 10 according to the embodiment will be explained with reference to the drawings.

[0038] Figure 3 is a flowchart illustrating the processing of the monitoring method for detecting an anomaly in the monitoring system 10 according to the first embodiment. Figures 4 to 6 are explanatory diagrams illustrating the anomaly detection process according to the first embodiment.

[0039] For example, in order to monitor areas with weak ground near railway tracks where landslides or rockfalls may occur, surveillance camera 2 is installed in a position where it can photograph the area as the target of monitoring, and surveillance camera 2 captures images of the target of monitoring.

[0040] Furthermore, a single surveillance target may be imaged by multiple surveillance cameras 2. Alternatively, if, for example, surveillance camera 2 is capable of panning, tilting, and zooming, and the shooting conditions (shooting angle and magnification) can be arbitrarily changed within its range of motion, multiple surveillance targets may be imaged by a single surveillance camera 2. In any case, the system manages which surveillance camera 2 captures images of which surveillance target, for example, by associating identification information that uniquely identifies surveillance camera 2 with identification information that uniquely identifies the surveillance target.

[0041] [Steps S101, S102] The monitoring process using image analysis shown in Figure 3 represents one cycle of processing flow. The image analysis unit 11 repeats the processing flow exemplified in Figure 3, for example, every 30 seconds.

[0042] The surveillance camera 2 provides the image data of the captured image to the information processing device 1. In the information processing device 1, the image acquisition unit 111 of the image analysis unit 11 acquires the image data of the image from the surveillance camera 2 (step S101), and stores the image data of the captured image in the storage unit 12 (step S102).

[0043] Image data of captured images includes data such as the image itself, location information of the monitored object (e.g., satellite positioning data such as GPS data), time of capture, and file format.

[0044] When image data is stored in the memory unit 12, the identification information of the surveillance camera 2, the image data of the captured image, the identification information that identifies the target of surveillance, and the time of capture are stored in association with each other. Since the surveillance camera 2 captures images continuously or intermittently at predetermined time intervals, the memory unit 12 stores image data files in chronological order for each surveillance camera 2 or for each target of surveillance camera 2.

[0045] [Steps S103, S104, S108] Next, the image analysis unit 11 analyzes the images captured by the surveillance camera 2 to determine whether or not an abnormality has occurred.

[0046] First, the reference image determination unit 112 reads a second image from the images stored in the storage unit 12 that was taken two hours prior to the current time (for example, 60 seconds prior), and determines whether the second image is a suitable image for detecting abnormalities during normal operation.

[0047] Various methods can be applied to determine whether the second image is suitable as a normal image, and one example is described below.

[0048] For example, one method is to compare a second image with an image taken slightly before the second image in a time series. If there is no change in the image, the second image can be considered a suitable representation of a normal state.

[0049] For example, as shown in Figure 4, the reference image determination unit 112 compares a first image taken one time before the current time (for example, 90 seconds ago) (see Figure 4(A)) with a second image taken two time before the current time (for example, 60 seconds ago) (see Figure 4(B)) and calculates the difference. In other words, the first image is an image taken a little earlier than the second image.

[0050] Then, when there is no difference between the first image and the second image, it is determined that the first image is an image that meets the conditions for a normal image (step S103 / meets). In this case, for each cycle, the second image that meets the conditions for a normal image is registered as a reference image for anomaly detection (S104). Since the second image has no difference from the first image from a little while ago, the second image can be determined to be a normal image that has not changed.

[0051] When comparing two images, there are always minor differences, such as the swaying of plants. However, in this embodiment, minor differences may be excluded from the difference between images. For example, various methods can be applied to determine minor differences. One method is to consider a difference as minor when the area of ​​the difference on the image (for example, the number of pixels) is less than a pre-set area threshold. The area threshold in this case can be set to a small value when the estimated distance from the surveillance camera 2 is far, and to a large value when it is close. Thus, the size of the area threshold may be set and changed as appropriate.

[0052] Furthermore, by using a second image from two time periods prior to the present as a reference image, it becomes unnecessary to pre-register a large number of normal images depending on the situation.

[0053] Here, we will explain, in comparison to conventional monitoring methods, why it is not necessary to pre-register a large number of normal images depending on the situation. Generally, in order to analyze the difference from the current image, it is necessary to prepare a large number of normal images that match the conditions and environment at the time of acquisition.

[0054] For example, when performing difference analysis on an image that currently contains a moving vehicle, the image containing the vehicle must be used as a reference image for a normal image. Otherwise, the "moving vehicle," which is not inherently abnormal, will become the difference, leading to a false positive. To avoid this false positive, one method is to pre-register the vehicles themselves and determine whether or not they match through image analysis. However, this would require pre-registering the shape, color, and design of all vehicles operating on each monitored route, which would be a heavy workload and impractical.

[0055] Furthermore, because sunlight and weather conditions vary depending on the time of day (morning, noon, and night), it is necessary to pre-register a large number of images representing normal conditions for each time of day and weather condition. Images can change throughout the day due to factors such as the reddish tint of sunlight in the morning and evening, changes in the shape of shadows of structures and buildings, and changes in brightness. Additionally, changes in weather can alter the image, such as the position of clouds changing due to cloud movement, changes in cloud shadows, and changes in brightness or the presence of rain or snow due to heavy rain or snowfall. In such cases, to avoid false detections, it would be necessary to pre-register images representing normal conditions for each time of day and weather condition, but this is practically difficult due to the sheer number of changing conditions, and similarly, this would be a significant workload.

[0056] Furthermore, for example, in the evening or at night, a car may pass along the railway line being monitored, and although the car itself may not be visible, only the headlights may be captured in the image. In this example, a portion of the captured image will be brightened by the headlights. Also, in order to avoid false detections due to changes in brightness or color in the captured image, such as when an electronic display board at a large store near the monitored area lights up, it would be necessary to register images in all cases as normal images, which would be a heavy workload and practically difficult.

[0057] Furthermore, there are other situations such as when birds like crows perched on overhead wires are captured in the image, when insects are captured in the image, or when raindrops are captured in the image. To avoid false detections, it is necessary to pre-register images of normal conditions.

[0058] Furthermore, while there are methods to train an AI (Artificial Intelligence) system using multiple abnormal images, such as machine learning, it is necessary to obtain images of all abnormal states, which also involves a considerable workload. Conversely, there are methods to train a system using multiple normal images and then classify everything else as abnormal, but similarly, this also requires training the system with all normal images, which also involves a considerable workload. Currently, at the time of filing this application, practical automated monitoring using image analysis has not yet been realized.

[0059] However, by setting a predetermined time (for example, 30 seconds) that is longer than the time it takes for a moving vehicle to pass but shorter than the time it takes for sunlight or weather to change, and using a second image taken a predetermined time ago (a second time ago) from the current time as a reference image, the number of reference images that need to be pre-registered according to the situation can be reduced. In addition, since the reference image is a past image taken a predetermined time ago in the time series, it becomes easier to compare it with the current image, making it easier to find differences and improving the accuracy of anomaly detection.

[0060] Furthermore, in this embodiment, the difference between the first image and the second image, which were captured in a time series, is analyzed, and if there is no difference (or only a small difference), the second image is identified. )child The system determines whether the first image satisfies the conditions for a normal image, and if it does, the second image is considered a normal image.

[0061] Furthermore, instead of simply using a second image from the past as a reference image, the system uses a second image that satisfies the conditions of a normal image as the reference image, thus improving the accuracy of anomaly detection.

[0062] Furthermore, as a variation, the reference image determination unit 112 may take the difference between the first image and the second image, and if the size of the area indicating the difference portion on the second image is less than a preset threshold, or if there is no difference, it may determine that the second image satisfies the conditions for a normal image (i.e., is suitable as a normal image), and if the size of the area indicating the difference portion is greater than or equal to a preset threshold, it may determine that the image does not satisfy the conditions for a normal image (i.e., is unsuitable).

[0063] If the conditions for a normal image are not met in step S103 (step S103 / not suitable), the reference image determination unit 112 discards the second image (step S108) and terminates the process. In other words, in this monitoring process, the reference image is discarded and not registered, the monitoring analysis is terminated, and after a predetermined time (for example, 30 seconds) has elapsed, the process returns to S101 and repeats the next monitoring process.

[0064] In other words, normal images acquired before the current cycle (past normal images) will not be used as reference images for the current cycle. The reason for this is that the latest normal image, which is automatically updated sequentially, is used as the reference image for anomaly detection, and anomaly detection processing is performed by comparing it with the normal image that shows the real-time situation. Therefore, the second image that is determined not to meet the conditions for a normal image in the current cycle will be discarded (of course, previous reference images will also be discarded), and no anomaly detection processing will be performed in the current cycle.

[0065] [Steps S105, S106] Next, the abnormality detection unit 113 reads a third image from the images stored in the memory unit 12 that is from a third time ago (for example, 30 seconds ago) from the current time. Then, the abnormality detection unit 113 compares the second image, which is a normal image, with the third image to determine whether or not there is an abnormality (step S105).

[0066] For example, as shown in Figure 5, the abnormality detection unit 113 compares a second image taken two hours prior to the current time (for example, 60 seconds prior) (see Figure 5(A)) with a third image taken three hours prior to the current time (for example, 30 seconds prior) (see Figure 5(B)) and calculates the difference (step S105).

[0067] Then, if there is no difference between the second image and the third image (step S105 / no abnormality), it is determined that there is no abnormality and the process ends. If there is a difference (step S105 / abnormality found), it is determined that there is an abnormality (see Figure 5(C)) and the process proceeds to step S106.

[0068] In other words, if there is no difference between the third image taken 30 seconds ago and the second image taken 60 seconds ago, it can be said that there has been no change in the subject (object) shown in the image, and therefore it can be determined that there is no abnormality.

[0069] On the other hand, the fact that there is a difference between the third image taken 30 seconds ago and the second image taken 60 seconds ago suggests that there has been a change in the subject (object) shown in the image, which raises the possibility of an anomaly occurring.

[0070] For example, in the case of Figure 5, let's assume there was a rockfall. In that case, although no rocks are visible in the second image, rocks are visible on the railway tracks in the third image. Therefore, it can be determined that a rockfall occurred between 60 seconds and 30 seconds ago, and that this was an anomaly.

[0071] [Step S106] If the anomaly detection unit 113 detects a difference between the second image, which represents a normal image, and the third image, it can be suspected that an anomaly has occurred. However, even if, for example, a crow temporarily perches on an overhead wire, this will be detected as a difference in the image. Therefore, in order to exclude objects that are not inherently abnormal but are only temporarily appearing in the image, the anomaly verification unit 114 verifies whether or not an anomaly exists.

[0072] Various methods can be applied to verify the presence or absence of an anomaly by the anomaly verification unit 114, but one example is described below.

[0073] For example, the anomaly verification unit 114 applies a method of analyzing the difference between the third image, which has been determined to potentially contain an anomaly, and an image taken at a more recent time than the third image.

[0074] In other words, the anomaly verification unit 114 verifies whether or not there is an anomaly based on the third image and the latest fourth image taken at the current time (step S106).

[0075] For example, as shown in Figure 6, the anomaly verification unit 114 compares a third image taken three hours prior to the current time (for example, 30 seconds prior) (see Figure 6(A)) with the current latest fourth image (see Figure 6(B)) and calculates the difference (step S106).

[0076] Then, if there is no difference between the third image and the fourth image, it is determined that the abnormal event detected in step S105 exists in the third image, and an abnormality is detected (step S106 / abnormality found). In other words, the absence of a difference means that the object of the abnormal event is not just something that was temporarily captured in the image, but is also captured in the current latest image, so an abnormality can be detected.

[0077] On the other hand, if there is a difference between the third image and the fourth image, it is determined that there is no abnormal event in the third image detected in step S105, and therefore no abnormality is found (step S106 / no abnormality). In other words, the presence of a difference means that the object in the third image may have been captured only temporarily, so it can be determined that there is no abnormality.

[0078] In this way, the anomaly verification unit 114 verifies the presence or absence of an anomaly, ultimately determining whether or not an anomaly exists, thus improving the accuracy of anomaly detection.

[0079] [Step S107] If it is determined in step S106 that an abnormality exists in the third image, the communication unit 13 notifies the monitoring terminal 3 of the abnormality detection result that an abnormality has occurred (step S107).

[0080] The monitoring terminal 3 displays the monitoring image on the display unit 31. At this time, in order to make it easier for the monitor to recognize the image being monitored, abnormal parts of the image are highlighted, for example, by superimposing a red rectangle on the screen. Note that the highlighting is not limited to a red rectangle.

[0081] Furthermore, when an anomaly is detected, the monitoring terminal 3 outputs an alarm sound and / or illuminates a light to indicate that an anomaly has been detected.

[0082] Furthermore, the system notifies other remote terminals via the NT network of the anomaly, along with the analyzed images.

[0083] Furthermore, the monitoring system 10 can be linked with the operation system, and may be configured to temporarily suspend operations on the route where an abnormality has occurred.

[0084] [Repeat the process] As illustrated in the flowchart in Figure 3, the image analysis unit 11 reads the "first image," "second image," "third image," and "fourth image," and performs image analysis as described above. This image analysis is repeated every 30 seconds.

[0085] Therefore, for example, the "second image" from this analysis will be used as the "first image" in the next analysis 30 seconds later, the "third image" from this analysis will be used as the "second image" in the next analysis 30 seconds later, and the "fourth image" from this analysis will be used as the "third image" in the next analysis 30 seconds later. Then, in the analysis 30 seconds later, a new image will be used as the "fourth image." This process is repeated.

[0086] (A-2-2) Examples of images that are not considered abnormal Figures 7 and 8 show examples of images that do not show abnormalities according to the first embodiment.

[0087] Figure 7(A) is a daytime image. This image can be treated as a second image suitable for a normal state, and if it were treated as a third image, it would not be considered abnormal.

[0088] Figure 7(B) shows an image of a train passing by. In some cases, the train in operation may be visible in the third image. When analyzing the difference between the third image and the reference image used is an image from 30 seconds earlier (for example, Figure 7(A)), which does not show a train, so the difference is large and it is judged that there is a possibility of an anomaly. However, in the fourth image, the train has already passed, so even if a difference occurs between the fourth image and the third image and it is judged that there is a possibility of an anomaly, the subsequent verification process for the presence or absence of an anomaly can determine that there is no anomaly.

[0089] Figure 7(C) is an image of a crow perched on a railway track. The crow may be perched temporarily, or it may be there for a relatively long time. In either case, if the crow is visible in the third image, the reference image is from 30 seconds earlier (for example, Figure 7(A)) and does not show the crow, so a difference occurs and it is judged that there is a possibility of an anomaly. Since the crow will not remain still in the same position and orientation for more than 30 seconds, a difference occurs between the third image and the fourth image. Similar to Figure 7(B), even if the third image is judged to be an anomaly, the subsequent verification process for the presence or absence of an anomaly can determine that there is no anomaly.

[0090] Figure 8(A) shows a nighttime image, and Figure 8(C) shows an image with snow cover. Conventional technology requires the prior preparation of reference images for nighttime and snow cover, separate from the daytime image (Figure 7(A)). However, since there is no significant difference between images within a 30-second interval during nighttime or snow cover, the reference images can be automatically updated sequentially as appropriate for the normal image, eliminating the need for prior registration.

[0091] Figure 8(B) is an image showing the headlights of a car driving along the road at night. As described above, even if the third image is judged to potentially indicate an anomaly, the headlights will not be visible in the same position in the fourth image 30 seconds later, resulting in a difference between the third and fourth images. Therefore, the subsequent verification process for the presence or absence of an anomaly can determine that there is no anomaly.

[0092] (A-2-3) Example of an image that indicates an abnormality Figures 9 and 10 show examples of images used to determine an abnormality according to the first embodiment.

[0093] An example of an image in which the anomaly detection unit 113 determines an anomaly has occurred in the third image is given below.

[0094] Figure 9(A) shows an image of a rockfall on the railway tracks, and Figure 9(B) shows an image of a rockfall beside the tracks. Figure 9(C) shows an image of a landslide, and Figure 10(A) shows an image of a landslide on the slope beside the tracks. In Figures 9(C) and 10(A), there may not be a direct impact on the tracks and railway operations may be able to continue, but it is necessary to deliberately judge them as abnormal occurrences because they represent dangerous changes. Depending on the estimated distance from the tracks, it may be possible to categorize the degree of danger into, for example, large, medium, and small, or to quantify the degree of danger and notify the monitor.

[0095] Figures 9(A) to 9(C) and 10(A) show typical examples of abnormalities in railway tracks, but other types of abnormalities include the following.

[0096] Figure 10(B) shows an image of a part falling from a utility pole next to the railway tracks, and Figure 10(C) shows an image of a car entering the vicinity of the railway tracks.

[0097] If these images are designated as the third image, a clear difference arises from the reference image, so it is determined that there is a possibility of an anomaly. Similarly, in the fourth image, the position remains the same, so it can be determined to be an anomaly during the anomaly verification process.

[0098] (A-3) Variation The image analysis unit 11 may be configured to detect minute differences over time that are recognized as precursors to major abnormal events.

[0099] For example, in the case of an abnormal event such as a landslide on the embankment next to the railway tracks that completely buries the tracks, there may be warning signs such as trees slowly falling or rockfalls occurring over time. Similarly, in the case of an abnormal event such as a river near the railway tracks overflowing due to heavy rain, there may be warning signs such as the river's water level gradually rising over time.

[0100] Various methods can be applied to detect precursors to such abnormal events. For example, if the abnormality determination unit 113 of the image analysis unit 11 performs image analysis using the four images described above and detects an abnormality such as a rockfall (i.e., determines it to be an abnormality that is a precursor to a major abnormal event (an initial stage abnormality)), the image analysis unit 11 may increase the number of images to be analyzed, or, in order to observe changes over time, change the images analyzed from 30-second intervals to 60-second intervals, and determine the changes over time in the difference portion that was determined to be an initial stage abnormality. In other words, after detecting an initial abnormality, the images to be analyzed may be changed, and the image analysis unit 11 may detect changes over time in the difference portion by comparing three images at 60-second intervals. If precursors to a major abnormal event can be detected in this way, it will be possible to suspend railway operations or take measures in the recovery plan.

[0101] (A-4) Effects of the first embodiment As described above, according to the first embodiment, by using past images from a predetermined time prior to the present as reference images, it is possible to detect whether or not an anomaly has occurred in the captured image. Therefore, depending on the situation, anomaly detection is possible without pre-registering a large number of normal images.

[0102] Furthermore, according to the first embodiment, since it is determined whether the second image used as a reference image is suitable as an image of a normal state, the accuracy of anomaly detection can be improved.

[0103] Furthermore, according to the first embodiment, when there is a possibility of an anomaly occurring in the third image, the presence or absence of the anomaly is verified using the latest fourth image, thereby improving the accuracy of anomaly detection.

[0104] (B) Second Embodiment Next, a second embodiment of the monitoring system, monitoring method, and monitoring program according to the present invention will be described in detail with reference to the drawings.

[0105] The configuration of the monitoring system and the image analysis processing of the information processing device in the second embodiment are basically the same as those described in the first embodiment, so the second embodiment will also be explained using Figures 1 and 2.

[0106] In the first embodiment described above, if there is no difference between the first image and the second image, the second image is considered a suitable image as a normal state, i.e., a reference image, and monitoring and analysis continues. If there is a difference, the second image is considered an unsuitable normal state image, and monitoring and analysis is not performed, and the next cycle is waited for. For example, if several crows are present on the railway tracks for a long period of time, there is no reference image available during that time, and analysis cannot be performed. If such a period continues for a long time, it may not be possible to detect anomalies such as rockfalls that occur during that time, which could hinder image analysis for anomaly detection.

[0107] Therefore, the second embodiment distinguishes, on a pixel-by-pixel basis, whether the location of the difference portion is within the area to be analyzed or outside the area to be analyzed. This allows, for example, the analysis of the "railway-side slope" as the area to be analyzed to continue separately from the portion of the crow that is present on the railway tracks while moving for a long time.

[0108] Figure 11 is an explanatory diagram illustrating the anomaly detection process according to the second embodiment.

[0109] In the information processing device 1, similar to the first embodiment, the image acquisition unit 111 of the image analysis unit 11 acquires image data from the surveillance camera 2 and stores the image data of the captured image in the storage unit 12.

[0110] The image analysis unit 11, similar to the first embodiment, performs anomaly detection by image analysis using four images, the first to fourth images, which are captured at different times on the time axis.

[0111] First, the reference image determination unit 112 compares the first image (see Figure 11(A)) with the second image (see Figure 11(B)) and analyzes the difference.

[0112] For example, the image in Figure 11(E) shows the result of difference analysis, with the portion where the difference value is greater than or equal to the first threshold shown in black. The white portion is the part to be analyzed. From Figures 11(A), 11(B), and 11(E), the first image taken 90 seconds before the current time does not show a moving vehicle, while the second image taken 60 seconds before shows a vehicle. By analyzing the difference between these images, the vehicle portion can be obtained as the difference portion, as shown in Figure 11(E).

[0113] The anomaly detection unit 113 compares the third image (see Figure 11(C)) and the fourth image (see Figure 11(D)) and analyzes the difference.

[0114] For example, the image in Figure 11(F) is the result of difference analysis, and the parts where the difference value is greater than or equal to the second threshold are shown in black. The white parts are the parts to be analyzed, and in this example, the entire screen is white. From Figures 11(C), 11(D), and 11(F), the third image taken 30 seconds before the current time and the fourth image taken immediately before it both show rockfalls on the railway tracks. Therefore, there is no difference between these images, and the entire image is white as in Figure 11(F).

[0115] Here, the white areas in Figure 11(E) and Figure 11(F) are defined as the pixels to be monitored.

[0116] Next, the anomaly detection unit 113 compares the second image (see Figure 11(B)) and the third image (see Figure 11(C)) and analyzes the difference. Here, we will explain the case where the brightness of each pixel in the image ranges from 0 to 255, a total of 256 levels.

[0117] The anomaly detection unit 113 then maps each pixel under surveillance with a value obtained by subtracting the brightness B of the second image from the brightness A of the third image, plus an offset value of 128 (if (A-B+128) is 1 or less, the value is set to "1"; if it is 256 or more, the value is set to "255"). The resulting image is shown in Figure 11(G), and pixels other than those under surveillance have a brightness of "0".

[0118] In Figure 11(G), if pixels whose values ​​differ from the offset value of 128 by a certain value (threshold) are set to white as abnormal parts, and all other pixels are set to black, then only the abnormal rockfall area is extracted as a white mass (Figure 11(H)). Thus, even if a train is visible in the second image (reference image), or even if the train is visible in the fourth image at a location far from the rockfall, the train portion is excluded from monitoring, and only the rockfall can be extracted.

[0119] The anomaly verification unit 114 derives the actual size of the anomaly portion shown in the image, and determines that it is an anomaly if the actual size of the anomaly portion is greater than or equal to a pre-set anomaly determination threshold. Conversely, if the actual size of the anomaly portion is less than the anomaly determination threshold, the anomaly verification unit 114 determines that it is not an anomaly.

[0120] In other words, by performing a difference analysis between images taken at different time points on the time axis, an anomaly is determined to exist when the detected anomaly is larger than a certain size.

[0121] Here, various methods can be applied to derive the actual size of the abnormal portion that appears in the image. For example, the abnormality verification unit 114 can use a method that derives the actual size of the abnormal portion based on the area (e.g., number of pixels) of the abnormal portion detected in the image and the estimated distance from the surveillance camera 2 to the actual object related to the abnormal portion.

[0122] As mentioned above, the anomaly verification unit 114 determined that the actual size of the object related to the anomaly was large. However, it may also determine that the anomaly is an anomaly if the size of the anomaly on the image (for example, the number of pixels) is above a certain size, rather than the actual size.

[0123] On the monitoring terminal 3, the rockfall area is highlighted as an abnormal area with a rectangle and displayed on the display unit 31, as shown in Figure 11(I).

[0124] (C) Other embodiments Although various modified embodiments were mentioned in the first and second embodiments described above, the present invention can also be applied to the following modified embodiments.

[0125] (C-1) Although the present invention has been applied to a railway track monitoring system, it is not limited to railway track monitoring systems, but can also be applied to monitoring systems for traffic infrastructure such as roads, and other systems as exemplified below.

[0126] (C-1-1) Applicable to monitoring systems that monitor changes such as defects / wear and tear in factory equipment. Here, it is assumed that checks should be performed on a daily basis.

[0127] For example, the capture time for the second and third images is set to noon every day. To confirm whether the image taken at noon on a given day (the second image) is suitable as a "normal image," for example, it is checked to see if there is no difference between the first and second images. If there is a difference, capture is continued every 30 seconds, and the second image at the point when there is no difference in consecutive images is registered as the "normal image." The same process is repeated at noon the next day, and this image is designated as the third image. The third image is confirmed to be valid if there is no difference between the third image and the fourth image taken 30 seconds later than the third image. Then, the difference between the third image and the previous day's "normal image = second image" is checked, and if there is a difference, it is considered an abnormality (missing / worn parts, etc.). After that, the third image is registered as the "normal image," and this process is repeated every day thereafter.

[0128] (C-1-2) This can be applied to a monitoring system that monitors changes over time, such as the tilt of building columns. Here, it is assumed that the check should be performed every month.

[0129] For example, the capture times for the second and third images are set to noon on the 1st of each month. To verify whether the image taken at noon on the 1st of a given month is suitable as the "normal image," for example, check that there is no difference between the second image and the first image taken 30 seconds earlier. If there is a difference, continue taking images every 30 seconds and register the image at the point when there is no difference in the consecutive images as the "normal image."

[0130] A similar image is taken at noon on the 1st of the following month and designated as the third image. After confirming that there is no difference between the third image and the fourth image taken 30 seconds later, the third image is considered the valid image. Then, the difference between the third image and the "normal image" from the previous month is checked, and any difference is identified as an anomaly (such as a tilted pillar). Subsequently, the third image is registered as the "normal image," and this process is repeated every month thereafter.

[0131] (C-1-3) Applicable to monitoring systems for monitoring the shutdown of manufacturing equipment, etc. It recognizes a shutdown when continuous operation is the normal state. Here, the second and third images are captured every 30 seconds. This is because we want to indicate an abnormality if the equipment remains stationary for 30 seconds or more.

[0132] For example, to confirm whether an captured image is suitable as a "normal image," one might check for a difference between the second image and the first image taken 30 seconds earlier. Since manufacturing equipment is generally in constant operation, if there is no difference, it is assumed that the equipment has not started operating. In this case, images are taken every 30 seconds, and the image at the point when a difference occurs in the consecutive images is registered as the "normal image."

[0133] A third image is taken after 30 seconds. The third image is considered valid if there is no difference between the third image and the fourth image, which is taken 30 seconds later than the third image. The difference between the third image and the "normal image (second image)" is checked, and if there is no difference, it is considered abnormal (stopped). After that, the third image is registered as the "normal image," and the process is repeated every 30 seconds thereafter.

[0134] (C-2) In the first and second embodiments and the modified embodiments described above, examples were given of anomaly detection using four images taken at 30-second intervals, but the interval is not limited to 30 seconds.

[0135] Here, we will explain the rationale for the time shift of n seconds (for example, 30 seconds) in the captured images.

[0136] For example, when monitoring railway tracks, it is necessary to consider the time it takes for a train to pass. In this case, although it depends on factors such as the route schedule, the length of the train, and the speed of the train, it should always be assumed that the time it takes for a train to pass a certain point is less than n seconds. In other words, among the images taken at n-second intervals on the time axis, the train will not be visible in two adjacent images. Therefore, it is meaningful to use images taken every n seconds.

[0137] Furthermore, animals such as crows do not remain still in the same position for n seconds. Therefore, it is meaningful to use images captured every n seconds.

[0138] Furthermore, for example, after a landslide or rockfall occurs (at least some time later), the condition remains unchanged for n seconds.

[0139] Furthermore, for example, let's assume that n seconds is a period of time during which there are no significant changes in brightness due to sunlight or weather. [Explanation of symbols]

[0140] 1: Information processing device, 2: Surveillance camera, 3: Surveillance terminal, 10: Surveillance system, 11: Image analysis unit, 12: Storage unit, 13: Communication unit, 31: Display unit, 32: Operation unit, 111: Image acquisition unit, 112: Reference image determination unit, 113: Anomaly determination unit, 114: Anomaly verification unit.

Claims

1. A monitoring system that monitors abnormalities in a monitored target by referring to a normal image of the monitored target from images captured in a time series, without registering a reference image for abnormality detection in advance, An imaging unit that captures an image of the subject being monitored, A storage unit that stores images captured in chronological order by the imaging unit, A reference image determination unit compares a first image, which is an image taken a first time before the current time, with a second image, which is an image taken a second time before the current time, which is less than the first time before the current time, and if there is no difference, the second image is determined to be a suitable image for normal conditions, and the second image determined to be a suitable image for normal conditions is sequentially updated as a reference image. The third image is an image taken three time periods earlier than the current time, which is less than two time periods earlier. The anomaly determination unit determines whether or not there is an anomaly in the third image by performing a difference analysis between the second image and the third image, which are used as reference images, according to the determination result by the reference image determination unit. When the anomaly detection unit determines that there is an anomaly in the third image, the anomaly verification unit verifies the anomaly in the third image by performing a difference analysis between the third image and the latest fourth image at the current time. A monitoring system characterized by comprising the following features.

2. If the reference image determination unit determines that the second image is a suitable image for the normal state, When the abnormality detection unit finds a difference between the third image and the reference image, it determines that there is an abnormality in the third image. The abnormality verification unit determines that there is an abnormality in the third image if there is no difference between the third image and the fourth image. The monitoring system according to claim 1.

3. If the reference image determination unit determines that the second image is not suitable as the normal image, The abnormality detection unit determines that there is an abnormality in the third image when there is no difference between the third image and the fourth image, and there is a difference between the second image and the third image. The abnormality verification unit determines that there is an abnormality if the size of the abnormal portion on the third image is greater than or equal to a preset threshold, and that there is no abnormality if it is less than the preset threshold. The monitoring system according to claim 1.

4. The aforementioned reference image determination unit, If the size of the region showing the difference between the first image and the second image is less than a pre-set threshold, or if there is no difference, it is determined to be suitable as the normal image. If the size of the region representing the difference exceeds a pre-set threshold, it is determined that the image is not suitable as a normal image. The monitoring system according to claim 1 or 2, characterized in that it is the same as described in claim 1 or 2.

5. A monitoring system that monitors abnormalities in a monitored target by referring to a normal image of the monitored target from images captured in a time series, without registering a reference image for abnormality detection in advance, The imaging unit captures an image of the monitored object, The memory unit stores images captured in chronological order by the imaging unit. The reference image determination unit compares the first image, which is an image taken a first time before the current time, with the second image, which is an image taken a second time before the current time, among the images stored in the storage unit. If there is no difference, the unit determines that the second image is a suitable image for normal conditions, and sequentially updates the second image, which has been determined to be a suitable image for normal conditions, as the reference image. The anomaly detection unit designates an image taken a third time earlier than the current time, which is less than a second time interval, as the third image, and, according to the determination result by the reference image determination unit, determines whether or not there is an anomaly in the third image by at least a difference analysis between the second image and the third image, which are used as reference images. When the anomaly verification unit determines that there is an anomaly in the third image by the anomaly determination unit, it verifies the anomaly in the third image by performing a difference analysis between the third image and the latest fourth image at the current time. A monitoring method characterized by the following features.

6. A monitoring program that monitors for abnormalities in a monitored target by referring to a normal image of the monitored target from images captured in a time series, without registering a reference image for abnormality detection in advance, Computers, A reference image determination unit compares a first image, which is an image taken a first time before the current time, with a second image, which is an image taken a second time before the first time, from among the time-series images of the monitored object stored in memory. If there is no difference, the unit determines that the second image is a suitable image for normal conditions and sequentially updates the second image, which has been determined to be a suitable image for normal conditions, as the reference image. The third image is an image taken three time periods earlier than the current time, which is less than two time periods earlier. The anomaly determination unit determines whether or not there is an anomaly in the third image by performing a difference analysis between the second image and the third image, which are used as reference images, according to the determination result by the reference image determination unit. When the anomaly detection unit determines that there is an anomaly in the third image, the anomaly verification unit verifies the anomaly in the third image by performing a difference analysis between the third image and the latest fourth image at the current time. A monitoring program characterized by its ability to function in this way.

Images