Monitoring system and monitoring method

Abstract

To enable image analysis processing using a learning model to be executed by enabling a manager to appropriately switch an operation state of a system.SOLUTION: Any operation mode is set among a first operation mode for collecting photographed images of a monitoring area as teacher images and generating a learning model by using the teacher images, a second operation mode for detecting an event that occurs in the monitoring area to perform an alarm by performing image analysis processing using the learning model generated in the first operation mode to the photographed images, and a third operation mode for detecting the event by the image analysis processing using the learning model to perform an alarm, also collecting the photographed images as the teacher images to perform additional learning for generating a learning model by using the teacher images, and updating the learning model to be used in the image analysis processing by a learning model generated by the additional learning, and processing regulated by the operation mode is executed.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a monitoring device and a monitoring method for detecting various events (such as abnormal events) occurring in a monitoring area and controlling reporting by performing image analysis processing on a captured image of the monitoring area using a learning model.

Background Art

[0002] Systems for detecting abnormal events based on captured images of a monitoring area taken by a camera are widely spread. Furthermore, in recent years, systems for detecting abnormal events using a learning model generated by machine learning such as deep learning are also being used.

[0003] In such a system for detecting abnormal events using a learning model, the accuracy may decrease due to environmental changes in the monitoring area. In particular, when abnormal detection is performed on an outdoor monitoring area for applications such as disaster prevention, compared to when an indoor monitoring area is targeted, the environmental changes are large, so the accuracy of abnormal detection decreases and false detections often occur.

[0004] Therefore, in a system for detecting abnormal events using a learning model, a technique for avoiding a decrease in accuracy due to environmental changes is desired. Regarding such a demand, conventionally, in addition to a camera that captures the monitoring area, a sensor for acquiring information on the current situation of the monitoring area is provided, and based on the detection result of the sensor, a technique for switching to a learning model suitable for the current situation of the monitoring area is known (see Patent Document 1).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, simply switching the learning model according to the current state of the monitoring area, as in conventional technologies, has limitations in improving the accuracy of image analysis processing using the learning model. Therefore, it is conceivable to combine it with other methods to improve the accuracy of image analysis processing using the learning model.

[0007] On the other hand, in systems that detect abnormal events using a learning model, a new learning model is created by collecting images of the monitoring area as training images and performing initial training before operation. Furthermore, the accuracy of the learning model can be improved by performing additional training using images of the monitoring area as training images during operation. For this reason, it is desirable to be able to perform additional training to improve the accuracy of the learning model as needed when the accuracy of the learning model is insufficient.

[0008] In a system that uses a learning model to detect anomalies, there are several possible operating states: a preliminary operation state (preparation state) where initial training is performed to collect training images and create a new learning model; a normal operation state where image analysis processing is performed using the learning model; and an operation state where updates are carried out while continuing the normal operation state and additional training is performed to improve the accuracy of the learning model. Therefore, it is desirable for administrators to be able to switch between these system operating states as needed, but conventional technologies have not taken this requirement into consideration at all.

[0009] Therefore, the main objective of the present invention is to provide a monitoring device and monitoring method that enable administrators to appropriately switch the operating state of the system, thereby enabling image analysis processing using a learning model to be performed with high accuracy. [Means for solving the problem]

[0010] This invention Monitoring system teeth, The system comprises: a server device that controls multiple cameras installed in a monitoring area, the collection of training images necessary for generating a learning model used for image analysis processing of images captured in the monitoring area set for each camera, and the notification of events detected in the monitoring area by performing image analysis processing on the captured images using the learning model; and a terminal that displays an operation screen including a notification screen regarding the notification of the event and a setting screen for setting the processing conditions for each camera performed by the server device, wherein the server device controls the user's operation in setting the processing conditions for each camera on the setting screen, The aforementioned captured images were collected as training images. death,A first operating mode generates the learning model using the training images; a second operating mode detects the event and issues an alert using the image analysis processing that utilizes the learning model generated in the first operating mode; and a third operating mode detects the event and issues an alert using the image analysis processing that utilizes the learning model, and collects the captured images as training images. death, One of the following operating modes: a third operating mode in which additional learning is performed to generate the learning model using the training images, and the learning model used in the image analysis process is updated using the learning model generated by the additional learning; Set this for each camera, The system is configured to execute the specified process in that operating mode.

[0011] Furthermore, the monitoring method of the present invention is A server device controls the following processes: collecting training images necessary for generating a learning model used for image analysis processing of images captured by multiple cameras installed in the monitoring area, and issuing reports of events detected in the monitoring area by performing image analysis processing on the captured images using the learning model. A monitoring method performed by, An operation screen is displayed on the terminal, which includes a notification screen regarding the notification of the aforementioned event and a settings screen for setting the processing conditions for each camera performed by the server device, and based on the user's operation in setting the processing conditions for each camera on the settings screen, The aforementioned captured images were collected as training images. death, A first operating mode generates the learning model using the training images; a second operating mode detects the event and issues an alert using the image analysis processing that utilizes the learning model generated in the first operating mode; and a third operating mode detects the event and issues an alert using the image analysis processing that utilizes the learning model, and collects the captured images as training images. death, One of the following operating modes: a third operating mode in which additional learning is performed to generate the learning model using the training images, and the learning model used in the image analysis process is updated using the learning model generated by the additional learning; Set this for each camera, The system is configured to execute the specified process in that operating mode. [Effects of the Invention]

[0012] According to the present invention, in response to instructions and operations from the administrator, Processing conditions for images captured by multiple cameras 1 、 The system is set to one of the second or third operating modes, and the processing defined for that operating mode is executed. This allows the administrator to monitor the system's operating status. Each camera By enabling switching between modes as needed, image analysis processing using the learned model can be performed with high accuracy.

Brief Description of the Drawings

[0013] [Figure 1] Overall configuration diagram of the monitoring system according to this embodiment [Figure 2] Explanatory diagram showing a photographed image of a monitoring area by a camera [Figure 3] Explanatory diagram showing a detection area and a mask area set on a photographed image of a camera [Figure 4] Explanatory diagram showing the setting status of three operation modes [Figure 5] Explanatory diagram showing the switching status of a learning model in the first model switching mode [Figure 6] Explanatory diagram showing the switching status of a learning model in the third model switching mode [Figure 7] Block diagram showing an overview of the processing performed in the first operation mode [Figure 8] Block diagram showing an overview of the processing performed in the second operation mode [Figure 9] Block diagram showing an overview of the processing performed in the third operation mode [Figure 10] Block diagram showing a schematic configuration of an image analysis server, an image management server, and a monitoring terminal [Figure 11] Flow chart showing the procedure of the processing performed in the image management server [Figure 12] Flow chart showing the procedure of the processing performed in the image analysis server in the first model switching mode [Figure 13] Flow chart showing the procedure of the processing performed in the image analysis server in the second model switching mode [Figure 14] Flow chart showing the procedure of the processing performed in the image analysis server in the third model switching mode [Figure 15] Explanatory diagram showing a monitoring screen before setting displayed on the monitoring terminal [Figure 16] Explanatory diagram showing a processing target setting screen displayed on the monitoring terminal [Figure 17] Explanatory diagram showing a camera setting screen displayed on the monitoring terminal [Figure 18] Diagram illustrating the processing condition setting screen displayed on the monitoring terminal. [Figure 19] Diagram illustrating the period switching setting screen displayed on the monitoring terminal. [Figure 20] Diagram illustrating the configured monitoring screen displayed on the monitoring terminal. [Figure 21] Diagram illustrating the annotation screen displayed on the monitoring terminal. [Figure 22] Diagram illustrating the monitoring screen displayed on the monitoring terminal during abnormal situations. [Figure 23] Diagram illustrating the notification screen displayed on the monitoring terminal. [Figure 24] Diagram illustrating the status confirmation screen displayed on the monitoring terminal. [Figure 25] Diagram illustrating the rollback settings screen displayed on the monitoring terminal. [Modes for carrying out the invention]

[0014] The first invention made to solve the aforementioned problem is: The system comprises: a server device that controls multiple cameras installed in a monitoring area, the collection of training images necessary for generating a learning model used for image analysis processing of images captured in the monitoring area set for each camera, and the notification of events detected in the monitoring area by performing image analysis processing on the captured images using the learning model; and a terminal that displays an operation screen including a notification screen regarding the notification of the event and a setting screen for setting the processing conditions for each camera performed by the server device, wherein the server device controls the user's operation in setting the processing conditions for each camera on the setting screen, The aforementioned captured images were collected as training images. death, A first operating mode generates the learning model using the training images; a second operating mode detects the event and issues an alert using the image analysis processing that utilizes the learning model generated in the first operating mode; and a third operating mode detects the event and issues an alert using the image analysis processing that utilizes the learning model, and collects the captured images as training images. death, One of the following operating modes: a third operating mode in which additional learning is performed to generate the learning model using the training images, and the learning model used in the image analysis process is updated using the learning model generated by the additional learning; Set this for each camera, The system is configured to execute the specified process in that operating mode.

[0015] According to this, in response to instructions and actions from the administrator, Processing conditions for images captured by multiple cameras 1 、 The system is set to one of the second or third operating modes, and the processing defined for that operating mode is executed. This allows the administrator to monitor the system's operating status. Each camera By enabling switching between modes as needed, image analysis processing using the learned model can be performed with high accuracy.

[0018] Also, Second The invention is, The server device The system generates the learning models for each of the multiple periods using the training images that have been divided into multiple periods, and the learning model for the period corresponding to the day of monitoring Switch to The system is configured to perform the aforementioned image analysis processing.

[0019] According to this, by selecting a learning model corresponding to the period (such as season) on the day of monitoring, image analysis processing is performed using a learning model that is suitable for the current conditions of the monitoring area, thereby improving the accuracy of the image analysis processing.

[0020] Also, The third The invention is, The server device This involves generating the learning model for each of the multiple weather categories using the training images that have been sorted into multiple weather categories, and then generating the learning model for the weather category corresponding to the day of monitoring. Switch to The system is configured to perform the aforementioned image analysis processing.

[0021] According to this, by selecting a learning model corresponding to the weather category on the day of monitoring, image analysis processing is performed using a learning model that is suitable for the current conditions of the monitoring area, thereby improving the accuracy of the image analysis processing.

[0022] Also, The fourth The invention is, The server device The system is configured to switch the learning model used in the image analysis process based on the likelihood included in the results of the image analysis process.

[0023] According to this, during transitional periods (such as seasonal changes), image analysis processing can be performed using a learning model suited to the current conditions of the monitoring area, thereby improving the accuracy of image analysis.

[0024] Also, Fifth The invention is, The server device teeth, The aforementioned Acquired by changing the camera's shooting conditions. The aforementioned Using images taken for each monitoring area The aforementioned The system will be configured to detect events and issue alerts for each monitoring area.

[0025] According to this, a single camera can monitor multiple surveillance areas. In this case, for example, by changing the shooting conditions (shooting angle, magnification, etc.) using PTZ (pan, tilt, zoom) functions, it is possible to acquire images for each of the multiple surveillance areas.

[0026] Also, The sixth The invention is, The server device This is used in the aforementioned image analysis processing. If the aforementioned learning model has been updated, the user can perform an operation to change the currently running learning model from the aforementioned learning model. The system is configured to perform a rollback process to revert to the previous learning model.

[0027] According to this, if the currently running learning model has many defects, a rollback process can be performed to reduce the number of defects.

[0028] Also, 7th The invention is, A server device controls the following processes: collecting training images necessary for generating a learning model used for image analysis processing of images captured by multiple cameras installed in the monitoring area, and issuing reports of events detected in the monitoring area by performing image analysis processing on the captured images using the learning model. A monitoring method performed by, An operation screen is displayed on the terminal, which includes a notification screen regarding the notification of the aforementioned event and a settings screen for setting the processing conditions for each camera performed by the server device, and based on the user's operation in setting the processing conditions for each camera on the settings screen, The aforementioned captured images were collected as training images. death, A first operating mode generates the learning model using the training images; a second operating mode detects the event and issues an alert using the image analysis processing that utilizes the learning model generated in the first operating mode; and a third operating mode detects the event and issues an alert using the image analysis processing that utilizes the learning model, and collects the captured images as training images. death, One of the following operating modes: a third operating mode in which additional learning is performed to generate the learning model using the training images, and the learning model used in the image analysis process is updated using the learning model generated by the additional learning; Set this for each camera, The system is configured to execute the specified process in that operating mode.

[0029] According to this, similar to the first invention, the system's operating status can be monitored by an administrator. Each camera By enabling switching between modes as needed, image analysis processing using the learned model can be performed with high accuracy.

[0030] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0031] Figure 1 is an overall diagram of the monitoring system according to this embodiment.

[0032] This system detects and notifies of abnormal events occurring in a monitored area based on images captured within that area. The system comprises a camera 1, an image analysis server 2 (monitoring device), an image management server 3 (monitoring device), and a monitoring terminal 4 (terminal device). Camera 1, image analysis server 2, image management server 3, and monitoring terminal 4 are connected via a network such as the internet or a private network.

[0033] Camera 1 captures images of the monitoring area and transmits the captured images in real time to image analysis server 2 and image management server 3.

[0034] Image analysis server 2 uses a learning model generated by machine learning, such as deep learning, to perform image analysis on real-time images captured by camera 1. It detects abnormal events occurring within the monitoring area and notifies users (monitoring personnel) of these abnormal events using monitoring terminal 4. Image analysis server 2 also generates a learning model using training images collected by image management server 3.

[0035] Image management server 3 receives and stores captured images periodically transmitted from camera 1. Image management server 3 also collects and manages training images from the captured images stored in its device, which are used for generating learning models performed by image analysis server 2.

[0036] Monitoring terminal 4 displays a monitoring screen. The monitoring screen displays real-time images captured by camera 1. This allows the user (monitoring officer) to check the current status of the monitoring area. Monitoring terminal 4 also displays a notification screen. This allows the user to quickly recognize if an abnormal event has occurred in the monitoring area. Furthermore, monitoring terminal 4 allows the user (administrator) to perform operations related to setting conditions for processing performed by image analysis server 2 and image management server 3.

[0037] While the image analysis server 2 and image management server 3 can be configured to operate on-premises, i.e., near the monitoring area, the image management server 3 and image analysis server 2 may also be operated in the cloud.

[0038] Furthermore, in this embodiment, various processes are performed in the image analysis server 2 and the image management server 3, but these processes may be performed on a single server. Also, the processes performed in the image analysis server 2 and the image management server 3 may be shared among multiple servers in a different combination than that of this embodiment.

[0039] In this embodiment, management operations are performed on the monitoring terminal 4, but a separate management terminal may be provided, and management operations may be performed on that management terminal.

[0040] Next, we will explain the monitoring area that is subject to anomaly detection processing. Figure 2 is an explanatory diagram showing the image of the monitoring area captured by camera 1.

[0041] Camera 1 is a PTZ (pan, tilt, zoom) camera capable of pan, tilt, and zoom operation, allowing the shooting conditions (shooting angle and magnification) to be arbitrarily changed within its movable range. This enables one camera 1 to capture different areas. Therefore, by periodically switching the shooting conditions of camera 1, it is possible to detect abnormal events in multiple monitoring areas with a single camera 1.

[0042] In this embodiment, the shooting conditions of camera 1 are set as a preset position to capture two monitoring areas, a first and a second, and the shooting conditions of camera 1 are controlled to switch at predetermined time intervals (for example, every 30 minutes).

[0043] Here, for example, the first monitoring area is the path of the moving object, and the second monitoring area is the area surrounding the path. In this example, as shown in Figure 2(A-1), the first monitoring area is the road, and as shown in Figure 2(B-1), the second monitoring area is the slope beside the road. The area surrounding the path that becomes the second monitoring area may be other than the slope beside the road, such as the outside of the road shoulder (sidewalk), the slope above the entrance and exit of a tunnel, ventilation equipment and lighting installed inside the tunnel, or a pedestrian bridge built over the road.

[0044] Incidentally, in monitoring for rainfall disaster prevention measures, for example, multiple areas where abnormal events occur are interconnected may be the target of monitoring. In this case, an abnormal event occurring in one monitoring area may cause an abnormal event to occur in another monitoring area. In this example, as shown in Figure 2(B-2), if a landslide occurs on the slope beside the road, which is the second monitoring area, the collapsed soil will accumulate on the road, which is the first monitoring area, as shown in Figure 2(A-2), obstructing vehicle traffic.

[0045] In this example, the moving object is a car and the travel path is a road, but the moving object and travel path are not limited to these. For example, the moving object could be a train and the travel path could be a railway track.

[0046] Furthermore, in this embodiment, image analysis processing using a learning model is performed on images captured in the monitoring area to detect abnormal events occurring in the monitoring area and trigger an alert. However, this is not limited to abnormal events; other events (events of interest) may also be detected.

[0047] When anomaly detection is performed in outdoor monitoring areas for disaster prevention and other purposes, the accuracy of anomaly detection decreases and false positives occur more frequently compared to indoor monitoring areas due to the greater environmental changes. For example, the condition of the captured images in the monitoring area changes significantly due to the effects of fluctuations in sunlight (changes in shadows, etc.) and weather (rainfall, snowfall, etc.), which reduces the accuracy of anomaly detection. Therefore, it is desirable to ensure the accuracy of anomaly detection and reduce false positives as much as possible.

[0048] Next, we will explain the detection area and mask area set on the image captured by camera 1. Figure 3 is an explanatory diagram showing the detection area and mask area set on the image captured by camera 1.

[0049] In this embodiment, a detection area subject to anomaly detection processing and a mask area excluded from anomaly detection processing are set on the image captured by camera 1. Note that if the entire captured image is to be subject to anomaly detection processing, the setting of the detection area and mask area can be omitted. Alternatively, only the detection area may be set, and the setting of the mask area may be omitted.

[0050] Specifically, as shown in Figure 3(A), in the image captured for the first monitoring area targeting the path of a moving object, the path (road) is set as the detection area, and the area surrounding the path (slope beside the road) is set as the mask area. On the other hand, as shown in Figure 3(B), in the image captured for the second monitoring area targeting the area surrounding the path, the area surrounding the path is set as the detection area, and the path itself is set as the mask area.

[0051] Next, we will explain the three operating modes that can be set in this system. Figure 4 is an explanatory diagram showing the settings for the three operating modes.

[0052] In this system, the system is sequentially set to one of three operating modes—the first, second, or third—depending on the user's (administrator's) actions.

[0053] The first operating mode is a provisional operation state (preparation state). The first operating mode is set when camera 1 is newly installed at the monitoring point. In the first operating mode, the system collects training images from the images captured by camera 1 and generates a new learning model using these training images (initial learning). Once the learning model is generated in the first operating mode, the user performs an operation to switch to the second operating mode.

[0054] The second operating mode is the normal operating state. In the second operating mode, image analysis is performed on the images captured by camera 1 using the learning model generated in the first operating mode to detect abnormal events occurring within the monitoring area. If sufficient accuracy cannot be obtained when operating in the second operating mode, the user can switch to the third operating mode.

[0055] The third operating mode aims to improve accuracy while continuing normal operation. In the third operating mode, similar to the second operating mode, image analysis using a learned model is performed on images captured by camera 1 to detect abnormal events within the monitoring area. However, in addition, a process is performed to collect new training images from the images captured by camera 1 and to generate a learned model using these training images (additional learning). Additional learning is performed at appropriate intervals, and the learned model used in image analysis is updated with the learned model generated by additional learning. Additional learning and updating of the learned model may be performed, for example, on a daily or weekly basis. If sufficient accuracy is obtained in the third operating mode, the user can perform an operation to return to the second operating mode.

[0056] Next, we will explain the three model switching modes that can be set in this system. Figure 5 is an explanatory diagram showing the switching status of the learning model in the first model switching mode. Figure 6 is an explanatory diagram showing the switching status of the learning model in the third model switching mode.

[0057] In this system, depending on which monitoring area camera 1 is currently capturing images of, the learning model used for image analysis processing for anomaly detection is switched to the learning model corresponding to the monitoring area (learning model switching process). Specifically, if the first monitoring area is currently being captured, the learning model for the first monitoring area is adopted, and if the second monitoring area is currently being captured, the learning model for the second monitoring area is adopted.

[0058] Furthermore, this system is configured to one of three model switching modes (1st, 2nd, or 3rd) depending on the user's (administrator's) specified operation. The image analysis server 2 performs a process to switch the learning model used in the image analysis process for anomaly detection based on the configured model switching mode (learning model switching process).

[0059] As shown in Figure 5, in the first model switching mode, a process is performed to switch the learning model for each period (season) based on the date (environmental conditions) of the day (the day the image analysis is performed) (period switching process). Specifically, a learning model for the first period (spring and summer) and a learning model for the second period (autumn and winter) are prepared. If the day falls within the first period (spring and summer), the learning model for the first period is selected, and if the day falls within the second period (autumn and winter), the learning model for the second period is selected. This allows the learning model to be switched in response to changes in the captured image due to the effects of changes in sunlight based on seasonal changes (such as changes in shadow length), thereby reducing false detections.

[0060] Furthermore, the first and second periods may be set based on seasonal reference dates (such as the vernal equinox, summer solstice, autumnal equinox, and winter solstice). Alternatively, the first and second periods may be set based on the altitude and direction of the sun.

[0061] The second model switching mode is set during period transitions (seasonal changes). In the second model switching mode, similar to the first model switching mode, a process (period switching process) is performed to switch the learning model for each period (season) based on the current date. However, if the image analysis results are not sufficiently reliable based on the likelihood of the image analysis results, the image analysis results from a different learning model are adopted (transition period likelihood switching process). During period transitions, the current situation of the monitoring area may be significantly unsuitable for a learning model set uniformly for the period. By switching the learning model based on the likelihood of the image analysis results, false positives can be reduced.

[0062] Specifically, during the transition period from the first period (spring / summer) to the second period (autumn / winter) (late summer and early autumn), or from the second period (autumn / winter) to the first period (spring / summer) (late winter and early spring), if the likelihood of the uniformly set learning model for each period falls below the threshold and the image analysis results are not sufficiently reliable, then the image analysis results using a different learning model will be adopted. This means that if the date falls within the first period (spring / summer), but the current situation of the monitoring area is suitable for the learning model of the second period (autumn / winter), then the image analysis results using the learning model of the second period will be adopted. Similarly, if the date falls within the second period (autumn / winter), but the current situation of the monitoring area is suitable for the learning model of the first period (spring / summer), then the image analysis results using the learning model of the first period will be adopted. This reduces false positives.

[0063] The second model switching mode is implemented temporarily during transitional periods (seasonal changes), but the duration of the second model switching mode may be predetermined by the administrator, or the administrator may decide to switch to the second model switching mode as appropriate.

[0064] As shown in Figure 6, in the third model switching mode, a process is performed to switch the learning model for each weather category based on the current weather (environmental conditions) of the monitoring area (weather switching process). Specifically, a learning model for the first weather category (sunny / cloudy), a learning model for the second weather category (rainfall), and a learning model for the third weather category (snowfall) are provided. If the current weather in the monitoring area corresponds to the first weather category, the learning model for the first weather category is selected. If the current weather in the monitoring area corresponds to the second weather category, the learning model for the second weather category is selected. If the current weather in the monitoring area corresponds to the third weather category, the learning model for the third weather category is selected. This allows the learning model to be switched in response to changes in the captured image due to the effects of rainfall or snowfall, thereby reducing false detections.

[0065] The third model switching mode may be set in overlap with the first model switching mode. In this case, for example, the first, second, and third weather classification learning models are used for each of the first and second periods.

[0066] In this embodiment, the learning model is switched based on the period and weather conditions, but in addition, the learning model may be switched based on the current environmental conditions of the monitoring area, specifically, the condition of the road and the status of moving objects (speed, number of objects, etc.) traveling on the road. In this case, the current environmental conditions of the monitoring area may be determined based on images captured by camera 1 or detection results from other sensors.

[0067] Next, we will describe the overview of the processes performed in the first, second, and third operating modes of this system. Figure 7 is a block diagram illustrating the overview of the processes performed in the first operating mode. Figure 8 is a block diagram illustrating the overview of the processes performed in the second operating mode. Figure 9 is a block diagram illustrating the overview of the processes performed in the third operating mode. Here, we will describe the case of the first model switching mode (period switching process) as an example.

[0068] As shown in Figure 7, in the first operating mode, the image management server 3 first performs an image collection process (image acquisition process) in which training images used for generating a learning model (initial training) are collected from the accumulated images captured by camera 1 and registered in the database.

[0069] Furthermore, the image management server 3 performs a process to distribute the accumulated captured images to the appropriate group (image distribution process). At this time, first, the captured images are distributed to one of the first or second monitoring area groups. Then, based on the set model switching mode, the captured images are distributed to the appropriate group. The example shown in Figure 7 is when the first and second model switching modes are set, and the captured images are distributed to one of the first or second period groups based on the shooting date. Also, if the third model switching mode is set, the captured images are distributed to one of the first, second, or third weather categories based on the weather.

[0070] Furthermore, the image management server 3 performs a labeling process (annotation process) in which label information regarding the status of the monitoring area (presence or absence of abnormal events) is added to the training image. At this time, the monitoring terminal 4 receives input from the user regarding the status of the monitoring area that appeared in the captured image, i.e., whether it is abnormal (a state in which an abnormal event has occurred) or normal (a state in which no abnormal event has occurred), and the entered status of the monitoring area is added to the training image as label information.

[0071] In the first operating mode, the image analysis server 2 uses the training images stored in the image management server 3 to generate a learning model using machine learning such as deep learning (learning model generation process). In this example, a learning model for each period is generated for each monitoring area. If cameras 1 are installed at multiple locations, a learning model for each monitoring area and period is generated for each camera 1.

[0072] As shown in Figure 8, in the second operating mode, the image analysis server 2 performs image analysis processing using a learned model on the real-time captured images transmitted from the camera 1, and performs an anomaly detection process to determine the presence or absence of an anomaly based on the analysis results. If an anomaly is detected, the monitoring terminal 4 notifies the user of the occurrence of the anomaly. At this time, a notification screen containing notification information corresponding to the anomaly that occurred is displayed on the monitoring terminal 4.

[0073] Furthermore, the image analysis server 2 performs a process to switch the trained model (model switching process) based on the configured model switching mode. The example shown in Figure 8 is when the first model switching mode is set, and a process is performed to switch the trained model for each period (season) based on the current date (period switching process). When the second model switching mode is set, the trained model is switched based on the likelihood as a result of image analysis processing using the trained model (transition period likelihood switching process). When the third model switching mode is set, a process is performed to switch the trained model for each weather condition based on the current weather in the monitoring area (weather switching process).

[0074] As shown in Figure 9, in the third operating mode, similar to the second operating mode, image analysis processing and anomaly detection processing are performed in the image analysis server 2. Furthermore, similar to the first operating mode, image collection processing, image sorting processing, and annotation processing are performed in the image management server 3, and learning model generation processing is performed in the image analysis server 2. In the learning model generation processing of the image analysis server 2, additional learning is performed using training images different from the training images collected in the first operating mode, and the learning model used for image analysis is updated to the learning model acquired through additional learning.

[0075] Next, the schematic configuration of the image analysis server 2, the image management server 3, and the monitoring terminal 4 will be described. Figure 10 is a block diagram showing the schematic configuration of the image analysis server 2, the image management server 3, and the monitoring terminal 4.

[0076] The image management server 3 comprises a communication unit 31, a storage unit 32, and a processor 33.

[0077] The communication unit 31 communicates with the camera 1, the image analysis server 2, and the monitoring terminal 4.

[0078] The memory unit 32 stores programs executed by the processor 33. The memory unit 22 stores captured images received from the camera 1 and training images collected from those captured images. The training images are stored in groups based on the monitoring area, period, and weather classification. Label information regarding the state of the monitoring area (presence or absence of abnormal events) that appears in each training image is associated with the stored training images. Information regarding the group to which the training images belong and the label information are registered and managed in a database as attribute information of the training images.

[0079] The processor 33 performs various processes by executing programs stored in the memory unit 32. In this embodiment, the processor 33 performs image acquisition processing, image sorting processing, and annotation processing, etc.

[0080] In the image acquisition process, the processor 33 collects training images from the accumulated captured images to be used in generating the learning model and registers them in the database. At this time, in response to user input operations performed on the monitoring terminal 4, captured images that are unsuitable as training images are excluded from the training images.

[0081] In the image sorting process, the processor 33 sorts the accumulated captured images into the appropriate groups. First, the captured images are sorted into groups related to the monitoring area. Then, based on the set model switching mode, the captured images are sorted into groups related to the period and weather category.

[0082] In the annotation process, the processor 33 labels the captured images, which serve as training images, by adding label information about the state of the monitoring area (the occurrence of abnormal events) that appeared in the captured images, in response to user input operations performed on the monitoring terminal 4. The annotation process is performed not only on the captured images used for initial training as training images, but also on the captured images used for additional training.

[0083] The image analysis server 2 comprises a communication unit 21, a storage unit 22, and a processor 23.

[0084] The communication unit 21 communicates with the camera 1, the image management server 3, and the monitoring terminal 4.

[0085] The memory unit 22 stores programs executed by the processor 33, etc. The memory unit 22 also stores information about learning models generated separately for each monitoring area, period, and weather category. Furthermore, the memory unit 22 stores information about learning models previously used for rollback purposes.

[0086] The processor 23 performs various processes by executing programs stored in the memory unit 22. In this embodiment, the processor 23 performs learning model generation processing, learning model management processing, learning model switching processing, image analysis processing, and anomaly detection processing.

[0087] In the learning model generation process, the processor 23 obtains training images from the image management server 3 and uses these training images to generate a learning model using machine learning such as deep learning. The learning model is generated separately for each monitoring area, period, and weather category.

[0088] In the learning model management process, processor 23 manages the learning models used in the image analysis process for anomaly detection. Specifically, processor 23 applies the learning model acquired through initial training to the image analysis process. Furthermore, processor 23 updates the learning model used in the image analysis process with the learning model acquired through additional training.

[0089] Furthermore, in the learning model management process, processor 23 performs a rollback process to revert the learning model used in image analysis processing back to a previous learning model specified by the user. The rollback process is executed when the user determines that a rollback is necessary because the currently running learning model has many problems, specifically because of frequent false positives, and instructs the system to perform a rollback.

[0090] In the learning model switching process, the processor 23 switches the learning model used in the image analysis process based on the set model switching mode. Specifically, in the first model switching mode, the learning model is switched based on the date of the day, according to the period (season) (period switching process). In the second model switching mode, the learning model is switched based on the likelihood as a result of the image analysis process using the learning model (transition period likelihood switching process). In the third model switching mode, the learning model is switched based on the current weather in the monitoring area, according to the weather (weather switching process).

[0091] In the image analysis process, the processor 23 uses the learning model generated in the learning model generation process to perform image analysis on real-time images captured from camera 1 and obtains a likelihood that an abnormal event has occurred. Specifically, the captured image is input to the learning model, and the likelihood of the abnormal event output from the learning model as an analysis result is obtained.

[0092] In the anomaly detection process, the processor 23 detects an anomaly based on the results of the image analysis process. At this time, the presence or absence of an anomaly is determined from the likelihood of the anomaly obtained in the image analysis process, based on the sensitivity specified by the user.

[0093] The monitoring terminal 4 comprises a display 41, an input device 42, a communication unit 43, a storage unit 44, and a processor 45.

[0094] The display 41 shows the screen. The input device 42 is a keyboard or mouse, and detects user input.

[0095] The communication unit 43 communicates with the image management server 3 and the image analysis server 2.

[0096] The memory unit 44 stores programs and other data executed by the processor 45.

[0097] The processor 45 performs various processes by executing programs stored in the memory unit 44. In this embodiment, the processor 45 performs display input control processing, etc.

[0098] In the display input control process, the processor 45 displays various screens, such as the monitoring screen 101 (see Figure 15), on the display 41, and acquires input operation information in response to the user's operation of the input device 42. The user's input operation information is sent to the image management server 3 and the image analysis server 2.

[0099] Next, we will explain the processing performed by the image management server 3. Figure 11 is a flowchart showing the procedure for the processing performed by the image management server 3.

[0100] First, the image management server 3 acquires real-time images captured by camera 1 (ST101). Next, the image management server 3 determines whether or not there is movement in the captured images (motion detection process) (ST102). For example, images that capture moving objects (cars, trains, etc.) moving on a road (road, railway tracks, etc.) or images captured while camera 1 is performing PTZ operation (pan, tilt, zoom) to switch monitoring areas will have motion in them.

[0101] If there is motion in the captured image (Yes in ST102), no special processing is performed on the image management server 3. Captured images with motion are excluded from the training images because there is a high possibility of false detection in the anomaly detection process.

[0102] On the other hand, if there is no movement in the captured image (No in ST102), the image management server 3 registers the captured image as a training image in the database (image acquisition process) (ST103).

[0103] Next, the image management server 3 distributes the captured images to the corresponding image groups (image distribution process) (ST104).

[0104] When the period is switched in the first model switching mode, the captured images are sorted into period-specific image groups based on the date and time of shooting. Specifically, if the date and time of shooting falls within the first period (spring / summer), the captured images are sorted into the image group for the first period. If the date and time of shooting falls within the second period (autumn / winter), the captured images are sorted into the image group for the second period.

[0105] Furthermore, when weather switching is performed in the third model switching mode, the captured images are sorted into image groups based on the weather (environment) on the day of shooting. Specifically, if the weather on the day of shooting corresponds to the first weather category (sunny / cloudy), the captured images are sorted into the image group for the first weather category. If the weather on the day of shooting corresponds to the second weather category (rainy), the captured images are sorted into the image group for the second weather category. If the weather on the day of shooting corresponds to the third weather category (snowy), the captured images are sorted into the image group for the third weather category.

[0106] Next, we will describe the processing performed in the first model switching mode on the image analysis server 2. Figure 12 is a flowchart showing the procedure for processing in the first model switching mode.

[0107] Image analysis server 2 first determines whether the current day (the day the image analysis is performed) falls within the first period (spring / summer) or the second period (autumn / winter) (ST201).

[0108] If the day falls within the first period (spring / summer) (referred to as "first period" in ST201), the image analysis server 2 performs image analysis processing using the learning model for the first period (ST202), and then uses the image analysis results to perform anomaly detection processing (ST203).

[0109] On the other hand, if the day falls within the second period (autumn / winter) (referred to as "second period" in ST201), the image analysis server 2 performs image analysis processing using the learning model for the second period (ST204), and then uses the image analysis results to perform anomaly detection processing (ST205).

[0110] Next, we will explain the processing performed in the second model switching mode on the image analysis server 2. Figure 13 is a flowchart showing the procedure for processing in the second model switching mode.

[0111] Image analysis server 2 first determines whether the current day (the day the image analysis is performed) falls within the first period (spring / summer) or the second period (autumn / winter) (ST201).

[0112] If the day falls within the first period (spring / summer) (referred to as "first period" in ST201), the image analysis server 2 performs image analysis processing using the learned model for the first period and obtains the likelihood as the image analysis result (ST202).

[0113] Next, the image analysis server 2 determines whether the likelihood is equal to or greater than a predetermined threshold value (e.g., 0.85), that is, whether the image analysis results are sufficiently reliable (ST211).

[0114] Here, if the likelihood is above the threshold value, that is, if the image analysis results are sufficiently reliable (Yes in ST211), the results of the image analysis processing using the learning model for the first period (spring and summer) are adopted and the anomaly detection processing is performed (ST212).

[0115] On the other hand, if the likelihood is below the threshold value, i.e., if the image analysis results are not sufficiently reliable (No in ST203), the image analysis process using the second period (autumn / winter) is performed (ST213), and the anomaly detection process is performed using the image analysis results (ST214).

[0116] Furthermore, if the day falls within the second period (autumn / winter) (referred to as "second period" in ST201), the image analysis server 2 performs image analysis processing using the learning model for the second period and obtains the likelihood as the image analysis result (ST204).

[0117] Next, the image analysis server 2 determines whether the likelihood is equal to or greater than a predetermined threshold value (e.g., 0.85), that is, whether the image analysis result has sufficient reliability (ST215).

[0118] Here, if the likelihood is above the threshold value, that is, if the image analysis results are sufficiently reliable (Yes in ST215), the results of the image analysis processing using the second period (autumn / winter) with the learned model are adopted and the anomaly detection processing is performed (ST216).

[0119] On the other hand, if the likelihood is below the threshold value, i.e., if the image analysis results are not sufficiently reliable (No in ST215), the image analysis process using the learning model for the first period (spring / summer) is performed (ST217), and the anomaly detection process is performed using the image analysis results (ST218).

[0120] In this example, image analysis is first performed using a learning model uniformly set for a given period. If the results obtained with sufficient reliability are not obtained using that learning model, the system switches to another learning model and performs image analysis using that model. Alternatively, the results (likelihood) of image analysis using each of the two learning models can be compared, and the learning model with the higher likelihood can be selected and its results adopted.

[0121] Next, we will explain the processing performed in the third model switching mode on the image analysis server 2. Figure 14 is a flowchart showing the procedure for processing in the third model switching mode.

[0122] The image analysis server 2 determines, based on real-time images captured by camera 1, whether the current weather in the monitoring area falls under the first weather category (sunny / cloudy), the second weather category (rainy), or the third weather category (snowy) (weather category determination process) (ST301).

[0123] At this time, the first weather category (sunny / cloudy) is determined based on the color of the sky and the shadow conditions included in the captured image. Furthermore, the second weather category (rainy) is determined based on the road surface conditions of the road included in the captured image. Finally, the third weather category (snowy) is determined based on the road surface conditions of the road included in the captured image.

[0124] If the current weather in the monitoring area corresponds to the first weather category (in ST301, "first weather category"), image analysis processing is performed using the learning model for the first weather category (ST302), and the results of that image analysis are used to perform anomaly detection processing (ST303).

[0125] Furthermore, if the current weather in the monitoring area falls under the second weather category (indicated as "second weather category" in ST301), image analysis processing is performed using the learning model for the second weather category (ST304), and the results of that image analysis are used to perform anomaly detection processing (ST305).

[0126] Furthermore, if the current weather in the monitoring area falls under the third weather category (indicated as "third weather category" in ST101), image analysis processing is performed using a learning model for the third weather category (ST306), and the results of that image analysis are used to perform anomaly detection processing (ST307).

[0127] In this example, the image analysis server 2 obtains the current weather in the monitoring area from real-time images captured by camera 1. However, the image analysis server 2 may also obtain the current weather in the monitoring area based on weather information distributed periodically (for example, every 5 minutes) from the weather information distribution system.

[0128] Next, we will explain the monitoring screen 101 displayed on the monitoring terminal 4 before configuration. Figure 15 is an explanatory diagram showing the monitoring screen 101 before configuration.

[0129] The monitoring screen 101 is provided with a monitoring area list display section 102. The monitoring area list display section 102 allows the user to select the target monitoring area. In this example, cameras 1 (road cameras #1 to #8) are installed at multiple locations within multiple monitoring areas (areas #1 to #8).

[0130] Furthermore, the monitoring screen 101 is equipped with a camera image list display unit 103. The camera image list display unit 103 displays real-time images (live images) from multiple cameras 1 installed in the monitoring area selected by the monitoring area list display unit 102. This allows the user (monitoring officer) to visually check the images captured by all cameras 1 that are filming the target monitoring area and confirm the current status of the monitoring area from each camera 1.

[0131] Furthermore, the camera image list display unit 103 is provided with a camera name display field 104 and a setting mode display field 105. The camera name display field 104 displays the name of camera 1. The setting mode display field 105 displays the settings for the operation mode and model switching mode. In this example, since the settings for the operation mode and model switching mode have not been completed, the text "Not set" is displayed.

[0132] The monitoring screen 101 also has a "Settings" button 107 and an "Operation" button 108. In the monitoring screen 101, the "Operation" button 108 is selected, and when the user operates the "Settings" button 107, the system transitions to the processing target settings screen 111 (see Figure 16).

[0133] Next, we will explain the processing target setting screen 111 displayed on the monitoring terminal 4. Figure 16 is an explanatory diagram showing the processing target setting screen 111.

[0134] The processing target setting screen 111 is provided with a monitoring area list display section 102. The monitoring area list display section 102 allows the user to select the target monitoring area.

[0135] Furthermore, the processing target setting screen 111 is provided with a camera image list display unit 112. The camera image list display unit 112 displays real-time captured images (live images) from each of the multiple cameras installed in the monitoring area selected in the monitoring area list display unit 102.

[0136] The camera image list display unit 112 allows the user to set whether or not each camera 1 should be included in the anomaly detection processing performed by the image analysis server 2. Specifically, when the user performs a predetermined operation (right-click) on the camera name display field 104, a menu 113 is displayed, and the user can select either to include or exclude the camera from processing in the menu 113. If "include" is selected, the corresponding camera 1 is set as a target for anomaly detection processing; if "exclude" is selected, the corresponding camera 1 is excluded from anomaly detection processing.

[0137] Furthermore, when a user selects a target camera 1 by operating one of the camera name display fields 104 for each camera 1 in the camera image list display unit 112, the system transitions to a camera settings screen 121 (see Figure 17) targeting the selected camera 1.

[0138] Next, we will explain the camera settings screen 121 displayed on the monitoring terminal 4. Figure 17 is an explanatory diagram showing the camera settings screen 121.

[0139] The camera settings screen 121 is equipped with a camera attribute information display unit 122. The camera attribute information display unit 122 displays the attribute information of the target camera 1, specifically the installation date and time, name, and IP address of camera 1. This allows the user to check the attribute information of camera 1.

[0140] Furthermore, the camera settings screen 121 is provided with a camera image display unit 123. The camera image display unit 123 displays real-time images (live images) captured by the target camera 1. In this example, images from the first monitoring area and the second monitoring area are displayed. This allows the user to visually check the shooting status of the target camera 1 by viewing the images captured by camera 1.

[0141] Furthermore, the camera settings screen 121 is provided with a camera settings information display unit 124. The camera settings information display unit 124 displays setting information related to the shooting conditions (preset positions) of the target camera 1, specifically the pan, tilt, and zoom settings of camera 1. In this example, setting information related to the shooting conditions for the first monitoring area and the second monitoring area is displayed. The user can also change the pan, tilt, and zoom settings on the camera settings information display unit 124.

[0142] Furthermore, the camera settings screen 121 is provided with a "Get Camera Information" button 125. When the user operates the "Get Camera Information" button 125, the image analysis server 2 retrieves the attribute information of camera 1 from the storage unit 22 and displays it on the camera attribute information display unit 122, retrieves the captured image from camera 1 and displays it on the camera image display unit 123, and retrieves the setting information of camera 1 from the storage unit 22 and displays it on the camera setting information display unit 124.

[0143] Furthermore, the camera settings screen 121 has a "Camera Settings" tab 126 and a "Processing Condition Settings" tab 127. In the camera settings screen 121, the "Camera Settings" tab 126 is selected, and when the user operates the "Processing Condition Settings" tab 127, the screen transitions to the processing condition settings screen 131 (see Figure 18).

[0144] Next, we will explain the processing condition setting screen 131 displayed on the monitoring terminal 4. Figure 18 is an explanatory diagram showing the processing condition setting screen 131.

[0145] The processing condition setting screen 131 is provided with a camera image display unit 132. The camera image display unit 132 displays real-time images (live images) captured by camera 1. When the detection area and mask area are set, the detection area and mask area are displayed on the captured image on the camera image display unit 132 (see Figure 3).

[0146] Furthermore, the processing condition setting screen 131 is provided with an operation mode setting unit 133. In the operation mode setting unit 133, the user can specify the operation mode for each of the first monitoring area and the second monitoring area. In this example, one of the first, second, or third operation modes is specified.

[0147] Furthermore, the processing condition setting screen 131 is provided with a model switching mode setting unit 134. In the model switching mode setting unit 134, the user can specify the model switching mode for each of the first monitoring area and the second monitoring area. In this example, one of the first, second, or third model switching modes is specified.

[0148] In the model switching mode setting unit 134, when the user selects the first model switching mode, the period switching setting screen 151 (see Figure 19) is displayed as a pop-up on the processing condition setting screen 131.

[0149] Furthermore, the processing condition setting screen 131 is provided with a detection area setting section 135. In the detection area setting section 135, the user can specify the detection areas that will be subject to anomaly detection processing for each of the first monitoring area and the second monitoring area. The detection area setting section 135 is provided with a detection area selection field 141, a coordinate display field 142, an "Add" button 143, a "Delete" button 144, and a "Set" button 145.

[0150] When the user operates the detection area selection field 141, a pull-down menu is displayed, allowing the user to select a detection area from the pull-down menu. At this time, the system transitions to the detection area input mode, and the user can use the input device 42 to specify the positions of multiple endpoints of the polygon representing the detection area on the captured image displayed on the camera image display unit 132 (see Figure 3).

[0151] The coordinate display area 142 shows the coordinates of multiple endpoints of the polygon representing the detection area.

[0152] When a user presses the "Add" button 143, they can add a new detection area. When a user presses the "Settings" button 144, the detection area specified by the user is set. When a user presses the "Delete" button 145, they can delete a previously set detection area.

[0153] Furthermore, the processing condition setting screen 131 is provided with a mask area setting section 136. In the mask area setting section 136, the user can specify mask areas to be excluded from the anomaly detection process for each of the first monitoring area and the second monitoring area. The mask area setting section 136 is provided with a mask area selection field 146, a coordinate display field 147, an "Add" button 148, a "Delete" button 149, and a "Set" button 150.

[0154] When the user operates the mask area selection field 146, a pull-down menu is displayed, allowing the user to select a mask area from the pull-down menu. At this time, the system transitions to mask area input mode, and the user can use the input device 42 to specify the positions of multiple endpoints of the polygon representing the mask area on the captured image displayed on the camera image display unit 132 (see Figure 3).

[0155] The coordinate display area 147 shows the coordinates of multiple endpoints of the polygon representing the mask area.

[0156] When a user clicks the "Add" button 148, they can add a new mask area. When a user clicks the "Set" button 149, the mask area specified by the user is set. When a user clicks the "Delete" button 150, they can delete a set mask area.

[0157] Furthermore, the processing condition setting screen 131 is provided with a sensitivity setting unit 137. In the sensitivity setting unit 137, the user can specify the sensitivity, which serves as the reference value for determining the presence or absence of abnormal events in the abnormality detection process, for each of the first and second monitoring areas. In this example, the sensitivity is entered as a numerical value (0 to 100). Note that if the sensitivity is set low, abnormal events will be less likely to be detected.

[0158] Furthermore, the processing condition setting screen 131 is provided with a "Register" button 138. When the user operates the "Register" button 138, the information entered on this screen is registered as setting information.

[0159] Next, we will explain the period switching setting screen 151 displayed on the monitoring terminal 4. Figure 19 is an explanatory diagram showing the period switching setting screen 151.

[0160] The period switching setting screen 151 is provided with a learning model display section 152. The learning model display section 152 displays the application period of the learning model (start date and end date) and information about the learning model (creation date and time, monitoring area, and content). In the application period display field, the user can enter the start date and end date.

[0161] In this example, the first learning model applied to the first period (spring and summer) and the second learning model applied to the second period (autumn and winter) are switched. For the first learning model, the application period is specified as from the vernal equinox to the day before the autumnal equinox. For the second learning model, the application period is specified as from the autumnal equinox to the day before the vernal equinox.

[0162] The screen also has a "Register" button 153 and a "Back" button 154. When the user operates the "Register" button 153, the information entered on this screen is registered as setting information. When the user operates the "Back" button 154, they return to the processing condition setting screen 131 (see Figure 18).

[0163] Next, we will explain the configured monitoring screen 161 displayed on the monitoring terminal 4. Figure 20 is an explanatory diagram showing the configured monitoring screen 161.

[0164] In the monitoring screen 161, which is configured, i.e., when the operating mode and model switching mode settings have been completed, the settings for the operating mode and model switching mode are displayed in the setting mode display field 105. Otherwise, it is the same as the monitoring screen 101 before configuration (see Figure 15).

[0165] Next, we will explain the annotation screen 171 displayed on the monitoring terminal 4. Figure 21 is an explanatory diagram showing the annotation screen 171.

[0166] The annotation screen 171 is provided with a collection period specification section 172. The collection period specification section 172 is provided with buttons 173 for day, week, month, and specified period. By operating the day, week, and month buttons 173, the administrator can specify the most recent day, week, and month, respectively, as the collection period. When the administrator operates the specified period button 173, a period input screen (not shown) is displayed, on which the administrator can arbitrarily specify the collection period. The image management server 3 collects images from the accumulated captured images that fall within the specified collection period as candidate training images.

[0167] Furthermore, the annotation screen 171 is provided with an information list display unit 174. The information list display unit 174 displays all captured images (candidate training images) included in the specified collection period, along with their additional information, in chronological order.

[0168] The information list display unit 174 is provided with a captured image display area 175. The captured image display area 175 is provided for each of the first monitoring area and the second monitoring area. The captured image display area 175 displays images captured by camera 1 for each of the first monitoring area and the second monitoring area as candidates for training images used to generate the learning model.

[0169] Furthermore, the information list display unit 174 is provided with a date and time display field 176 and a shooting time display field 177. The date and time display field 176 displays the date and time when each captured image (candidate training image) was taken. The shooting time display field 177 displays the time (minutes) when each captured image was taken. In this example, the first shooting area and the second shooting area are switched at 30-minute intervals, with the shooting time for the first shooting area being from 0 minutes to 29 minutes into each time slot, and the shooting time for the second shooting area being from 30 minutes to 59 minutes into each time slot.

[0170] Furthermore, the information list display unit 174 is provided with an exclusion designation field 178. The exclusion designation field 178 is provided for each of the first monitoring area and the second monitoring area. In the exclusion designation field 178, the user can specify whether or not to exclude a captured image (a candidate for a training image) from the training images. Specifically, for example, if the user performs a predetermined operation (right-click) on the exclusion designation field 178, a menu (not shown) will be displayed, and if the user selects to exclude, the captured image will be set as an exclusion target. In this case, for example, if the captured image contains only a part of a vehicle, the captured image will be excluded from the training images if it is inappropriate as a training image.

[0171] Furthermore, the information list display unit 174 is provided with a status input field 179 (label information input field). A status input field 179 is provided for each of the first and second monitoring areas. In the status input field 179, the user can specify the state of the monitoring area that appears in the captured image, that is, whether it is abnormal (a state in which an abnormal event has occurred) or normal (a state in which no abnormal event has occurred). Specifically, for example, when the user performs a predetermined operation (right-click) in the status input field 179, a menu (not shown) is displayed, where the user can input the state of the monitoring area. At this time, the user can determine the presence or absence of an abnormal event by visually inspecting the captured image in the captured image display unit 175. As a result, the state of the monitoring area is acquired as label information to be attached to the training image, and the training image is labeled.

[0172] Furthermore, the information list display unit 174 is provided with a weather display field 180. The weather display field 180 is provided for each of the first monitoring area and the second monitoring area. The weather display field 180 displays the results of the captured image sorting process performed by the image management server 3. In this example, the captured images are sorted by weather, and it displays whether the weather is the first weather category (sunny / cloudy), the second weather category (rainy), or the third weather category (snowy).

[0173] In this example, images are sorted by weather conditions, but images may also be sorted by period (season). In this case, the information list display unit 174 is provided with a period display field, which indicates whether it is the first period (spring / summer) or the second period (autumn / winter).

[0174] Furthermore, the annotation screen 171 is provided with a "Create Learning Model" button 181. When a user operates the "Create Learning Model" button 181, a process is initiated to create a learning model using the captured images included in the specified collection period as training images. At this time, captured images designated as exclusion targets in the exclusion specification field 178 of the information list display unit 174 are excluded from the training images used to create the learning model.

[0175] Incidentally, the annotation screen 171 is displayed on the monitoring terminal 4 for annotating (labeling) captured images used for initial training as training images, but it is also displayed on the monitoring terminal 4 for annotating (labeling) captured images used for additional training. In this case, the status of the monitoring area is displayed in the status input field 179 (label information input field) as a result of detecting an abnormal event, and the user visually checks the captured images to confirm whether the detection result of the abnormal event is correct or incorrect. If it is a false detection, the user corrects the status of the monitoring area in the status input field 179. Alternatively, only captured images in which an abnormal event was detected may be extracted as candidates for training images.

[0176] Next, we will explain the monitoring screen 191 displayed on the monitoring terminal 4 during an anomaly. Figure 22 is an explanatory diagram showing the monitoring screen 191 during an anomaly.

[0177] In the event of an anomaly, i.e., when an abnormal event is detected, the monitoring screen 191 highlights the camera name display field 104 of the camera image list display unit 103, which represents the camera 1 that captured the image in which the abnormal event was detected, and the region name display field 106 of the monitoring region list display unit 102, which represents the monitoring region in which that camera 1 is installed. For example, the background of the camera name display field 104 and the region name display field 106 is displayed in red.

[0178] When the user operates the camera name display field 104, the system transitions to the status confirmation screen 211 (see Figure 24).

[0179] Next, we will explain the notification screen 201 displayed on the monitoring terminal 4. Figure 23 is an explanatory diagram showing the notification screen 201.

[0180] When an abnormal event is detected, notification screen 201 is displayed as a pop-up on the abnormal monitoring screen 191 (see Figure 22).

[0181] The notification screen 201 displays the name of camera 1 that captured the monitoring area where the abnormal event was detected, a description of the monitoring area where the abnormal event was detected, and text indicating that an abnormal event has occurred.

[0182] The notification screen 201 is equipped with a "Confirm" button 202. When the user operates the "Confirm" button 202, the notification screen 201 is closed and the system transitions to the abnormal monitoring screen 191 (see Figure 22).

[0183] Next, we will explain the status confirmation screen 211 displayed on the monitoring terminal 4. Figure 24 is an explanatory diagram showing the status confirmation screen 211.

[0184] The status confirmation screen 211 is equipped with a live image display unit 212. The live image display unit 212 displays real-time captured images (live images) of the monitoring area where an abnormal event has been detected and is subject to notification. By visually inspecting the real-time captured images (live images), the user (monitor) can confirm the current status of the monitoring area where the abnormal event has been detected in detail.

[0185] Furthermore, the status confirmation screen 211 is equipped with a captured image list display unit 213. The captured image list display unit 203 displays a chronological list of captured images of the first monitoring area and the second monitoring area at the time the abnormal event was detected. By visually inspecting the captured images, the user can confirm in detail the status of the monitoring area at the time the abnormal event was detected.

[0186] The image list display unit 213 is also provided with a date and time display field 214, a shooting time display field 215, and an alarm level display field 216. The date and time display field 214 displays the date and time when each image was taken. The shooting time display field 215 displays the time (in minutes) when each image was taken. The alarm level display field 216 displays the alarm level (high, medium, low).

[0187] Furthermore, the status confirmation screen 211 is equipped with a learning model information display unit 217. The learning model information display unit 217 displays the identification information of the learning model currently in operation, that is, the learning model currently used for image analysis processing, specifically the creation date and time of the learning model.

[0188] Furthermore, the status confirmation screen 211 is equipped with a button 218 for instructing the user to roll back the learning model. If the user determines that the currently running learning model has many problems, specifically that there are many false positives, and that a rollback to a previous learning model is necessary, they operate button 218. This causes the rollback settings screen 221 (see Figure 25) to pop up on the status confirmation screen 211.

[0189] Next, we will explain the rollback settings screen 221 displayed on the monitoring terminal 4. Figure 25 is an explanatory diagram showing the rollback settings screen 221.

[0190] The rollback settings screen 221 includes a model selection section 222. The model selection section 222 displays a list of the creation date, monitoring area, and content for each learning model. The user can select a learning model by manipulating the display field for any of the learning models.

[0191] Furthermore, the rollback settings screen 221 is equipped with a "Perform Switch" button 223 and a "Back" button 224. When the user operates the "Perform Switch" button 223, a rollback process is performed to switch the learning model used for image analysis processing to the learning model selected in the model selection unit 222. When the user operates the "Back" button 224, they return to the status confirmation screen 211 (see Figure 24). When a learning model is switched due to a rollback of the learning model, it is desirable to display information such as the learning model name and version number representing the content of the currently applied learning model on the status confirmation screen 211. The user can refer to this displayed information when deciding whether the updated learning model will work effectively the next time they adopt it.

[0192] As described above, embodiments have been explained as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these embodiments and can be applied to embodiments that have been modified, replaced, added, or omitted. Furthermore, it is possible to create new embodiments by combining the components described in the above embodiments. [Industrial applicability]

[0193] The monitoring device and monitoring method according to the present invention have the effect of enabling the administrator to appropriately switch the operating state of the system and to perform image analysis processing using a learning model with high accuracy. By performing image analysis processing using a learning model on images captured in the monitoring area, the device and monitoring method are useful for detecting various events (such as abnormal events) that occur in the monitoring area and controlling the issuance of alarms. [Explanation of symbols]

[0194] 1 Camera 2. Image analysis server (monitoring device) 3. Image management server (monitoring device) 4. Monitoring terminals 23 processors 33 processors 45 processors

Claims

1. Multiple cameras installed in a monitoring area, A server device that controls the collection of training images necessary for generating a learning model used for image analysis processing of images captured in a monitoring area set for each camera, and the notification of events detected in the monitoring area by performing image analysis processing on the captured images using the learning model. The system includes a terminal that displays an operation screen including a notification screen regarding the notification of the aforementioned event and a settings screen for setting the processing conditions for each camera performed by the server device, The server device is Based on the user's operation to set the processing conditions for each camera on the aforementioned settings screen, A first operating mode involves collecting the aforementioned captured images as training images and generating the learning model using those training images, A second operating mode in which the event is detected and an alert is issued by the image analysis processing using the learning model generated in this first operating mode, A surveillance system characterized by setting one of the following operating modes for each camera: an operating mode in which the system detects the event and issues an alert by image analysis processing using the learning model described above; a third operating mode in which the system collects the captured images as training images, performs additional learning to generate the learning model using the training images, and updates the learning model used in the image analysis processing using the learning model generated by the additional learning; and the system executes the processing defined in the operating mode.

2. The server device is Using the training images that have been divided into multiple time periods, the learning models for each of the multiple time periods are generated. The monitoring system according to claim 1, characterized in that it switches to the learning model for the period corresponding to the monitoring day and performs the image analysis processing.

3. The server device is Using the training images that have been sorted into multiple weather categories, a learning model is generated for each of the multiple weather categories. The monitoring system according to claim 1, characterized in that it switches to the learning model for the weather category corresponding to the day of monitoring and performs the image analysis processing.

4. The server device is The monitoring system according to claim 2 or 3, characterized in that it switches the learning model used in the image analysis process based on the likelihood included in the results of the image analysis process.

5. The server device is The monitoring system according to claim 1, characterized in that it detects events and issues alerts for each monitoring area using images taken for each monitoring area obtained by changing the shooting conditions of the camera.

6. The server device is The monitoring system according to claim 1, characterized in that, when the learning model used in the image analysis processing has been updated, a rollback process is performed by the user to revert from the currently operating learning model to the previous learning model.

7. A monitoring method in which a server device performs the following processes: collecting training images necessary for generating a learning model used for image analysis processing of images captured in a monitoring area set for each of the multiple cameras installed in the monitoring area, and controlling the notification of events detected in the monitoring area by performing image analysis processing on the captured images using the learning model, An operation screen is displayed on the terminal, which includes a notification screen regarding the notification of the aforementioned event and a settings screen for setting the processing conditions for each camera performed by the server device. Based on the user's operation to set the processing conditions for each camera on the aforementioned settings screen, A first operating mode involves collecting the aforementioned captured images as training images and generating the learning model using those training images, A second operating mode in which the event is detected and an alert is issued by the image analysis processing using the learning model generated in this first operating mode, A monitoring method characterized by setting one of the following operating modes for each camera: an operating mode in which an event is detected and an alert is issued by image analysis processing using the learning model; a third operating mode in which the captured images are collected as training images, additional learning is performed to generate the learning model using the training images, and the learning model used in the image analysis processing is updated using the learning model generated by the additional learning; and the processing specified in the operating mode is executed.

Images