Method, computing unit and computer program

Abstract

The invention relates to a method which comprises receiving (201) a number of alarm events from a monitoring device (2), determining (202) an alarm event value for the alarm event of the number of alarm events by means of a machine-learning-based method, selecting (203) an alarm event from the number of alarm events based at least on the alarm event value; receiving (206) at least one piece of information about the selected alarm event; and adjusting (207) the machine-learning-based method using the received information.

Description

[0001] Description

[0002] title

[0003] Process, computing unit and computer program

[0004] The present invention relates to a method as well as a computing unit and a computer program for its implementation.

[0005] Background of the invention

[0006] The surveillance of surveillance areas, such as buildings and / or open spaces, is often carried out using sensors and / or video cameras. The video cameras and / or sensors provide surveillance footage, which is often evaluated and reviewed by security personnel in central control centers or security operations centers (SOCs).

[0007] In this context, DE 10 2016 222 134 A1 describes a video analysis device for a surveillance device for monitoring a surveillance area, wherein the surveillance device comprises at least one video camera, wherein the video camera is arranged in the surveillance area for monitoring a section of the surveillance area, wherein the video camera provides video data and metadata.

[0008] Disclosure of the invention

[0009] According to the invention, a method, in particular for selecting an alarm event from a set of alarm events and / or for adapting the machine learning-based method, as well as a computing unit and a computer program for implementing the method are proposed, having the features of the independent patent claims. Advantageous embodiments are the subject of the dependent claims and the following description.

[0010] The background to the invention is the following insight: Video surveillance cameras can be configured to trigger alarms, e.g., in perimeter security facilities, when the screen detects intruder movement within a restricted area. These alarms are then forwarded to security personnel or security control centers for further action. However, most of these alarms (up to 90%) are so-called "false alarms" caused by non-critical disturbances or, e.g., severe weather conditions or other natural disturbances in the environment. Typically, the user must review, evaluate, comment on, and document the actions taken for each of these detected alarms when an alarm occurs, so that the audit trail is updated and the actions taken are documented in a security log.However, it is observed that not all alarms are always documented in the security log with a comment on the situation that triggered the alarm. This may be due to a site being disarmed, for example, during bad weather (e.g., a thunderstorm), which typically leads to many false alarms for a video analytics setup, so that all alarms are simply suppressed (i.e., not appearing in the list), or that not every consecutive alarm is commented on, or that the user was too busy and only commented on some of the alarms (at the user's discretion as to which alarms to document).

[0011] The invention presents a way to improve the overall performance of the system and significantly reduce the number of detected false alarms by using a machine learning-based evaluation method to select only certain alarm events for information input by the user (user review, commenting, or classification), particularly those whose detection or classification result is ambiguous, e.g., in the form of an alarm event value that lies outside one or more specific, unambiguous ranges. A user, such as a security officer, can then enter information about a selected alarm event, particularly by labeling, falsifying, or annotating it.The user's annotations on these alarms can be referred to as "ground truth" (GT) (a true description of the actual situation that triggered the alarm), which can then be used to retrain and adapt the machine learning-based procedure or model and / or to update the system's parameters to improve the overall system performance.

[0012] The invention moves away from a purely user-based decision regarding which alarm events to review toward an automated decision. This specifically solves the problem that the user typically doesn't know which alarm events are important for the underlying model to measure and improve performance based on the real label.

[0013] By improving the security protocol over time, the number of potential false alarms is optimized for the user, giving them more time to detect and respond to, for example, only the relevant security issues and incidents of the object, while significantly improving system performance.

[0014] Security logs in SOCs are a common method for reporting and documenting incidents. Alarm events can be displayed to the user (e.g., in a table / list view). Clicking on such an event can display the alarm event with associated video clips and / or a summarized image (best shot) to expedite the validation process. After validating the alarm event and assessing it as either a true alarm (to initiate further action) or a false alarm (suppressing / dismissing the alarm event) in the case of malfunctions or identified non-critical activities, the user can complete the alarm event investigation and move on to the next alarm event.In one embodiment, the invention improves such a process by providing for a user dialog to be displayed, which, in particular, prompts the user to review the alarm event and provide information or feedback. This display or prompt is generated dynamically, i.e., the user dialog does not appear for every alarm event, but only for the selected ones.

[0015] In one embodiment, the selection of an alarm event from the set of alarm events is further carried out based on at least one criterion selected from: the type of alarm event (burglar alarm, loitering alarm, object in the area, ...), the time at which the user investigated the alarm event, i.e. the time of a review of the alarm event, the time at which the alarm event occurred (e.g. during the night), the number of further alarm events that occurred in a certain period of time (e.g. a few seconds to a few minutes) around the alarm event, the measured image / video quality, the review time that the user spent watching the videos.

[0016] In one embodiment, the user dialog for receiving information for the selected alarm event is displayed based on the time elapsed since the last time the user dialog for receiving information for a (different) selected alarm event was displayed. In other words, the user is not prompted to check more than once within a certain period of time to avoid causing frustration or disrupting the user's daily routine.

[0017] In one embodiment, selecting an alarm event from the set of alarm events is still based on the number of displayed alarm events. For example, if there aren't many alarm events and the user's workload is light, the user may be able to spend more time providing feedback than in a hectic situation, where the additional time spent flagging alarm events and the user's workload should be minimized.

[0018] In one embodiment, the user dialog for receiving the information is displayed in response to user input. In other words, the user has the opportunity to provide feedback at any time. Even if the machine learning-based process has not selected the alarm event, the user is free to do so anyway. This means that the user can add information to a current alarm event at any time.

[0019] In one embodiment, the user dialog includes a set of predetermined information that can be selected by the user. To ensure seamless and intuitive use for the user, it is advantageous if the user has to enter as little information as possible, but can instead select predetermined general tags, e.g., "rain," "snow," "animal," "dog," "cat," "bird," "insect," "spider," "cobweb," "wind," "light show," etc. Provision can also be made to generate new, predetermined, selectable information from entered information.

[0020] In one embodiment, the user dialog includes a set of predetermined information that can be selected by the user depending on the time of day. In particular, the lighting conditions and thus also the lights and shadows visible in the image depend on the time of day.

[0021] A computing unit according to the invention, e.g. a control unit of a video analysis device, is configured, in particular in terms of programming, to carry out a method according to the invention.

[0022] The implementation of a method according to the invention in the form of a computer program or computer program product with program code for carrying out all method steps is also advantageous, since this entails particularly low costs, in particular if an executing control unit is also used for other tasks and is therefore already present. Finally, a machine-readable storage medium is provided with a computer program stored thereon, as described above. Suitable storage media or data carriers for providing the computer program are, in particular, magnetic, optical, and electrical memories, such as hard disks, flash memories, EEPROMs, DVDs, and others. Downloading a program via computer networks (Internet, intranet, etc.) is also possible. Such a download can be wired or cable-based or wireless (e.g., via a WLAN network, a 3G, 4G, 5G, or 6G connection, etc.).

[0023] Further advantages and embodiments of the invention will become apparent from the description and the accompanying drawings.

[0024] The invention is illustrated schematically in the drawing using exemplary embodiments and is described below with reference to the drawing.

[0025] Short description of the drawings

[0026] Figure 1 shows schematically a video analysis device which may form the basis of an embodiment of the invention.

[0027] Figure 2 shows an exemplary implementation of an embodiment for selecting an alarm event from a set of alarm events.

[0028] Embodiment(s) of the invention

[0029] Figure 1 shows a video analysis device 1 for a monitoring device 2. The monitoring device comprises a plurality of sensors 3 and a plurality of video cameras 4. The sensors 3 are, for example, fire detectors, thermal sensors, motion detectors, chip card readers, or other sensors. The video cameras 4 are, in particular, color video cameras and are designed, for example, as CCD or CMOS cameras. The video cameras 4 and / or the sensors 3 are arranged in a monitoring area 5, wherein the sensors 3 and / or video cameras 4 are arranged regularly in the monitoring area 5; alternatively and / or additionally, the video cameras 4 and the sensors 3 are arranged irregularly in the monitoring area 5. The video cameras 4 and the sensors 3 are designed to monitor the monitoring area 5 visually and / or using sensors.The video cameras 4 and / or the sensors 3 each monitor a section of the surveillance area 5, wherein the sections recorded and / or monitored by the individual video cameras 4 and / or sensors 3 preferably overlap, so that the entire surveillance area 5 can be monitored visually and / or by sensors. Real objects 6 are arranged in the surveillance area 5, wherein the real objects 6 are, for example, people, animals and / or objects. In particular, the real objects in the surveillance area are variable, so that the position and / or properties of the real objects 6 can change over time. The change in the position and / or properties of the real objects 6 corresponds in particular to an alarm event in the surveillance area 5.

[0030] For example, a sensor 3 is designed as a motion sensor, so that a motion sensor 3 can display and / or record the movement of the real object 6 in the surveillance area 5. The video cameras 4 provide video data 7, and the sensors 3 provide sensor data.

[0031] The monitoring device 2 comprises a data generation unit 9, wherein the data generation unit 9 is designed to provide video data 7 and metadata 8 based on the video data 7 and / or the sensor data of the video analysis device 1. The metadata 8 includes, in particular, information about the real objects 6 in the monitoring area 5, such as their position, their size, and / or other information. The video data includes, in particular, video images of the section of the monitoring area 5 from one and / or the video cameras 4 in the monitoring area 2. The video analysis device 1 here comprises two input interfaces 10, wherein the input interfaces 10 are connected to the monitoring device 2 for data purposes, wherein the input interfaces 10 are designed to receive the video data 7 and the metadata 8.

[0032] The video analysis device 1 comprises a central processing module 11. In particular, the central processing module 11 is designed as a central processor unit, for example, as a microprocessor. The central processing module 11 is supplied, in particular, with the metadata 8 and the video data 7.

[0033] The process module 11 is configured to evaluate the video data 7 based on the metadata 8 and the video data 7, in particular to determine an alarm event value for each alarm event. For this purpose, a machine learning-based method is implemented in the process module 11, which serves as a false alarm classification system, i.e., classifies the alarm events as a true alarm or a false alarm.

[0034] In principle, all machine learning methods that enable classification are conceivable here. In particular, all different types of neural networks can be used. Supervised learning can be used as a method. Supervised learning means that a user checks the classification result and corrects it if necessary in order to improve the false alarm classification system.

[0035] An exemplary sequence of an embodiment of the method is explained below with reference to Figures 1 and 2.

[0036] The method may begin with an optional step 200, which includes a learning or training mode. In such a training mode, all alarm events received by the monitoring device 2 are displayed to a user, for example, on a display means or a human-machine interface (HMI) 20. In particular, the display of no alarm events is suppressed, even those that are potentially false alarms. In this training mode, a user dialog is displayed for each alarm event to receive information, which in particular includes a classification of the selected alarm event as an alarm or false alarm. Based on these classifications, the system can, in particular, "learn" which display criterion, e.g., a value range, an alarm event value must correspond to in order to be certain of being a genuine alarm, and which suppression criterion, e.g.,A range of values ​​an alarm event value must correspond to in order to be certain that it is a false alarm. The system can run in this mode for some time (depending on the number of alarm events) until enough feedback has been collected and the false alarm classification system has been sufficiently trained.

[0037] In the case of supervised learning, it is necessary to provide labeled training data, which is already assigned to one of the predefined classes. Various options are possible for this. In addition to the manual labeling of training data described here (entering information), for example, by a user indicating whether a true alarm or a false alarm has occurred, semi-automatic or automatic labeling can be used alternatively or additionally. For example, a larger data set can be automatically labeled based on a small labeled data set; optionally, this semi-automatic labeling can then be manually reviewed afterwards.

[0038] In a next step 201, the video analysis device 1 then switches to regular operating mode. In this regular operating mode, the video analysis device 1 receives a set of alarm events, in the example described from the monitoring device 2. This receipt can occur in real time, e.g., whenever movement is detected, or from a recorded memory.

[0039] In a next step 202, an alarm event value is determined for each alarm event using the machine learning-based method. In a step 203, all alarm events whose alarm event value corresponds to the display criterion are displayed, and the display of all alarm events whose alarm event value corresponds to a suppression criterion is suppressed. As a result of the previously performed learning process, the two classes can thus be assigned to a large number of alarm events. Furthermore, one or more alarm events are selected from the set of alarm events based at least on the alarm event value, in particular for user verification, and are also displayed.

[0040] In one embodiment, in particular the alarm events at the boundary between the suppression criterion and the display criterion, or - if such a boundary does not exist - alarm events that neither the suppression criterion nor the display criterion (clearly) fulfill, are selected.

[0041] It is also possible to measure the distance in the feature space of the extracted features of the false alarm classification system to decide which alarm events should be reviewed and annotated. This helps ensure that the labels cover the entire feature space as well as possible.

[0042] The displayed alarm events are viewed by a user in a step 204, in particular in a known manner on a display means such as a monitor.

[0043] In a step 205, it is determined for the alarm event just viewed whether it belongs to the alarm events selected in step 203. If not, branch 0, the user can view further alarm events, step 204. If so, branch 1, the user is prompted in a step 206, for example by means of a user dialog, particularly at the end of the viewing, which can be determined, for example, by clicking a corresponding button, to review this alarm event and, in particular, to enter information, in particular comprising the classification as a real alarm or false alarm.

[0044] The time spent by the user triaging the alarm event can be used to distinguish alarms that are difficult for the user and require a significant amount of time. These are important alarms that should be commented on and used to improve the underlying classification system. Either they are false alarms, meaning the system should suppress them (if the system is not running in training mode) and the user need not pay attention to them, or they are genuine alarms, in which case the user needs to be guided as to what is happening in the scene, e.g., by seeing where the activity is taking place.

[0045] If the user repeatedly replays a video associated with the alarm event and zooms in on the footage, this may also indicate that the scene is challenging. Therefore, these videos should be carefully annotated to obtain sophisticated data for training and testing system performance.

[0046] It's also beneficial to count / measure how many annotations were made by each user and take this into account when selecting, particularly favoring users with fewer annotations. This ensures that user reviews are evenly distributed among multiple people, which generally ensures better quality.

[0047] In a step 207, the information is used to adapt or "retrain" the machine learning-based method.

[0048] Since training the model can be very complex depending on the amount of data and the algorithm used, it is also possible to perform the training phase of the model on a processing unit with more computing / memory power, for example in a data center, and then transfer the trained model obtained in this way to another processing unit, such as a PC for video surveillance.

[0049] For evaluation, the alarm events, essentially as image data, can then be input to the trained classifier, which then provides an alarm event value and / or one of the predefined classes as output, e.g., true alarm or false alarm. In all cases, it is possible for various steps described here together or in a single unit to be executed separately in time and / or space. The process module 11 can also be implemented in any desired manner, e.g., as a central control unit or part thereof, as a control computer, via an external server, as a cloud service, or others.

[0050] It is also possible for alarm events or their data to be stored, at least temporarily, and processed at a later time or permanently for further analysis. It goes without saying that corresponding storage units can be located anywhere. This means that, in principle, any time period can elapse between the recording of a video signal and the subsequent processing and evaluation steps; however, it is also possible for the signals to be further processed and evaluated immediately.

[0051] In all cases, suitable user interfaces such as displays, screens, speakers, touchscreens, or other output elements can also be used, for example, to display the results of the status evaluation, to map intermediate steps for a user, to provide indications of error states, to indicate problems during the evaluation, or to output other information to a user. Input devices can also be provided, e.g., a keyboard and / or mouse, a touchscreen, a microphone for voice input, or any other common input devices, via which, for example, process parameters can be selected or changed.

Claims

Claims 1. Procedure comprising: Receiving (201) a set of alarm events from a monitoring device (2), Determining (202) an alarm event value for the, in particular for each, alarm event of the set of alarm events by means of a method based on machine learning, selecting (203) an alarm event from the set of alarm events based at least on the alarm event value; Receiving (206) at least one item of information about the selected alarm event; Adapting (207) the machine learning-based method using the received information.

2. The method of claim 1, further comprising: Displaying (206) a user dialog for receiving the information for the selected alarm event.

3. The method of claim 2, wherein the displaying (206) of the user dialog for receiving the information for the selected alarm event is based on a period of time that has elapsed since the last display of the user dialog for receiving the information for another selected alarm event.

4. The method of any preceding claim, further comprising: displaying (206) a user dialog for receiving the information for a non-selected alarm event in response to a user input.

5. The method according to any one of claims 2 to 4, wherein the user dialog comprises a set of predetermined information that is selectable by the user.

6. The method according to any one of the preceding claims, further comprising: displaying (201) all alarm events whose alarm event value corresponds to a display criterion.

7. The method according to any one of the preceding claims, further comprising: suppressing (201) the display of all alarm events whose alarm event value corresponds to a suppression criterion.

8. The method according to any one of the preceding claims, wherein the selection of an alarm event from the set of alarm events is further performed based on at least one criterion selected from a type of alarm event; a time of a review of the alarm event; a time at which the alarm event occurred; a number of further alarm events occurring within a specific period around the alarm event; an image / video quality; and a review period.

9. Method according to one of the preceding claims, wherein the information about the selected alarm event comprises a classification of the selected alarm event as an alarm or a false alarm.

10. A computing unit (11) configured to carry out all method steps of a method according to one of the preceding claims.

11. A computer program which causes a computing unit to perform all method steps of a method according to any one of claims 1 to 9 when executed on the computing unit.

12. A machine-readable storage medium having a computer program according to claim 11 stored thereon.

Images