Anomaly detection device, anomaly detection method, and anomaly detection program

Abstract

To suppress a false detection as an abnormality when the background in an area to be observed changes without reducing the operation rate of a system.SOLUTION: A frame image formed by capturing an image of an observation object area is inputted into an image input unit. A detection unit detects an object captured in the frame image inputted into the image input unit by means of a detection model constructed by machine learning using learning data including the frame image formed by imaging the observation object area. A determination unit determines whether an abnormal event has occurred in the observation object area on the basis of the object detected by the detection unit. When the determination unit determines that an abnormal event has occurred within the observation object area, an output unit outputs the fact.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a technique for determining whether an abnormal state in which an abnormal event has occurred within an observation area is present, based on a frame image of the observation area. [Background technology]

[0002] Conventionally, a technique for determining whether an observed state is normal or abnormal using a model constructed by machine learning has been used in various fields. For example, Patent Document 1 describes a device that uses a variational autoencoder (hereinafter referred to as VAE (Variational Autoencoder)) to determine whether an observed state is normal or abnormal.

[0003] A VAE is a model that includes an encoder that calculates latent variables from observation data, which is input data, and a decoder that restores the observation data from the latent variables. A VAE is generated by machine learning that mainly uses observation data from normal observations of the object being observed. Here, the observation data restored from the latent variables is called restored data. The observation data is, for example, frame images of the object being observed.

[0004] The device described in Patent Document 1 is configured to calculate the difference between observed data and restored data, and determine whether the state of the object to be observed is abnormal or normal based on this difference. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 6740247 Summary of the Invention [Problem to be solved by the invention]

[0006] However, when the background of the observation area changes, the model, which was built using machine learning using frame images of the observation area, sometimes mistakenly recognized (determined) that an abnormality had occurred in the area where the background had changed. For example, when a poster that had been attached to a wall was replaced with another poster, the model sometimes detected the replaced poster as an object and mistakenly determined that an abnormality had occurred in the observation area.

[0007] To prevent such erroneous judgments, it is conceivable to reconstruct the model by performing machine learning using frame images of the observation area each time the background in the observation area changes, such as when posters are replaced. However, this approach would mean halting system operation from the time the posters are replaced until the model reconstruction by machine learning is complete. Reconstructing the model takes a relatively long time, which reduces the system's operational rate. In particular, if the observation area is a place where posters are replaced frequently, this approach of reconstructing the model would result in a very low system operational rate.

[0008] The object of this invention is to provide a technology that can suppress the false recognition that anomalies have occurred in areas where the background has changed within an observation area, without reducing the operational rate of a system that detects anomalies using a model constructed using machine learning. [Means for solving the problem]

[0009] In order to achieve the above object, the anomaly detection device of the present invention is configured as follows.

[0010] The detection unit detects objects captured in frame images input to the image input unit using a detection model constructed by machine learning using training data including frame images of the observation target area. The determination unit determines whether an abnormal event has occurred in the observation target area based on the object detected by the detection unit. If the determination unit determines that an abnormal event has occurred in the observation target area, the output unit outputs that fact.

[0011] For example, an object detected by the detection unit is stored in a storage unit. A setting reception unit receives a setting of classification information for an object stored in the storage unit, which indicates whether or not an abnormal event is to be determined to have occurred in an observation target area when the object is detected. The classification information is, for example, a label indicating, for each object stored in the storage unit, whether or not the object is to be determined to be in an abnormal state in which an abnormal event has occurred, or in a normal state in which no abnormal event has occurred, when the object is detected. The determination unit determines whether or not an abnormal event has occurred in an observation target area using a determination model constructed by machine learning using the objects stored in the storage unit and the classification information received by the setting reception unit.

[0012] This configuration can suppress erroneous recognition that an abnormality has occurred in a portion of the observation area where the background has changed. For example, even if a poster is replaced within the observation area and the replaced poster is detected as an object, the judgment model can determine that no abnormal event has occurred. In this case, for example, a judgment model can be constructed that distinguishes between suspicious individuals and workers (security guards, cleaning staff, etc.) and determines whether an abnormal event has occurred within the observation area. That is, even if a worker is detected within the observation area, it can be determined that no abnormal event has occurred, and conversely, if a suspicious individual is detected within the observation area, it can be determined that an abnormal event has occurred.

[0013] Furthermore, for example, the classification information may be a label indicating, for each type of object, whether or not it is determined that an abnormal event has occurred in the observation target area when an object of that type is detected.

[0014] Furthermore, the determination unit may be configured to determine whether an abnormal event has occurred within the observation area based on, for example, an area in a frame image of the observation area where an object detected by the detection unit is captured. Even with this configuration, it is possible to determine that no abnormal event has occurred, for example, even if a poster has been replaced within the observation area and the replaced poster is detected as an object.

[0015] For example, the configuration may include a setting receiving unit that receives, for a frame image of an observation area, the setting of an area mask that defines a non-determination area that is not involved in determining whether an abnormal event has occurred within the observation area, and the determination unit may determine that an abnormal event has occurred within the observation area if the area in the frame image of the observation area in which an object detected by the detection unit has been imaged is not a non-determination area of ​​the area mask.In other words, the determination unit may be configured to determine that no abnormal event has occurred within the observation area if the area in the frame image of the observation area in which an object detected by the detection unit has been imaged is a non-determination area of ​​the area mask.

[0016] Furthermore, for example, the area mask may be configured to accept the setting of a use condition using the area mask, and the determination unit may use the area mask that satisfies the use condition to determine whether an abnormal event has occurred within the observation target area. The use condition may be one or more of, for example, day type, time period, validity period, etc.

[0017] Also, for example, a composite image generation unit that generates a composite image using frame images of an observation target area and an image of an object detected by the detection unit; The detection unit may additionally include an evaluation unit that evaluates the result of object detection performed on the composite image generated by the composite image generation unit.

[0018] This configuration allows the detection model to be evaluated during system operation. For example, a synthetic image can be created for an abnormal image that has never been observed, and then evaluated. [Effects of the Invention]

[0019] According to this invention, it is possible to suppress the erroneous recognition that an abnormality has occurred in a part of the observation target area where the background has changed, without reducing the operation rate of the system. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a diagram illustrating a configuration of a detection system to which an anomaly detection device of this example is applied. [Figure 2] 1 is a block diagram showing the configuration of a main part of an anomaly detection device according to this embodiment. [Figure 3] FIG. 4 is a diagram showing detection information stored in a storage unit. [Figure 4] 4 is a flowchart showing an abnormality determination process in the abnormality detection device of this example. [Figure 5] 10 is a flowchart showing a label setting process of the anomaly detection device of this example. [Figure 6] 10 is a flowchart showing additional learning processing of the anomaly detection device of this example. [Figure 7] FIG. 10 is a block diagram showing the configuration of a main part of an anomaly detection device according to a first modified example. [Figure 8] 10 is a flowchart showing an abnormality determination process in the abnormality detection device of Modification 1. [Figure 9] FIG. 10 is a block diagram showing the configuration of a main part of an anomaly detection device according to a second modification. [Figure 10] FIG. 10 is a diagram illustrating an example of a mask image. [Figure 11] 10 is a flowchart showing an abnormality determination process in the abnormality detection device of Modification 2. [Figure 12] 11 is a flowchart showing an abnormality determination process in the abnormality detection device of Modification 3. [Figure 13] FIG. 10 is a block diagram showing the configuration of a main part of an anomaly detection device according to a fourth modification. [Figure 14] 13 is a flowchart showing an evaluation process of an anomaly detection device according to a fourth modification. DETAILED DESCRIPTION OF THE INVENTION

[0021] Hereinafter, an embodiment of the present invention will be described.

[0022] <1. Application Examples> 1 is a diagram showing the configuration of a detection system to which the anomaly detection device of this example is applied. The detection system 100 of this example includes an anomaly detection device 1, a setting terminal 2, and a camera 5.

[0023] The detection system 100 in this example determines whether an observation area is in an abnormal state where an abnormal event has occurred, or in a normal state where no abnormal event has occurred, and outputs the determination result. The observation area can be various locations such as a station ticket gate, a train platform within a station, an entrance to an office building, or an entrance to an apartment building. An abnormal event can be a suspicious person entering the observation area, a suspicious object left in the observation area, etc.

[0024] The camera 5 is installed at an angle that fits the observation target area into the imaging range. In this example, the camera 5 is a video camera that captures moving images. The frame rate of the camera 5 is, for example, several tens of frames / sec (e.g., approximately 10 to 30 frames / sec). The camera 5 outputs frame images of the moving images captured of the observation target area to the anomaly detection device 1.

[0025] The anomaly detection device 1 selects frame images to be processed from among the frame images of the observation area captured by the camera 5. The anomaly detection device 1 may select the input frame images in order as frame images to be processed, or may select frame images to be processed from the input frame images at intervals of a predetermined number of frames (for example, every 3 frames, every 5 frames), or may select frame images to be processed from the input frame images at a predetermined imaging time interval (for example, every 100 msec, every 300 msec).

[0026] The camera 5 may be configured to select frame images to be processed from among the frame images of the captured video, and output the selected frame images to be processed to the anomaly detection device 1. In this case, the camera 5 may or may not output to the anomaly detection device 1 frame images that were not selected as frame images to be processed.

[0027] When the anomaly detection device 1 selects a frame image to be processed, it uses a detection model to detect an object captured in the frame image to be processed. This detection model is an image recognition model (AI (Artificial Intelligence) image recognition model) constructed by machine learning using training data including frame images of an observation target area. The detection model receives the frame image to be processed as input, estimates the captured area of ​​the object in the frame image to be processed, and outputs the estimated area.

[0028] Furthermore, the anomaly detection device 1 determines whether an abnormal state, in which an abnormal event has occurred within an observation area, is present, or whether an abnormal event has not occurred and the state is normal, based on the object detected by the detection model. The anomaly detection device 1 makes this determination using a determination model. The determination model is an image recognition model (AI image recognition model) constructed by machine learning using learning data that associates an image of an object detected by the detection model with a label indicating whether the state should be determined to be abnormal or normal when this object is detected. The determination model receives as input an image of an object detected by the detection model (an image of the imaging area of ​​the object detected by the detection model), estimates whether the state is abnormal or normal, and outputs the result.

[0029] The anomaly detection device 1 outputs the judgment result of the judgment model. The anomaly detection device 1 may be configured to output the judgment result when the judgment model judges that an abnormal state exists. In other words, the anomaly detection device 1 may be configured not to output the judgment result when the judgment model judges that an abnormal state exists.

[0030] The setting terminal 2 is a well-known, general-purpose personal computer. The operator uses the setting terminal 2 to associate a label with an object detected by the detection model of the anomaly detection device 1, indicating whether the object will be judged to be in an abnormal state or a normal state when detected. That is, the setting terminal 2 accepts an input operation (input operation by the operator) of a label to be set for each object detected by the detection model. Based on the operator's input operation, the setting terminal 2 outputs the label to be set for the object detected by the detection model to the anomaly detection device 1. This label corresponds to the classification information referred to in this invention. In this example, the label "0" indicates that the state is judged to be normal, and the label "1" indicates that the state is judged to be abnormal.

[0031] In this example detection system, for example, if a poster that was attached to a wall in an observation area is replaced with another poster and the detection model detects the replaced poster as an object, the determination model can determine that the state is normal. Also, for example, if the detection model detects a person moving within the observation area, and this person is a cleaner or the like, the determination model can determine that the state is normal. On the other hand, if the person detected by the detection model is a suspicious person or the like, the determination model can determine that the state is abnormal.

[0032] In this way, when the background in the observation area changes, the detection system in this example can prevent erroneous determinations that an abnormality has occurred in areas where the background has changed, without having to rebuild the model through machine learning. Therefore, it is possible to prevent erroneous determinations that an abnormality has occurred in areas where the background in the observation area has changed, without reducing the system's operational efficiency.

[0033] <2.Configuration example> 2 is a block diagram showing the configuration of the main parts of the anomaly detection device 1 of this example. The anomaly detection device 1 includes a control unit 11, an image input unit 12, a setting reception unit 13, a storage unit 14, and an output unit 15.

[0034] The control unit 11 controls the operation of each part of the main body of the abnormality detection device 1. The control unit 11 also has a detection unit 11a, a determination unit 11b, and a label setting unit 11c. The detection unit 11a, the determination unit 11b, and the label setting unit 11c of the control unit 11 will be described later.

[0035] A camera 5 is connected to the image input unit 12. Frame images of a moving image of the observation target area captured by the camera 5 are input to the image input unit 12. The camera 5 is installed at an angle that fits the observation target area into the imaging area.

[0036] A setting terminal 2 is connected to the setting reception unit 13. An operator operates the setting terminal 2 to set a label to be associated with an object detected by the anomaly detection device 1. The label to be associated with the object detected by the anomaly detection device 1 is input from the setting terminal 2 to the setting reception unit 13.

[0037] The storage unit 14 stores detection information related to each object detected by the anomaly detection device 1. FIG. 3 is a diagram showing the detection information stored in the storage unit. As shown in FIG. 3, the detection information includes an object image, an imaging position of the object in the frame image, an imaging time of the frame image, and the like. The detection information is also associated with a label set for the object. The detection information may also include information such as the imaging date, the day type of the imaging date (day of the week, and public holiday), and the like. The storage unit 14 is configured with a storage medium such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), for example.

[0038] The output unit 15 outputs the determination result (abnormal state or normal state) determined by the control unit 11 for the observation target area to a higher-level device (not shown).

[0039] Next, the detection unit 11a, the determination unit 11b, and the label setting unit 11c included in the control unit 11 will be described.

[0040] The detection unit 11a receives frame images of an observation target area as input and detects objects (people and objects) captured in the frame images. The detection unit 11a detects objects using a detection model constructed by machine learning using learning data including frame images of the observation target area. The detection unit 11a stores detection information of objects detected by the detection model in the storage unit 14.

[0041] The determination unit 11b determines the state (abnormal state or normal state) within the observation area based on the object detected by the detection unit 11a. The determination unit 11b determines the state within the observation area using a determination model constructed by machine learning using learning data including detection information associated with labels stored in the storage unit 14.

[0042] The label setting unit 11c associates the label received by the setting receiving unit 13 with the detection information stored in the storage unit .

[0043] The control unit 11 of the anomaly detection device 1 is composed of a hardware CPU, memory, and other electronic circuits. When the hardware CPU executes the anomaly detection program according to the present invention, it operates as a detection unit 11a, a determination unit 11b, and a label setting unit 11c. The memory also has an area for expanding the anomaly detection program according to the present invention and an area for temporarily storing data generated when the anomaly detection program is executed. The control unit 11 may be an LSI that integrates the hardware CPU, memory, and the like. The hardware CPU is also a computer that executes the anomaly detection method according to the present invention.

[0044] <3. Example of operation> The operation of the anomaly detection device of this example will be described below with reference to a flowchart of FIG.

[0045] The detection unit 11a of the anomaly detection device 1 selects a frame image to be processed from among the frame images (frame images related to moving images of the observation target area captured by the camera 5) input to the image input unit 12 (s1). The detection unit 11a executes object detection processing to detect an object captured in the frame image to be processed selected in s1 (s2). The detection unit 11a detects the object captured in the frame image to be processed using a detection model.

[0046] The determination unit 11b determines whether the detection unit 11a has detected an object captured in the frame image to be processed (s3). If the detection unit 11a has not detected an object captured in the frame image to be processed in s3, the determination unit 11b determines that the state is normal. If the detection unit 11a has detected an object captured in the frame image to be processed in s3, the determination unit 11b generates detection information for the object detected this time and stores the generated detection information in the storage unit 14 (s4). The detection information stored in the storage unit 14 in s4 is not associated with a label.

[0047] If the detection unit 11a detects a plurality of objects captured in the frame image to be processed in s3, the determination unit 11b generates detection information for each detected object and stores the generated detection information in the storage unit 14. Furthermore, the determination unit 11b executes a determination process to determine whether the state is abnormal or normal based on the object captured in the processing target frame image detected by the detection unit 11a in s3 (s5). The determination unit 11b determines whether the state is abnormal or normal using a determination model.

[0048] When the determination unit 11b of the anomaly detection device 1 determines that the state is abnormal, the output unit 15 outputs that the state is abnormal to the higher-level device (s6, s7), and the process returns to s1. When the determination unit 11b determines that the state is not abnormal (i.e., that the state is normal), the output unit 15 of the anomaly detection device 1 outputs that the state is normal to the higher-level device (s6, s8), and the process returns to s1.

[0049] If the detection unit 11a does not detect an object captured in the frame image to be processed in s3, the abnormality detection device 1 outputs to the host device at the output unit 15 that the state is normal (s8), and returns to s1.

[0050] 5 is a flowchart showing the label setting process of the anomaly detection device of this example. This label setting process is started, for example, when an operator performs an input operation to start the process on the setting terminal 2. The label setting unit 11c selects detection information to be processed from the detection information stored in the storage unit 14 that is not associated with a label (s11). The label setting unit 11c outputs the detection information selected in s11 to the setting terminal 2 connected to the setting reception unit 13 (s12).

[0051] As described above, the setting terminal 2 is a general PC, and is an information processing device that allows an operator to set labels for detection information. The setting terminal 2 displays the detection information output by the anomaly detection device 1 on the screen of a display device, and accepts setting operations for labels to be associated with this detection information. At this time, the operator checks the detection information (particularly, object images) displayed on the screen of the display device, and performs an input operation to set (associate) a label indicating whether an object related to this detection information is to be judged as an abnormal state or a normal state when it is detected. The setting terminal 2 outputs the label to be associated with the detection information to the anomaly detection device 1 in response to the operator's input operation.

[0052] When the setting receiving unit 13 receives a label to be associated with the detection information selected in s11, the anomaly detection device 1 associates the currently received label with the corresponding detection information stored in the storage unit 14 (s13, s14). In s14, the currently received label is stored in association with the corresponding detection information stored in the storage unit 14.

[0053] The anomaly detection device 1 determines whether there is unprocessed detection information (s15), and if there is unprocessed detection information, returns to s11 and repeats the above process. On the other hand, if there is no unprocessed detection information, the anomaly detection device 1 ends this process. Unprocessed detection information is detection information stored in the storage unit 14 that is not associated with a label.

[0054] In the above description, the anomaly detection device 1 is configured to output each piece of detection information not associated with a label to the setting terminal 2, and receive a label to be associated with the output detection information from the setting terminal 2. However, the anomaly detection device 1 may be configured to output all pieces of detection information not associated with a label to the setting terminal 2 in a lump, and receive a label to be associated with each piece of output detection information from the setting terminal 2.

[0055] Fig. 6 is a flowchart showing the additional learning process of the anomaly detection device of this example. For example, when an operator determines that the accuracy of the anomaly determination process shown in Fig. 4 has decreased, the operator causes the anomaly detection device 1 to execute this additional learning process. The accuracy of the anomaly determination process decreases when the background of the observation area changes. For example, the accuracy decreases when a poster that was attached to a wall in the observation area is replaced with another poster, causing a change in the background of the observation area.

[0056] The determination unit 11b reads out detection information to be used for additional learning of the determination model from the detection information stored in the storage unit 14 (s31). In s31, detection information relating to a plurality of objects that have not been used in the machine learning of the determination model is read out. Also in s31, detection information to which a label has been set and which is an object recently detected by the detection unit 11a is read out.

[0057] The determination unit 11b uses the detection information read out in s31 as learning data to perform additional learning of the determination model (s32), and then ends this process.

[0058] The judgment model constructed by this additional learning is a model that can suppress erroneous recognition that an abnormality has occurred in a portion of the observation area where the background has changed. For example, even if a poster is replaced within the observation area and the replaced poster is detected as an object, the judgment model constructed by this additional learning can determine that the state is normal. Furthermore, this additional learning can also construct a judgment model that can distinguish between, for example, a suspicious person and a worker (security guard, cleaning staff, etc.) and determine whether an abnormal event has occurred within the observation area. As a result, even if a worker is detected within the observation area, the state can be determined to be normal, and conversely, if a suspicious person is detected within the observation area, the state can be determined to be abnormal.

[0059] Furthermore, the detection model does not need to be reconstructed.

[0060] Therefore, in this example, it is possible to suppress the false recognition that an abnormality has occurred in a part of the observation target area where the background has changed, without reducing the operation rate of the system.

[0061] <4. Modifications> Variation 1 In the above example, a label is set for each object detected by the detection unit 11a, but in this modification 1, a label is set for each type of object. The object types referred to here are people, bags, luggage, posters, signs, etc.

[0062] 7 is a block diagram showing the configuration of the main parts of the anomaly detection device of Modification 1. Instead of storage unit 14 that stores detection information of objects detected by detection unit 11a, anomaly detection device 1A of Modification 1 includes storage unit 14A that stores determination information that associates object types with labels that indicate whether an abnormal state or a normal state is to be determined when an object of that type is detected.

[0063] Furthermore, the determination unit 11Ab of the control unit 11A of the anomaly detection device 1 of this modified example 1 determines the type of object detected by the detection unit 11a using a determination model. The determination model is an image recognition model constructed by machine learning using images of various objects as learning data. The determination model in this example receives as input an image of the object detected by the detection model (an image of a region of the frame image to be processed in which the detection model estimates that the object is captured), and estimates the type of this object.

[0064] As shown in FIG. 7, the control unit 11A of the abnormality detection device 1 of this modified example 1 does not have to include the label setting unit 11c described in the above example.

[0065] FIG. 8 is a flowchart showing the abnormality determination process of the abnormality detection device of the first modification.

[0066] The detection unit 11a of the anomaly detection device 1A of the first modification selects a frame image to be processed from frame images of a moving image of an observation target area captured by the camera 5 and input to the image input unit 12 (s41). The detection unit 11a executes an object detection process to detect an object captured in the frame image to be processed selected in s41 (s42).

[0067] The determination section 11Ab determines whether or not the detection section 11a has detected an object captured in the frame image to be processed (s43). Steps s41 to s43 are the same processes as steps s1 to s3 in the above example.

[0068] If the detection unit 11a has not detected an object captured in the frame image to be processed in s43, the determination unit 11Ab determines that the state is normal. If the detection unit 11a has detected an object captured in the frame image to be processed in s43, the determination unit 11Ab performs type determination processing to determine the type of each object detected this time (s44). The determination unit 11Ab determines the type of the detected object using a determination model.

[0069] The determination unit 11Ab performs a determination process to determine whether the state is abnormal or normal based on the type of each object captured in the processing target frame image detected by the detection unit 11a in s43, as determined in the type determination process of s44 (s45). The determination unit 11Ab determines that the state is abnormal if the objects captured in the processing target frame image detected by the detection unit 11a in s43 include an object of a type associated with a label that determines that the state is abnormal. In other words, the determination unit 11Ab determines that the state is normal if all of the objects captured in the processing target frame image detected by the detection unit 11a in s43 are objects of a type associated with a label that determines that the state is normal.

[0070] When the determination unit 11Ab of the abnormality detection device 1A determines in s45 that the abnormal state exists, the output unit 15 outputs the fact that the abnormal state exists to the higher-level device (s46, s47), and the process returns to s41. When the determination unit 11Ab determines that the abnormal state does not exist (i.e., determines that the normal state exists), the output unit 15 of the abnormality detection device 1A outputs the fact that the normal state exists to the higher-level device (s46, s48), and the process returns to s41.

[0071] If the detection unit 11a does not detect an object captured in the frame image to be processed in s43, the abnormality detection device 1A outputs to the host device at the output unit 15 that the state is normal (s48), and returns to s41.

[0072] For example, by associating a label indicating a normal state with a poster, this anomaly detection device 1A can determine that a normal state exists even if a poster is replaced within the observation area and the replaced poster is detected as an object. Also, in this example, not only is it unnecessary to reconstruct the detection model, but additional learning of the determination model is also unnecessary, further minimizing declines in operational efficiency.

[0073] Variation 2 FIG. 9 is a block diagram showing the configuration of the main parts of the anomaly detection device of this modified example 2. The anomaly detection device 1B of this modified example 2 includes a storage unit 14B that stores a mask image, instead of the storage unit 14 that stores detection information of an object detected by the detection unit 11a. The mask image is an image in which a mask area is defined in a frame image of the observation target area captured by the camera 5, in which a normal state is determined even if an object is detected. FIG. 10 is a diagram showing an example of a mask image. The mask image shown in FIG. 10 has two mask areas (mask area X and mask area Y) set. Mask area X and mask area Y correspond to the non-determination areas referred to in this invention.

[0074] Furthermore, the determination unit 11Bb of the control unit 11B of the anomaly detection device 1 of this modified example 2 determines that an abnormal state exists if the imaging position (position of the frame image) of an object detected by the detection unit 11a is outside the mask area set in the mask image. In other words, the determination unit 11Bb determines that a normal state exists if the detection unit 11a does not detect an object whose image is outside the mask area set in the mask image. Unlike the above example, the determination unit 11Bb of this modified example 2 does not have a determination model constructed by machine learning.

[0075] Furthermore, the mask image is generated by, for example, an operator setting a mask area on a frame image captured by the camera 5 on the setting terminal 2. Furthermore, although Fig. 10 shows an example in which the mask area X and the mask area Y are rectangular, they may have any shape, such as an ellipse, a circle, a semicircle, or a shape surrounded by a curve.

[0076] FIG. 11 is a flowchart showing the abnormality determination process of the abnormality detection device of the second modification.

[0077] The detection unit 11a of the anomaly detection device 1B of the second modification selects a frame image to be processed from frame images of a moving image of an observation target area captured by the camera 5 and input to the image input unit 12 (s51). The detection unit 11a executes an object detection process to detect an object captured in the frame image to be processed selected in s51 (s52). The determination unit 11Bb determines whether the object captured in the frame image to be processed has been detected by the detection unit 11a (s53). Steps s51 to s53 are the same as steps s1 to s3 in the example described above.

[0078] If the detection unit 11a has not detected an object captured in the frame image to be processed in s53, the determination unit 11Bb determines that the state is normal. If the detection unit 11a has detected an object captured in the frame image to be processed in s53, the determination unit 11Bb performs a determination process for each currently detected object to determine whether the image capture position of that object (image capture position on the frame image) is inside or outside the mask area of ​​the mask image (s54).

[0079] The determination unit 11Bb determines that the state is normal if the imaging positions of all objects captured in the processing target frame image detected by the detection unit 11a in s53 are within the mask area of ​​the mask image.The determination unit 11Bb determines that the state is abnormal if the imaging position of any object captured in the processing target frame image detected by the detection unit 11a in s53 is outside the mask area of ​​the mask image.

[0080] When the determination unit 11Bb of the abnormality detection device 1B determines that the state is abnormal, the output unit 15 outputs the abnormal state to the higher-level device (s55, s56), and the process returns to s51. When the determination unit 11Bb determines that the state is not abnormal (i.e., determines that the state is normal), the output unit 15 of the abnormality detection device 1B outputs the normal state to the higher-level device (s55, s57), and the process returns to s51.

[0081] If the detection unit 11a does not detect an object captured in the frame image to be processed in s53, the abnormality detection device 1B outputs to the host device at the output unit 15 that the state is normal (s57), and returns to s51.

[0082] This abnormality detection device 1B can also determine that the state is normal even if a poster is replaced, for example, by using a mask image in which the position on the frame image where a poster attached to a wall or the like within the observation area is captured is used as a mask area, and the replaced poster is detected as an object.

[0083] Variation 3 The abnormality detection device of the third modification has the configuration shown in Fig. 9. The abnormality detection device 1B of the third modification differs from the second modification in that it executes the abnormality determination process shown in Fig. 12 .

[0084] The abnormality determination process of the abnormality detection device of the third modified example will be described with reference to FIG.

[0085] The detection unit 11a of the anomaly detection device 1B of the third modification selects a frame image to be processed from frame images of a moving image of an observation target area captured by the camera 5 and input to the image input unit 12 (s61). The detection unit 11a performs a masking process on the frame image to be processed selected in s61 to exclude a mask area set in the mask image from an object detection area (s62). The detection unit 11a executes an object detection process on the frame image to be processed that has been subjected to the masking process to detect an object that has been captured (s63). The detection unit 11a detects objects in the non-masked areas and does not detect objects in the masked areas.

[0086] The determination unit 11Bb determines whether the detection unit 11a has detected an object captured in the processing target frame image that has been subjected to mask processing (s64). If the detection unit 11a has detected an object captured in the processing target frame image that has been subjected to mask processing, the determination unit 11Bb determines that an abnormal state has occurred, and outputs the abnormal state to the host device via the output unit 15 (s65), and the process returns to s61. On the other hand, if the detection unit 11a has not detected an object captured in the processing target frame image that has been subjected to mask processing, the determination unit 11Bb determines that the state is normal, and outputs the normal state to the host device via the output unit 15 (s66), and the process returns to s61.

[0087] As with variant 2, variant 3 uses a mask image in which the position on the frame image where a poster attached to a wall or the like within the observation area is captured is used as a mask area, so that even if the poster is replaced and the replaced poster is detected as an object, it can be determined to be in a normal state.

[0088] Furthermore, for the mask images shown in Modifications 2 and 3, a valid period (e.g., May 10, 2021 to September 15, 2021) may be set for the mask image, a valid time zone (e.g., 1:30 AM to 5:00 AM) may be set, or a valid date type (e.g., day of the week, public holiday) may be set. Furthermore, multiple valid periods, valid time zones, and valid date types may be set for the mask image. The anomaly detection device 1B performs the anomaly determination process using a mask image whose valid period, valid time zone, and valid date type correspond to the capture date and time of the frame image to be processed. In other words, in the anomaly determination process, the anomaly detection device 1B does not use a mask image whose valid period, valid time zone, and valid date type do not correspond to the capture date and time of the frame image to be processed.

[0089] Variation 4 Fig. 13 is a block diagram showing the configuration of the main parts of Modification 4. As shown in Fig. 13, anomaly detection device 1C of Modification 4 differs from anomaly detection device 1 shown in Fig. 2 in that a control unit 11C additionally includes a composite image generation unit 11d and an evaluation unit 11e.

[0090] The composite image generating unit 11d generates a composite image by pasting an object stored in the storage unit 14 (an object detected by the detecting unit 11a) onto a frame image obtained by capturing an image of the observation target area.

[0091] The evaluation unit 11e evaluates the determination accuracy of the abnormality determination process using the composite image generated by the composite image generation unit 11d.

[0092] The anomaly detection device 1C of this modified example 4 executes the anomaly determination process shown in Fig. 4, the label setting process shown in Fig. 5, and the additional learning process shown in Fig. 6. The anomaly detection device 1C of this modified example 4 differs from the anomaly detection device 1 shown in Fig. 2 in that it performs the evaluation process described below.

[0093] 14 is a flowchart showing the evaluation process of the anomaly detection device of Modification 4. The anomaly detection device 1C may be configured to start this evaluation process at a predetermined timing, or may be configured to start the evaluation process when an input operation related to a start instruction is performed by an operator or the like.

[0094] The composite image generation unit 11d generates a predetermined number of frames of composite images to be used in the current evaluation process (s71). The detection unit 11a selects a composite image to be processed from the composite images generated in s71 (s72). The detection unit 11a executes object detection processing to detect an object in the selected composite image (s73). S73 is the same processing as s2 described above.

[0095] The determination unit 11b executes a determination process to determine whether the object detected by the detection unit 11a in s73 and captured in the composite image to be processed is in an abnormal state or a normal state (s74). The process in s74 is the same as the process in s5 described above.

[0096] The evaluation unit 11e performs a determination result evaluation process to determine whether the object detection result of s73 and the determination process of s74 are appropriate (s75). The evaluation unit 11e determines whether the object pasted on the composite image has been detected appropriately, as the evaluation of the detection unit 11a. For example, the evaluation unit 11e increments the appropriate detection count value by one if the pasted object has been detected appropriately, and increments the inappropriate detection count value by one if the pasted object has not been detected appropriately.

[0097] Furthermore, the evaluation unit 11e determines whether the abnormal state or normal state determination made by the determination unit 11b is appropriate as the evaluation of the determination unit 11b. For example, the evaluation unit 11e increments the appropriate determination count value by one if the determination result is appropriate, and increments the inappropriate determination count value by one if the determination result is inappropriate.

[0098] The anomaly detection device 1C determines whether or not there is an unprocessed composite image (s76), and if there is an unprocessed composite image, returns to s72. If there is no unprocessed composite image, the anomaly detection device 1C calculates evaluation values ​​of the detection unit 11a and the determination unit 11b, outputs the calculated evaluation values ​​as evaluation results (s77), and ends this process.

[0099] In s77, for example, the evaluation values ​​of the detection unit 11a and the determination unit 11b are calculated as follows: Evaluation value of the detection unit 11a=(detection proper count value) / (detection proper count value+detection improper count value) Evaluation value of the determination unit 11b=(determination appropriate count value) / (determination appropriate count value+determination inappropriate count value) It is calculated as follows. Evaluation value of the detection unit 11a=(inappropriate detection count value) / (appropriate detection count value) Evaluation value of the determination unit 11b=(inappropriate determination count value) / (appropriate determination count value) It may also be calculated by:

[0100] In this fourth modification, the operator can be notified of evaluation values ​​that evaluate the accuracy of the models (detection model, judgment model) for each of the detection unit 11a and the judgment unit 11b. Therefore, the operator can easily determine the timing to reconstruct the detection model and the timing to perform additional learning of the judgment model. In addition, the operator can easily adjust parameters.

[0101] The above-described configuration for evaluating the detection unit 11a and the determination unit 11b can also be applied to the anomaly detection devices according to the first to third modifications described above.

[0102] It should be noted that this invention is not limited to the above-described embodiments, and that the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined. Furthermore, the order of each step in the flowcharts described in all the above examples is merely an example, and may be appropriately changed within the scope of the present invention.

[0103] Furthermore, the correspondence between the configuration according to the present invention and the configuration according to the above-described embodiment can be described as follows: <Additional Notes> an image input unit (12) to which a frame image of an observation target area is input; a detection unit (11a) that detects an object captured in a frame image input to the image input unit (12) using a detection model constructed by machine learning using learning data including frame images of the observation target area; a determination unit (11b) that determines whether an abnormal event has occurred in the observation target area based on the object detected by the detection unit (11a); and an output unit (15) that outputs a message indicating that an abnormal event has occurred in the observation target area when the determination unit (11b) determines that an abnormal event has occurred in the observation target area. [Explanation of symbols]

[0104] 1, 1A, 1B, 1C...Abnormality detection device 2...Setting terminal 5. Camera 11, 11A, 11B, 11C...Control unit 11a...Detection unit 11b, 11Ab, 11Bb...judgment section 11c...Label setting section 11d...Synthetic image generation unit 11e…Evaluation section 12...Image input unit 13...Settings reception section 14, 14A, 14B...Storage section 15...Output section 100...Detection system

Claims

1. an image input unit to which frame images of an observation target area are input; a detection unit that detects an object captured in a frame image input to the image input unit using a detection model constructed by machine learning using learning data including frame images of the observation target area; a determination unit that determines whether an abnormal event has occurred in the observation target area based on the object detected by the detection unit; and an output unit that, when the determination unit determines that an abnormal event has occurred in the observation target area, outputs that fact, a composite image generation unit that generates a composite image using frame images of the observation target area and an image of the object detected by the detection unit; an evaluation unit that calculates, at a predetermined timing, a first evaluation value that evaluates the accuracy of the detection unit based on a detection result obtained by having the detection unit detect an object in a composite image generated by the composite image generation unit, and calculates a second evaluation value that evaluates the accuracy of the determination unit based on a determination result obtained by having the determination unit determine whether an abnormal event has occurred in the observation area based on the object detected by the detection unit in the composite image generated by the composite image generation unit.

2. a storage unit that stores the object detected by the detection unit; a setting receiving unit that receives, for an object stored in the storage unit, a setting of classification information indicating whether or not to determine that an abnormal event has occurred in the observation target area when the object is detected, The determination unit determines whether an abnormal event has occurred in the observation target area using a determination model constructed by machine learning using the objects stored in the storage unit and the classification information received by the setting reception unit. The anomaly detection device according to claim 1 .

3. the setting reception unit receives, as the classification information, a label indicating whether or not it is determined that an abnormal event has occurred in the observation target area when the object stored in the storage unit is detected; The anomaly detection device according to claim 2 .

4. the setting reception unit receives, as the classification information, a label indicating, for each type of object, whether or not it is determined that an abnormal event has occurred in the observation target area when an object of that type is detected; The anomaly detection device according to claim 2 .

5. the determination unit determines whether an abnormal event has occurred in the observation target area based on an area in which an object detected by the detection unit is captured in a frame image of the observation target area. The anomaly detection device according to claim 1 .

6. a setting receiving unit that receives a setting of an area mask that defines a non-determination area that is not involved in determining whether an abnormal event has occurred in the observation target area, for a frame image that captures the observation target area; the determination unit determines that an abnormal event has occurred in the observation target area if an area in a frame image capturing the observation target area in which an object detected by the detection unit is captured is not a non-determination area of ​​the area mask. The anomaly detection device according to claim 5 .

7. the setting receiving unit receives, for the region mask, a setting of a usage condition using the region mask; the determination unit determines whether an abnormal event has occurred in the observation target area using the area mask that satisfies the use conditions. The anomaly detection device according to claim 6.

8. a detection step of detecting an object captured in a frame image input to the image input unit using a detection model constructed by machine learning using learning data including frame images capturing an observation target area input to the image input unit; a determination step of determining whether an abnormal event has occurred in the observation target area based on the object detected in the detection step; an output step of causing an output unit to output information indicating that an abnormal event has occurred in the observation target area when the determination step determines that an abnormal event has occurred in the observation target area, The computer further comprises: a composite image generating step of generating a composite image using frame images of the observation target area and an image of the object detected in the detecting step; an evaluation step of calculating, at a predetermined timing, a first evaluation value that evaluates the accuracy of the detection step based on a detection result of an object detected in the detection step in the composite image generated in the composite image generation step, and calculating a second evaluation value that evaluates the accuracy of the determination step based on a determination result of whether or not an abnormal event has occurred in the observation area based on the object detected in the detection step in the composite image generated in the composite image generation step.

9. a detection step of detecting an object captured in a frame image input to the image input unit using a detection model constructed by machine learning using learning data including frame images capturing an observation target area input to the image input unit; a determination step of determining whether an abnormal event has occurred in the observation target area based on the object detected in the detection step; an output step of outputting, when it is determined in the determination step that an abnormal event has occurred in the observation target area, a message to that effect to an output unit, The computer further comprises: a composite image generating step of generating a composite image using frame images of the observation target area and an image of the object detected in the detecting step; an evaluation step of calculating, at a predetermined timing, a first evaluation value that evaluates the accuracy of the detection step based on a detection result of an object detected in the detection step in the composite image generated in the composite image generation step, and calculating a second evaluation value that evaluates the accuracy of the determination step based on a determination result of whether or not an abnormal event has occurred in the observation target area based on the object detected in the detection step in the composite image generated in the composite image generation step.

Images