Monitoring device, monitoring method, and monitoring program

Abstract

The system uses AI-based image analysis systems to detect unsafe conditions, while allowing users to easily change the detection conditions. [Solution] An object information acquisition unit 22 acquires object information for one or more objects detected from the target image data. A determination unit 23 determines whether or not the object information acquired by the object information acquisition unit 22 satisfies a detection pattern indicated by an editable detection pattern master. A notification unit 25 issues a notification when the determination unit 23 determines that the detection pattern is satisfied.

Description

[Technical field]

[0001] The present disclosure relates to surveillance technology using image data. [Background technology]

[0002] In places such as factories and construction sites, various types of equipment are used, and if workers are not careful, their safety may be threatened. To increase the safety of workers, cameras are used for monitoring, and warnings are issued if an unsafe condition occurs.

[0003] Previously, people would visually check image data obtained by surveillance cameras to see if an unsafe state or the like occurred. In recent years, it has been considered to analyze image data obtained by surveillance cameras using an image analysis system that uses AI and issue a warning when an unsafe state or the like is detected. AI is an abbreviation for Artificial Intelligence. For example, Patent Document 1 describes a method of determining whether or not workers at a construction site are wearing equipment such as helmets and gloves by image analysis. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-043932 Summary of the Invention [Problem to be solved by the invention]

[0005] Unsafe conditions in places such as factories and construction sites change due to factors such as the introduction of new equipment, changes to equipment layout, and changes to safety standards. It is difficult for users to change the detection conditions of an image analysis system that uses AI, and in order to change the detection conditions, they must request the vendor to modify the image analysis system. Having an image analysis system modified takes time and money, which is inconvenient for users. The present disclosure aims to detect unsafe conditions using an AI-based image analysis system, etc., while allowing users to easily change the detection conditions. [Means for solving the problem]

[0006] Monitoring device according to the present disclosure an object information acquisition unit that acquires object information for one or more objects detected from the target image data; a determination unit that determines whether the object information acquired by the object information acquisition unit satisfies a detection pattern indicated by an editable detection pattern master; a notification unit that issues a notification when the determination unit determines that the detection pattern is satisfied; Equipped with. Effect of the Invention

[0007] In the present disclosure, it is determined whether or not object information about one or more objects detected from image data satisfies a detection pattern indicated by an editable detection pattern master. This determines whether or not there is an unsafe state, and determines whether or not to notify. The detection pattern master is editable, and the user can change the detection pattern by editing it. Therefore, the detection conditions can be easily changed. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a configuration diagram of a monitoring system 100 according to a first embodiment. [Diagram 2] 1 is a configuration diagram of a monitoring device 10 according to a first embodiment. [Diagram 3] FIG. 3 is an explanatory diagram of a setting file 31 according to the first embodiment. [Figure 4] FIG. 2 is an explanatory diagram of a detection pattern master 32 according to the first embodiment. [Diagram 5] FIG. 4 is an explanatory diagram of a result file 33 according to the first embodiment. [Figure 6] 4 is a flowchart showing the flow of operations of the monitoring device 10 according to the first embodiment. [Figure 7] FIG. 2 is a diagram showing an example of a detection pattern master 32 according to the first embodiment. [Figure 8] FIG. 2 is a diagram showing an example of an unsafe state according to the first embodiment. [Figure 9] FIG. 2 is a diagram showing an example of an unsafe state according to the first embodiment. [Figure 10] FIG. 2 is a diagram showing an example of an unsafe state according to the first embodiment. [Figure 11] FIG. 3 is an explanatory diagram of a measurement process in the first embodiment. [Figure 12] FIG. 11 is a configuration diagram of a monitoring device 10 according to a second embodiment. [Figure 13] FIG. 11 is an explanatory diagram of an associated pattern master 34 according to the second embodiment. [Figure 14] FIG. 11 is a diagram showing an example of a detection pattern master 32 according to the second embodiment. [Figure 15] 10 is a flowchart showing the flow of operations of the monitoring device 10 according to the second embodiment. [Figure 16] FIG. 11 is an explanatory diagram of an association pattern according to the second embodiment. [Figure 17] FIG. 13 is an explanatory diagram of a selection screen according to the third embodiment. [Figure 18] FIG. 13 is an explanatory diagram of a work monitoring screen according to the third embodiment. [Figure 19] FIG. 13 is an explanatory diagram of a detection pattern setting screen according to the third embodiment. [Figure 20] FIG. 11 is an explanatory diagram of setting a distance reference value according to the third embodiment. [Figure 21] FIG. 13 is an explanatory diagram of a combination setting screen according to the third embodiment. [Figure 22] FIG. 13 is an explanatory diagram of a related pattern setting screen according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Embodiment 1 ***Configuration Description*** The configuration of a monitoring system 100 according to the first embodiment will be described with reference to FIG. The surveillance system 100 includes a surveillance device 10, a plurality of cameras 50, and one or more alarm devices 60. The surveillance device 10 is connected to each of the cameras 50 and each of the alarm devices 60 via a network 70. The monitoring device 10 is a computer that detects and notifies unsafe conditions. Each camera 50 is a monitoring camera installed to capture an image of an area to be monitored. The alarm 60 is a device that notifies workers of unsafe conditions by sound and light.

[0010] The configuration of the monitoring device 10 according to the first embodiment will be described with reference to FIG. The monitoring device 10 is a computer. The monitoring device 10 includes the following hardware components: a processor 11, a memory 12, a storage 13, and a communication interface 14. The processor 11 is connected to other hardware components via signal lines and controls the other hardware components.

[0011] The processor 11 is an IC that performs processing. IC is an abbreviation for Integrated Circuit. Specific examples of the processor 11 include a CPU, a DSP, and a GPU. CPU is an abbreviation for Central Processing Unit. DSP is an abbreviation for Digital Signal Processor. GPU is an abbreviation for Graphics Processing Unit.

[0012] The memory 12 is a storage device that temporarily stores data. Specific examples of the memory 12 include SRAM and DRAM. SRAM is an abbreviation for Static Random Access Memory. DRAM is an abbreviation for Dynamic Random Access Memory.

[0013] The storage 13 is a storage device that stores data. Specific examples of the storage 13 include an SSD or an HDD. SSD stands for Solid State Drive. HDD stands for Hard Disk Drive. The storage 13 may also be a portable recording medium such as an SD (registered trademark) memory card, CompactFlash (registered trademark), NAND flash, a flexible disk, an optical disk, a compact disk, a Blu-ray (registered trademark) disk, or a DVD. SD stands for Secure Digital. DVD stands for Digital Versatile Disk.

[0014] The communication interface 14 is an interface for communicating with an external device. Specific examples of the communication interface 14 include Ethernet (registered trademark), USB, and HDMI (registered trademark) ports. USB is an abbreviation for Universal Serial Bus. HDMI is an abbreviation for High-Definition Multimedia Interface.

[0015] The monitoring device 10 includes, as functional components, a video acquisition unit 21, an object information acquisition unit 22, a determination unit 23, a measurement unit 24, a notification unit 25, and an operation unit 26. The functions of the functional components of the monitoring device 10 are realized by software. The storage 13 stores programs that realize the functions of each functional component of the monitoring device 10. These programs are loaded into the memory 12 by the processor 11 and executed by the processor 11. In this way, the functions of each functional component of the monitoring device 10 are realized.

[0016] The storage 13 stores a setting file 31, a detection pattern master 32, and a result file 33.

[0017] 2 shows only one processor 11. However, there may be a plurality of processors 11, and the plurality of processors 11 may cooperate to execute programs that realize the respective functions.

[0018] ***Explanation of Operation*** The operation of the monitoring device 10 according to the first embodiment will be described with reference to FIGS. The operation procedure of the monitoring device 10 according to the embodiment 1 corresponds to the monitoring method according to the embodiment 1. Moreover, the program that realizes the operation of the monitoring device 10 according to the embodiment 1 corresponds to the monitoring program according to the embodiment 1.

[0019] The setting file 31 according to the first embodiment will be described with reference to FIG. The setting file 31 sets basic information about the monitoring device 10. The setting file 31 includes a video switching flag, a threshold number of times, a video storage folder, a video storage pattern, a video storage time, a video storage period, the number of video storage frames, the number of cameras, setting information, a warning sound, and a reference size. The video switching flag is a flag for switching the source of video data between the camera 50 and an external storage device. The threshold number of times is a reference number of times for determining that the state is unsafe. The video storage folder is a folder for storing the video data. The video storage pattern is a condition for storing the video data. The video storage time is the time of the video data to be stored as one video file. The video storage period is the period for which the stored video data is to remain. The number of video storage frames is the frame rate of the video data to be stored. The number of cameras is the number of cameras 50 included in the surveillance system 100. The setting information includes a video file name, which is identification information for the file of the video data of the camera 50, a camera No., a camera name, and a connection URL, which is a connection destination of the camera 50, set for each camera 50. The warning sound is a sound that is output when it is determined that the state is unsafe, etc. One or more warning sounds are set, and a warning sound number is assigned. The reference size is the size of a reference object, and is used in measuring distance, which will be described later. The reference size is set for each type of object.

[0020] The detection pattern master 32 according to the first embodiment will be described with reference to FIG. One or more detection patterns indicating conditions for detecting a target for which a warning is issued are set in the detection pattern master 32. Each detection pattern includes a pattern number, a camera number, a warning sound number, and a judgment condition. The pattern number is an identification number of the detection pattern. The camera number is an identification number of the camera 50 to which the detection pattern corresponds. In other words, the detection pattern is set for each camera 50. The warning sound number is an identification number of the warning sound corresponding to the detection pattern. The judgment condition is a condition as to whether or not the detection pattern is met. A plurality of detection pattern masters 32 can be set for each camera 50. Also, one detection pattern master 32 can be shared by a plurality of cameras 50. The correspondence between the cameras 50 and the detection pattern masters 32 can be one-to-one, one-to-many, many-to-one, or many-to-many.

[0021] The judgment conditions include one or more of a detected object, an area, and a distance. Each detection object is a condition for an object to be detected. Each detection object includes a type of object to be detected and a classification of whether the object to be detected is detected or not. The type of object indicates what the object to be detected is. The type of object is set to, for example, a person, a mask, a crane, or the like. The classification of whether the condition is satisfied if the object is detected, or whether the condition is satisfied if the object is not detected. The area is a condition for the area in which the object to be detected is detected. Whether or not the condition is met is determined based on the presence or absence of detection in the area indicated by the area. If the area is not set, whether or not the condition is met is determined based on the presence or absence of detection in the entire shooting area of ​​the camera 50. The distance is a condition regarding the distance between two objects. The distance includes a distance reference value and a perspective division. The distance reference value is a reference value of the distance between two objects. The perspective division indicates whether the condition is met when the measured distance is farther than the distance reference value, or whether the condition is met when the measured distance is closer than the distance reference value. The distance reference value is also set with coordinate values ​​indicating the positions of detected object 1 and detected object 2. This position is information indicating which detected object is used to set the distance reference value.

[0022] Here, the operation unit 26 accepts editing of the detection pattern master 32 as necessary. For example, when a new state that is to be defined as an unsafe state occurs, the operation unit 26 accepts addition of a detection pattern indicating that state. The operation unit 26 may also accept changes to the definition of an existing detection pattern.

[0023] The result file 33 according to the first embodiment will be described with reference to FIG. Information regarding the detection of a target for which a warning is issued is set in the result file 33. The result file 33 includes a camera number, a detection date and time, a pattern number, a warning sound number, and a video file. The camera number is the camera number of the camera 50 that captured the video data of the detection source of the target for which a warning is issued. The detection date and time is the date and time when the target for which a warning is issued was detected. The pattern number is the pattern number of the detection pattern in which the target for which a warning is issued was detected. The warning audio number is the warning audio number of the warning audio corresponding to the detection pattern indicated by the pattern number. The video file is a video file of the video data in which the target for which a warning is issued was detected.

[0024] The flow of operations of the monitoring device 10 according to the first embodiment will be described with reference to FIG. The processes from step S101 to step S111 are executed for each camera 50 as the target camera 50.

[0025] (Step S101: Video acquisition process) The video acquisition unit 21 acquires image data that is a new frame of video data captured by the target camera 50 in accordance with the setting file 31 . Specifically, the video acquisition unit 21 sets the acquisition destination of the video data to either the target camera 50 or an external storage device, according to the video switching flag in the setting file 31. The video acquisition unit 21 acquires image data from the set acquisition destination of the video data.

[0026] (Step S102: Image storage process) The video acquisition unit 21 stores the image data acquired in step S101 in the storage 13 in accordance with the setting file 31. Specifically, when the conditions indicated by the video save pattern in the setting file 31 are met, the video acquisition unit 21 stores the image data in a destination folder indicated by the video save folder in the setting file 31. At this time, the video acquisition unit 21 stores the image data while dividing it into different video files for each time indicated by the video save time in the setting file 31. Furthermore, the video acquisition unit 21 stores the image data while thinning it out as necessary so as to achieve a frame rate indicated by the number of video save frames in the setting file 31.

[0027] (Step S103: Object information acquisition process) The object information acquisition unit 22 acquires object information indicating the type and position of one or more objects detected from the image data of the object, using the image data acquired in step S101 as the image data of the object. Specifically, the object information acquisition unit 22 inputs the image data of the target to the object detection model. The object detection model is a model that detects an object of a detection target type using AI technology. The object detection model is, for example, a model configured using CNN. CNN is an abbreviation for Convolutional Neural Network. The object detection model detects an object of a detection target type from the input image data, and outputs object information indicating the type and position of each detected object. The object detection model outputs a detection frame surrounding the object as the position. The object information acquisition unit 22 acquires object information indicating the type and position of each object output from the object detection model.

[0028] The processes from step S104 to step S111 are executed with one or more detection patterns corresponding to the target camera 50 in the detection pattern master 32 as target detection patterns.

[0029] (Step S104: Area object extraction process) The determination unit 23 determines whether or not the position of the object indicated by the object information acquired in step S103 is included within the area indicated by the area in the target detection pattern. The determination unit 23 extracts objects indicated by the object information that are included within the area indicated by the area in the target detection pattern. When the area in the target detection pattern has not been set, the determination unit 23 extracts all objects indicated by the object information acquired in step S103.

[0030] (Step S105: Object determination process) The determination unit 23 determines whether the combination of the object type and presence / absence indicated by the object information for the object extracted in step S104 matches the combination of the type and presence / absence class of each of one or more detected objects in the target detection pattern. If the combination of the object type and presence / absence indicated by the object information matches the combination of the detected object type and presence / absence classification in the target detection pattern, the determination unit 23 advances the process to step S106. On the other hand, if there is no match, the determination unit 23 ends the process for the target detection pattern.

[0031] Assume that the detection pattern of pattern number 1 shown in Fig. 7 is the target detection pattern. Two detection objects, detection object 1 and detection object 2, are set in the target detection pattern. The detection object 1 has a type of person and a presence / absence classification of Yes. The detection object 2 has a type of crane and a presence / absence classification of Yes. In this case, when the object information indicates an object of type human and the object information indicates an object of type crane, the determination unit 23 determines that the combination of the object type and presence / absence indicated by the object information matches the combination of the detection object type and presence / absence classification in the target detection pattern.

[0032] Assume that the detection pattern of pattern number 2 shown in FIG. 7 is the target detection pattern. Two detection objects, detection object 1 and detection object 2, are set in the target detection pattern. The type of detection object 1 is a helmet and the presence / absence classification is yes. The type of detection object 2 is a mask and the presence / absence classification is yes. In this case, when the object information indicates that there is an object of type helmet and the object information indicates that there is an object of type mask, the determination unit 23 determines that the combination of the object type and presence / absence indicated by the object information matches the combination of the detection object type and presence / absence classification in the target detection pattern.

[0033] Assume that the detection pattern of pattern number 3 shown in FIG. 7 is the target detection pattern. Two detection objects, detection object 1 and detection object 2, are set in the target detection pattern. The type of detection object 1 is a helmet, and the presence / absence classification is no. The type of detection object 2 is a person, and the presence / absence classification is yes. In this case, when there is no object of the type of helmet as an object indicated by the object information, and there is an object of the type of person as an object indicated by the object information, the determination unit 23 determines that the combination of the object type and presence / absence indicated by the object information matches the combination of the detection object type and presence / absence classification in the target detection pattern.

[0034] (Step S106: Measurement and determination process) The determination unit 23 determines whether or not a distance is set in the target detection pattern. If a distance in the target detection pattern has been set, the determination unit 23 instructs the measurement unit 24 to measure the distance, and proceeds to step S107. On the other hand, if a distance in the target detection pattern has not been set, the determination unit 23 determines that an unsafe state or the like has been detected, and proceeds to step S109. At this time, the determination unit 23 stores in the memory 12 the pattern number of the target detection pattern and the time when the unsafe state or the like was detected.

[0035] If the target detection pattern is the detection pattern of pattern numbers 1 and 2 shown in Fig. 7, a distance has been set, and so processing proceeds to step S107. On the other hand, if the target detection pattern is the detection pattern of pattern number 3 shown in Fig. 7, a distance has not been set, and so processing proceeds to step S109. That is, in the case of the detection pattern of pattern number 3, as shown in Fig. 8, a person has been detected without a helmet, that is, a person not wearing a helmet has been detected, and it is determined that an unsafe state has occurred.

[0036] (Step S107: Measurement process) The measurement unit 24 measures the distance between an object of the type of detection object 1 and an object of the type of detection object 2 in the target detection pattern. In addition, when at least one of the objects of the type of detection object 1 and the object of the type of detection object 2 exists in plural, the measurement unit 24 measures the distance for all combinations of the objects of the type of detection object 1 and the object of the type of detection object 2. The distance between the two objects is, for example, the distance between the positions of the detection frames of the two objects that are determined in advance. In the first embodiment, it is the distance between the upper left points of the detection frames of the two objects. In addition, the distance between the two objects may be the shortest distance between the detection frames of the two objects. In this case, the distance reference value of the detection pattern master 32 shown in FIG. 4 is set to use the shortest distance as the reference according to the detection frames of the two objects. Specifically, it is set in the detection pattern master 32 that 16 distances between the four corner points determined by the detection frame of one object and the four corner points determined by the detection frame of the other object are calculated, and the shortest distance among them is set as the calculation result in step S107. The method of measuring the distance will be described later.

[0037] (Step S108: Distance determination process) The determination unit 23 compares the distance measured in step S107 with the distance reference value in the target detection pattern to determine whether the condition of the far-distance division is satisfied. If the condition of the far-distance division is satisfied, the determination unit 23 determines that an unsafe state or the like has occurred. At this time, the determination unit 23 stores in the memory 12 the pattern number of the target detection pattern and the time when the unsafe state or the like was detected. Assume that the target detection pattern is the detection pattern with pattern number 1 shown in Fig. 7. In this case, as shown in Fig. 9, the determination unit 23 determines that the condition is satisfied when the distance measured in step S107 is closer than the distance reference value. In other words, when the distance between the crane and the person is close, the condition is satisfied and it is determined that an unsafe state has occurred. Assume that the target detection pattern is the detection pattern with pattern number 2 shown in Fig. 7. In this case, as shown in Fig. 10, the determination unit 23 determines that the condition is satisfied when the distance measured in step S107 is greater than the distance reference value. In other words, when the distance between the helmet and the mask is great, the condition is satisfied and it is determined that an unsafe state is occurring.

[0038] (Step S109: Threshold determination process) The determination unit 23 determines whether or not an unsafe state has occurred a threshold number of times or more in the configuration file 31 . Specifically, the determination unit 23 determines whether or not an unsafe state has been determined to have occurred for the target detection pattern a threshold number of times or more in the configuration file 31 within a past reference time period. If an unsafe state has occurred a threshold number of times or more in the configuration file 31, the determination unit 23 determines that the unsafe state has been confirmed, and proceeds to step S110. On the other hand, if an unsafe state has not occurred a threshold number of times or more in the configuration file 31, the determination unit 23 ends the process for the target detection pattern.

[0039] (Step S110: Result storage process) The determination unit 23 stores the detection result in a result file 33 . Specifically, the judgment unit 23 writes to the result file 33 the camera number of the target camera 50, the current date and time, the pattern number of the target detection pattern, the warning sound number of the warning sound corresponding to the target detection pattern, and the identification information of the video file in which the image data was stored in step S102.

[0040] (Step S111: Notification process) The notification unit 25 notifies workers or the like that an unsafe state has occurred. Specifically, the notification section 25 outputs an audio warning corresponding to the target detection pattern to the alarm device 60. This causes the alarm device 60 to light up and output an audio warning.

[0041] A method of measuring distance in the measurement process (step S107 in FIG. 6) in the first embodiment will be described with reference to FIG. First, the measurement unit 24 measures the distance between two objects in the target image data. The distance here is, for example, the number of pixels. In the first embodiment, the distance between two objects is the distance between the upper left points of the detection frames of the two objects. Therefore, in FIG. 11, the measurement unit 24 measures the number of pixels between the upper left points of the detection frames of the two objects. Here, it is assumed that the distance between the two objects is 40 pixels.

[0042] Next, the measurement unit 24 corrects the measured distance between the objects using the ratio between the size of the detected object and the reference size corresponding to the type of the detected object. Here, it is assumed that the reference size of the detected object 1 in the target detection pattern is 30 pixels wide and 40 pixels high. In contrast, it is assumed that the size of the detected object 1 is 40 pixels wide and 55 pixels high. First, the measurement unit 24 calculates the ratio as follows. (1) Diagonal distance of the reference size = √(30 2 +40 2 )=50 (2) The diagonal distance of the detected object size = √(40 2 +55 2 )=68.00... (3) Ratio=68.00 / 50=1.36... Next, the measurement unit 24 corrects the distance between the two objects in the image data of the target by the ratio. (4) Corrected distance = 40 / 1.36... = 29.41... Based on this calculation, 40 pixels are corrected to 29.41 pixels. In the distance determination process (step S108 in FIG. 6), the corrected distance is compared with a distance reference value to determine whether or not the perspective division conditions are met.

[0043] Here, the ratio is calculated from the reference size of the detected object 1 and the size of the detected object 1. However, the ratio may be calculated from the reference size of the detected object 2 and the size of the detected object 2. The ratio used for correction may be calculated using the ratio between the reference size of the detected object 1 and the size of the detected object 1, and the ratio between the reference size of the detected object 2 and the size of the detected object 2. For example, the average value of the ratio between the reference size of the detected object 1 and the size of the detected object 1, and the ratio between the reference size of the detected object 2 and the size of the detected object 2 may be used as the ratio used for correction.

[0044] ***Advantages of the First Embodiment*** As described above, the monitoring device 10 according to the first embodiment judges whether or not object information indicating the type of object for one or more objects detected from image data satisfies the detection pattern indicated by the editable detection pattern master 32. This judges whether or not there is an unsafe state, and judges whether or not to notify. The detection pattern master 32 is editable, and the user can change the detection pattern by editing it. Therefore, the detection conditions can be easily changed.

[0045] The monitoring device 10 according to the first embodiment defines a detection pattern that determines whether or not an unsafe state exists by combining the presence or absence of each type of object, the distance between objects of a specified type, and the area in which the object is detected. This makes it possible to set detection patterns that appropriately determine various situations.

[0046] The monitoring device 10 according to the first embodiment does not make a judgment using the distance in the image data of the target as it is, but corrects the distance in the image data of the target using the reference size of each type of object. This corrects the error in the distance due to the way the object is captured, making it possible to accurately judge whether or not the situation is unsafe.

[0047] The monitoring device 10 according to the first embodiment can switch the source of image data from the camera 50 to an external storage device. This allows the user to re-learn the object detection model using past image data. For example, if there is a new type of object to be detected, it is possible to have the object detection model learn the object so that it can detect that type of object. As a result, it is also possible to set a detection pattern using that type of object. When the monitoring device 10 acquires image data as real-time images from the camera 50, it is difficult to verify the determination of pattern detection, but when a video file is used, it is possible to check the validity of the detection pattern.

[0048] ***Other configurations*** <Variation 1> In the first embodiment, only the distance between the detected object 1 and the detected object 2 is set as a condition using distance. However, it is also possible to set a condition using distances between three or more detected objects. For example, it is also possible to set a condition using the distance between the detected object 1 and the detected object 2 and the distance between the detected object 2 and the detected object 3. As a specific example, the condition may be set based on whether the distance between detection object 1 and detection object 2 or the distance between detection object 2 and detection object 3 is longer. <Variation 2> In the first embodiment, the flow of the process in the management device 10 has been described with reference to FIG. 6. The threshold determination process in step S109 in FIG. 6 is a process for correctly determining whether or not an unsafe state has occurred, assuming that a certain number of false detections of the object detection model occur when acquiring object information in step S103. Step S109a may be added after step S109 and before step S110. In step S109a, the determination unit 23 may add a determination as to whether or not the time during which YES is obtained in step S109 is equal to or longer than a predetermined situation duration after the first end of the process in step S109. If YES is obtained in step S109a, it can be correctly determined that an unsafe state has occurred due to the situation duration. If NO is obtained in step S109a, it is determined that an unsafe state has not occurred due to the situation duration, and the process for the target detection pattern is terminated. If YES is obtained in step S109a, the process from step S110 onwards is as described in the first embodiment.

[0049] <Variation 3> In the first embodiment, each functional component is realized by software. However, as a third modification, each functional component may be realized by hardware. The following describes the third modification, focusing on the differences from the first embodiment.

[0050] When each functional component is realized by hardware, the monitoring device 10 includes an electronic circuit instead of the processor 11, the memory 12, and the storage 13. The electronic circuit is a dedicated circuit for realizing the functions of each functional component, the memory 12, and the storage 13.

[0051] The electronic circuits include single circuits, composite circuits, programmed processors, parallel programmed processors, logic ICs, GAs, ASICs, and FPGAs. GA stands for Gate Array. ASIC stands for Application Specific Integrated Circuit. FPGA stands for Field-Programmable Gate Array. Each functional component may be realized by one electronic circuit, or each functional component may be realized by distributing it among a plurality of electronic circuits.

[0052] <Variation 4> As a fourth modification, some of the functional components may be realized by hardware, and other functional components may be realized by software.

[0053] The processor 11, the memory 12, the storage 13, and the electronic circuitry are collectively referred to as a processing circuit. In other words, the functions of the functional components are realized by the processing circuit.

[0054] Furthermore, the term "part" in the above description may be read as a "circuit," "step," "procedure," "processing," or "processing circuit."

[0055] Embodiment 2 The second embodiment differs from the first embodiment in that a notification is made when a condition is satisfied simultaneously for a set of multiple detection patterns. In the second embodiment, this difference will be described, and a description of the same points will be omitted.

[0056] ***Configuration Description*** The configuration of a monitoring device 10 according to the second embodiment will be described with reference to FIG. The monitoring device 10 differs from the monitoring device 10 shown in FIG.

[0057] ***Explanation of Operation*** The operation of the monitoring device 10 according to the second embodiment will be described with reference to FIGS. An operation procedure of the monitoring device 10 according to the second embodiment corresponds to a monitoring method according to the second embodiment. Moreover, a program for realizing the operation of the monitoring device 10 according to the second embodiment corresponds to a monitoring program according to the second embodiment.

[0058] The associated pattern master 34 according to the second embodiment will be described with reference to FIG. In the associated pattern master 34, a plurality of detection patterns corresponding to different cameras 50 are grouped and set as associated patterns. 13, the detection pattern of pattern number 4 and the detection pattern of pattern number 5 are grouped as related patterns. As shown in Fig. 14, the detection pattern of pattern number 4 is a detection pattern corresponding to the camera 50 with the camera number 1. The detection pattern of pattern number 5 is a detection pattern corresponding to the camera 50 with the camera number 2.

[0059] The flow of operations of the monitoring device 10 according to the second embodiment will be described with reference to FIG. The processes from step S201 to step S209 are the same as the processes from step S101 to step S109 in Fig. 6. The processes from step S212 to step S213 are the same as the processes from step S110 to step S111 in Fig. 6.

[0060] (Step S210: Relatedness determination process) The determination unit 23 determines whether or not a related pattern exists for the target detection pattern. If the target detection pattern is grouped with other detection patterns, it is determined that the target detection pattern has a related pattern. If a related pattern exists in the target detection pattern, the determination unit 23 advances the process to step S211. On the other hand, if a related pattern does not exist in the target detection pattern, the determination unit 23 determines that an unsafe state has been confirmed, and advances the process to step S212.

[0061] (Step S211: Related pattern determination process) The determination unit 23 determines whether or not it has been determined in the process of step S209 that an unsafe state has occurred a threshold number of times or more in the configuration file 31 for all detection patterns in the same group as the target detection pattern. If it is determined that an unsafe state has occurred a threshold number of times or more for all detection patterns in the same group, the determination unit 23 determines that the unsafe state has been confirmed and proceeds to step S212. On the other hand, if this is not the case, the determination unit 23 ends the process for the target detection pattern.

[0062] When the determination unit 23 finishes processing for a target detection pattern, the determination unit 23 stores in the memory 12, as an associated pattern result, that it has been determined that an unsafe state has occurred a threshold number of times or more for the target detection pattern. By referring to the associated pattern result, the determination unit 23 can determine whether or not an unsafe state has been determined to have occurred a threshold number of times or more for other detection patterns in the same group. In order to prevent erroneous judgments from being made by referring to old related pattern results, the judgment unit 23 may delete related pattern results that are older than a certain period of time.

[0063] With reference to FIG. 16, an association pattern according to the second embodiment will be described. Multiple detection patterns corresponding to cameras 50 capturing images of the same area from different angles are grouped together as related patterns. As a specific example, a condition using the distance between two specified types of objects is indicated, and multiple detection patterns that indicate the same condition for the specified combination are grouped as related patterns. This is because it may be difficult to accurately determine the distance between two objects using only image data captured from one direction.

[0064] 16, the camera 50 with camera number 1 photographs the area X from the right side, and the camera 50 with camera number 2 photographs the area X from the front side. In the image data photographed by the camera 50 with camera number 2, the distance between the HP and the torch is close, but in the image data photographed by the camera 50 with camera number 1, the distance between the HP and the torch is far. HP is an abbreviation for Hand Planer. Assume that it is desired to determine that an unsafe state exists when the distance between the HP and the torch is close. In this case, if the determination is made based only on the image data captured by the camera 50 with camera number 2, the distance between the HP and the torch is close, and therefore the state is determined to be unsafe. However, in the image data captured by the camera 50 with camera number 1, which captures images from a different angle than the camera 50 with camera number 2, the distance between the HP and the torch is farther. In other words, although the HP and the torch appear to be close from the angle of the camera 50 with camera number 2, in reality there is a sufficient distance between the HP and the torch. Therefore, if the judgment is made based only on the image data taken by the camera 50 with camera number 2, the judgment will not be correct. Therefore, the detection pattern for the camera 50 with camera number 1 and the detection pattern for the camera 50 with camera number 2 are grouped as related patterns. This makes it possible to not only notify the image data taken by the camera 50 with camera number 2, but also to notify only when the image data taken by the camera 50 with camera number 1 is judged to be similar. As a result, erroneous judgments are less likely to be made.

[0065] As another specific example, a detection pattern for detecting a button of a specific color corresponding to a certain camera 50 and a detection pattern for detecting a specific device or the like corresponding to another camera 50 may be grouped as related patterns. This makes it possible to detect, for example, that an attempt is being made to perform a task despite the task being prohibited by a button of a specific color.

[0066] ***Effects of the second embodiment*** As described above, the monitoring device 10 according to the second embodiment issues a notification when a condition is satisfied simultaneously for a set of detection patterns corresponding to a plurality of cameras 50. This makes it possible to increase the accuracy of judgment and perform complex judgments.

[0067] ***Other configurations*** <Variation 5> In the second embodiment, the flow of the processing in the management device 10 has been described with reference to FIG. 15. In the processing in FIG. 15, as shown in the second modification of the first embodiment, after the processing for correctly determining whether or not an unsafe state occurs for the related pattern in step S211, step S211a may be added after step S211 and before step S212. In step S211a, the determination unit 23 may add a determination as to whether or not the time during which YES is obtained in step S211 is equal to or longer than a predetermined situation duration after the processing in step S211 is initially completed. If YES is obtained in step S211a, it can be correctly determined that an unsafe state has occurred even due to the situation duration. If NO is obtained in step S211a, it is determined that an unsafe state has not occurred due to the situation duration, and the processing for the target detection pattern is terminated. If YES is obtained in step S211a, the processing from step S212 onwards is as described in the second embodiment.

[0068] Embodiment 3 In the third embodiment, an editing screen for the detection pattern master 32, the associated pattern master 34, and the like will be described.

[0069] ***Explanation of Operation*** The operation of the monitoring device 10 according to the third embodiment will be described with reference to FIGS. An operation procedure of the monitoring device 10 according to the embodiment 3 corresponds to a monitoring method according to the embodiment 3. Moreover, a program for realizing the operation of the monitoring device 10 according to the embodiment 3 corresponds to a monitoring program according to the embodiment 3.

[0070] As shown in Fig. 17, the operation unit 26 displays a selection screen for work monitoring, detection pattern setting, and combination setting. Work monitoring is an item for viewing live images from each camera. Detection pattern setting is an item for editing the detection pattern master 32. Combination setting is an item for editing the associated pattern master 34.

[0071] With reference to FIG. 18, a work monitoring screen according to the third embodiment will be described. When work monitoring is selected on the selection screen, operation unit 26 displays the screen in the upper left of Fig. 18. On the upper left screen, images from multiple cameras 50 are displayed in small sizes. In Fig. 18, images from four cameras 50 with camera numbers 1 to 4 are displayed in small sizes. When the Next button is pressed, operation unit 26 displays the screen in the lower left. On the lower left screen, images from multiple cameras 50 subsequent to the camera 50 displayed on the upper left screen are displayed in small sizes. In Fig. 18, images from four cameras 50 with camera numbers 5 to 8 are displayed in small sizes. When the Next button is pressed again, images from multiple cameras 50 subsequent to the camera 50 displayed on the lower left screen are displayed in small sizes. On the other hand, when the Back button is pressed, the screen returns to the upper left screen. When the live display button for any of the cameras 50 is pressed while the images from the multiple cameras 50 are displayed small, as in the upper left or lower left screen, the image from the camera 50 corresponding to the live display button is enlarged. When the back button is pressed on the enlarged screen, the screen returns to the original screen on which the images from the multiple cameras 50 are displayed small. When the enlarged screen is clicked or otherwise manipulated, the enlarged screen is displayed in full screen and even larger. When a button such as Esc on the keyboard is pressed while the full screen is displayed, the screen returns to the enlarged screen.

[0072] When detection pattern setting is selected on the selection screen, operation unit 26 displays the detection pattern setting screen of Fig. 19. On the detection pattern setting screen, each detection pattern registered in detection pattern master 32 is displayed, and each item of each detection pattern can be edited. In addition, a new detection pattern can be added by pressing the Add button. In addition, an existing detection pattern can be deleted by pressing the Delete button. In the selection screen of FIG. 19, the distance to the object 1-2 is set as the length of the X axis×the length of the Y axis, the width, the height, and the distance from any one of the four corners of the rectangle.

[0073] Here, when setting the distance reference value, as shown in Fig. 20, the operation unit 26 displays image data when an actual danger is recognized. The image data to be displayed is selected by the user. Then, by performing a drag operation from the position of the detected object 1 to the position of the detected object 2, the distance from the position of the detected object 1 to the position of the detected object 2 is measured, and the measured distance is set as the distance reference value. Additionally, the positions of the detected object 1 and the detected object 2 are set. When editing the distance reference value, the operation unit 26 displays the set positions of the detected object 1 and the detected object 2 and the distance between them. Then, the distance reference value is edited by, for example, redoing the drag operation from the position of the detected object 1 to the position of the detected object 2.

[0074] When combination setting is selected on the selection screen, operation unit 26 displays the combination setting screen of FIG. 21. On the combination setting screen, the associated patterns registered in associated pattern master 34 are displayed for each camera 50 and each detection pattern. When the edit button for the detection pattern to be edited is pressed, operation unit 26 displays an edit screen shown in FIG. 22, which shows the current setting contents as an initial display. On the edit screen of FIG. 22, in addition to editing the associated pattern, it is also possible to edit the name of camera 50 and the URL to which camera 50 is connected. URL is an abbreviation for Uniform Resource Locator. When the add button is pressed on FIG. 21, operation unit 26 displays the edit screen of FIG. 22, which shows all fields as an initial display with all fields blank. Also, in the detection pattern setting screen shown in Fig. 19, close NG and far NG are displayed in the near / far section, but it can be used to detect not only unsafe states but also safe states. In that case, close OK and far OK can be selected, and the detection pattern master 34 shown in Fig. 4 also displays 3: close OK and 4: far OK. Also, a sound that makes the user aware of safety is selected for the warning sound number in the detection pattern master 34.

[0075] ***Effects of the Third Embodiment*** As described above, the monitoring device 10 according to the third embodiment can check the live video of each camera 50 and set the detection pattern master 32 and the related pattern master 34. This allows the monitoring device 10 to be appropriately operated.

[0076] Various aspects of the present disclosure are summarized below as appendices. (Appendix 1) an object information acquisition unit that acquires object information for one or more objects detected from the target image data; a determination unit that determines whether the object information acquired by the object information acquisition unit satisfies a detection pattern indicated by an editable detection pattern master; a notification unit that issues a notification when the determination unit determines that the detection pattern is satisfied; A monitoring device comprising: (Appendix 2) the object information indicates a type of the object, The detection pattern indicates a combination of conditions regarding whether or not a specified type of object is detected. 2. A monitoring device as described in claim 1. (Appendix 3) The object information indicates a type and a position of the object, The detection pattern indicates a condition using the distance between two specified types of objects. 3. The monitoring device according to claim 1 or 2. (Appendix 4) The detection pattern indicates a condition using an area in which an object is detected. 4. The monitoring device according to claim 1 , (Appendix 5) the object information acquisition unit acquires object information using image data acquired by each of a plurality of cameras as image data of the object; the detection patterns are provided corresponding to the plurality of cameras, The determination unit determines, with each of the plurality of cameras as a target camera, whether or not the object information for image data acquired by the target camera satisfies the detection pattern corresponding to the target camera. 5. The monitoring device according to claim 1 , (Appendix 6) Multiple detection patterns corresponding to different cameras are grouped together, The notification unit issues a notification when it is determined that the grouped detection patterns satisfy all of the detection patterns included in the same group. 6. A monitoring device as described in claim 5. (Appendix 7) Multiple detection patterns are grouped together, corresponding to cameras capturing the same area from different angles. 7. A monitoring device as described in claim 6. (Appendix 8) The same two types are specified, and multiple detection patterns that indicate conditions using the distance between the two specified types of objects are grouped together. 8. The monitoring device according to claim 7. (Appendix 9) The monitoring device further comprises: A measurement unit that measures a distance between two specified types of objects, the measurement unit measuring the distance between the objects by correcting the distance between the objects in the image data using a ratio between a size of the detected object and a reference size corresponding to the type of the detected object. 3. The monitoring device of claim 2, comprising: (Appendix 10) The monitoring device further comprises: An operation section for accepting editing of the detection pattern master 10. The monitoring device according to claim 1, further comprising: (Appendix 11) The computer obtains object information for one or more objects detected from the image data of the target; The computer determines whether or not the object information satisfies a detection pattern indicated by an editable detection pattern master; The monitoring method includes providing a notification when the computer is determined to satisfy the detection pattern. (Appendix 12) an object information acquisition process for acquiring object information for one or more objects detected from the target image data; a determination process for determining whether or not the object information acquired by the object information acquisition process satisfies a detection pattern indicated by an editable detection pattern master; a notification process for issuing a notification when it is determined by the determination process that the detection pattern is satisfied; A monitoring program that causes a computer to function as a monitoring device to:

[0077] The above describes the embodiments and modifications of the present disclosure. Some of these embodiments and modifications may be combined and implemented. Also, one or some of them may be partially implemented. Note that the present disclosure is not limited to the above embodiments and modifications, and various modifications are possible as necessary. [Explanation of symbols]

[0078] 100 surveillance system, 10 surveillance device, 11 processor, 12 memory, 13 storage, 14 communication interface, 21 video acquisition unit, 22 object information acquisition unit, 23 judgment unit, 24 measurement unit, 25 notification unit, 26 operation unit, 31 setting file, 32 detection pattern master, 33 result file, 34 related pattern master, 50 camera, 60 alarm.

Claims

1. an object information acquisition unit that acquires object information indicating a type of one or more objects detected from image data of the object, the image data being acquired by each of the plurality of cameras as object image data; a determination unit, in which the object information acquired by the object information acquisition unit is a detection pattern indicated by an editable detection pattern master, the detection patterns being provided corresponding to each of the plurality of cameras, and the determination unit, regarding each of the plurality of cameras as a target camera, determines whether or not the object information for image data acquired by the target camera satisfies the detection pattern corresponding to the target camera; a notification unit that performs notification when a plurality of detection patterns corresponding to different cameras and indicating conditions using different types of objects are grouped together, and the determination unit determines that all detection patterns included in the same group are satisfied a threshold number of times or more within a past reference time; A monitoring device comprising:

2. the object information indicates a type of the object, The detection pattern indicates a combination of conditions regarding whether or not a specified type of object is detected. The monitoring device according to claim 1.

3. The object information indicates a type and a position of the object, The detection pattern indicates a condition using the distance between two specified types of objects. The monitoring device according to claim 1.

4. The detection pattern indicates a condition using an area in which an object is detected. The monitoring device according to claim 1.

5. Multiple detection patterns are grouped together, corresponding to cameras capturing the same area from different angles. The monitoring device according to claim 1.

6. The monitoring device further comprises: A measurement unit that measures a distance between two specified types of objects, the measurement unit measuring the distance between the objects by correcting the distance between the objects in the image data using a ratio between a size of the detected object and a reference size corresponding to the type of the detected object. The monitoring device according to claim 2 .

7. The monitoring device further comprises: An operation section for accepting editing of the detection pattern master The monitoring device according to claim 1 .

8. a computer acquires object information indicating a type of one or more objects detected from the image data of the object, the image data being acquired by each of the plurality of cameras as image data of the object; the computer refers to the detection patterns, which are indicated by an editable detection pattern master, corresponding to each of the plurality of cameras, and determines whether or not the object information for image data acquired by each of the plurality of cameras corresponds to the detection pattern corresponding to the target camera, with each of the plurality of cameras being a target camera; A monitoring method in which a plurality of detection patterns corresponding to different cameras and indicating conditions using different types of objects are grouped together, and a computer issues a notification when it determines that all of the detection patterns included in the same group have been satisfied a threshold number of times or more within a past reference time.

9. an object information acquisition process for acquiring object information indicating a type of one or more objects detected from the image data of the object, the image data being taken as object image data of the object; a determination process in which the object information acquired by the object information acquisition process is a detection pattern indicated by an editable detection pattern master, the detection patterns being provided corresponding to each of the plurality of cameras, and in which the object information is determined as a target camera by referring to the detection patterns provided corresponding to each of the plurality of cameras, and the object information for image data acquired by the target camera satisfies the detection pattern corresponding to the target camera; A notification process is a process in which a plurality of detection patterns corresponding to different cameras and indicating conditions using different types of objects are grouped together, and when all the detection patterns included in the same group are satisfied by the determination process, a notification process is performed when the detection process has been performed a threshold number of times or more within a past reference time. A monitoring program that causes a computer to function as a monitoring device to:

Images