Monitoring device, monitoring method, and monitoring program

Abstract

To enable the user side to easily change a detection condition while detecting an unsafe state by utilizing an image analysis system and the like using AI.SOLUTION: An object information acquisition unit 22 acquires object information on one or more objects detected from target image data. A determination unit 23 determines whether or not the object information acquired by the object information acquisition unit 22 satisfies a detection pattern indicated by a detection pattern master provided to be editable. A notification unit 25 performs notification when it is determined by the determination unit 23 that the detection pattern is satisfied.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to a monitoring technique using image data.

Background Art

[0002] In places such as factories and construction sites, various devices are used, and if workers or the like do not pay attention, their safety may be threatened. In order to improve the safety of workers or the like, monitoring is performed by cameras, and warnings are given when an unsafe state or the like occurs.

[0003] Previously, people visually inspected the image data obtained by surveillance cameras to monitor whether an unsafe state or the like occurred. In recent years, it has been studied to analyze the image data obtained by surveillance cameras by an image analysis system using AI, and to give a warning when an unsafe state or the like is detected. AI is an abbreviation for Artificial Intelligence. For example, Patent Document 1 describes determining whether workers at a construction site are wearing equipment such as helmets and gloves by image analysis.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Unsafe conditions at places such as factories and construction sites change due to factors such as the introduction of new equipment, changes in the arrangement of equipment, and changes in safety standards. It is difficult for the user to change the detection conditions of an image analysis system using AI, and in order to change the detection conditions, it is necessary to request the vendor to modify the image analysis system. Modifying the image analysis system by the vendor takes time and costs, and is inconvenient for the user. An object of the present disclosure is to enable easy change of detection conditions on the user side while detecting an unsafe state using an image analysis system using AI or the like.

Means for Solving the Problems

[0006] A monitoring device according to the present disclosure An object information acquisition unit that acquires object information about one or more objects detected from target image data, A determination unit that determines whether or not the object information acquired by the object information acquisition unit satisfies a detection pattern indicated by an editable detection pattern master, A notification unit that performs notification when it is determined by the determination unit that the detection pattern is satisfied is provided.

Effects of the Invention

[0007] In the present disclosure, it is determined whether or not object information about one or more objects detected from image data satisfies a detection pattern indicated by an editable detection pattern master. Thereby, it is determined whether or not it is an unsafe state or the like, and it is determined whether or not to notify. The detection pattern master is provided so as to be editable, and the detection pattern can be changed by editing on the user side. Therefore, the detection conditions can be easily changed.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Embodiments for Carrying Out the Invention

[0009] Embodiment 1. ***Description of the Configuration*** With reference to FIG. 1, the configuration of the monitoring system 100 according to Embodiment 1 will be described. The monitoring system 100 includes a monitoring device 10, a plurality of cameras 50, and one or more alarm devices 60. The monitoring device 10 is connected to each camera 50 and each alarm device 60 via a network 70. The monitoring device 10 is a computer that detects an unsafe state or the like and issues a notification. Each camera 50 is a monitoring camera installed to photograph a monitoring target area. The alarm device 60 is a device for notifying workers or the like of an unsafe state or the like by sound and light.

[0010] With reference to FIG. 2, the configuration of the monitoring device 10 according to Embodiment 1 will be described. The monitoring device 10 is a computer. The monitoring device 10 includes hardware such as a processor 11, a memory 12, a storage 13, and a communication interface 14. The processor 11 is connected to other hardware via signal lines and controls these other hardware.

[0011] The processor 11 is an IC that performs processing. IC is an abbreviation for Integrated Circuit. As specific examples, the processor 11 is a CPU, a DSP, or a GPU. CPU is an abbreviation for Central Processing Unit. DSP is an abbreviation for Digital Signal Processor. GPU is an abbreviation for Graphics Processing Unit.

[0012] Memory 12 is a storage device that temporarily stores data. As specific examples, Memory 12 is SRAM or DRAM. SRAM is the abbreviation of Static Random Access Memory. DRAM is the abbreviation of Dynamic Random Access Memory.

[0013] Storage 13 is a storage device that stores data. As specific examples, Storage 13 is an SSD or an HDD. SSD is the abbreviation of Solid State Drive. HDD is the abbreviation of Hard Disk Drive. Also, Storage 13 may be a portable recording medium such as an SD (registered trademark) memory card, CompactFlash (registered trademark), NAND flash, flexible disk, optical disk, compact disk, Blu-ray (registered trademark) disk, or DVD. SD is the abbreviation of Secure Digital. DVD is the abbreviation of Digital Versatile Disk.

[0014] Communication interface 14 is an interface for communicating with an external device. As specific examples, Communication interface 14 is a port for Ethernet (registered trademark), USB, or HDMI (registered trademark). USB is the abbreviation of Universal Serial Bus. HDMI is the abbreviation of High-Definition Multimedia Interface.

[0015] Surveillance device 10 includes, as functional components, a video acquisition unit 21, an object information acquisition unit 22, a determination unit 23, a measurement unit 24, a notification unit 25, and an operation unit 26. The functions of each functional component of Surveillance device 10 are realized by software. The storage 13 stores a program that realizes the functions of each functional component of the surveillance device 10. This program is read into the memory 12 by the processor 11 and executed by the processor 11. Thereby, the functions of each functional component of the surveillance device 10 are realized.

[0016] The storage 13 stores a setting file 31, a detection pattern master 32, and a result file 33.

[0017] In FIG. 2, only one processor 11 was shown. However, there may be a plurality of processors 11, and a plurality of processors 11 may execute in cooperation a program for realizing each function.

[0018] ***Description of Operations*** With reference to FIGS. 3 to 11, the operations of the monitoring device 10 according to Embodiment 1 will be described. The operation procedure of the monitoring device 10 according to Embodiment 1 corresponds to the monitoring method according to Embodiment 1. Also, a program for realizing the operations of the monitoring device 10 according to Embodiment 1 corresponds to the monitoring program according to Embodiment 1.

[0019] With reference to FIG. 3, the setting file 31 according to Embodiment 1 will be described. Basic information of the monitoring device 10 is set in the setting file 31. The setting file 31 includes a video switching flag, a threshold number, a video storage folder, a video storage pattern, a video storage time, a video storage period, a video storage frame number, the number of cameras, setting information, a warning sound, and a reference size. The video switching flag is a flag for switching the acquisition source of video data between the camera 50 and an external storage device. The threshold number of times is the number of times serving as a criterion for determining an unsafe state. The video storage folder is the folder for the storage destination of video data. The video storage pattern is the condition for storing video data. The video storage time is the time of video data stored as one video file. The video storage period is the period for leaving the stored video data. The video storage frame rate is the frame rate of the video data to be stored. The number of cameras is the number of cameras 50 included in the monitoring system 100. The setting information is set for each camera 50 with a video file name which is identification information of the file of the video data of the camera 50, a camera No., a camera name, and a connection URL which is the connection destination of the camera 50. The warning voice is the voice output when it is determined to be unsafe or the like. One or more warning voices are set and warning voice numbers are assigned. The reference size is the size of a reference object and is used for the measurement of distance described later. The reference size is set for each type of object.

[0020] Referring to FIG. 4, the detection pattern master 32 according to Embodiment 1 will be described. One or more detection patterns indicating conditions for detecting a target for warning are set in the detection pattern master 32. Each detection pattern includes a pattern number, a camera number, a warning voice number, and a determination condition. The pattern number is the identification number of the detection pattern. The camera number is the identification number of the camera 50 corresponding to the detection pattern. That is, the detection pattern is set for each camera 50. The warning voice number is the identification number of the warning voice corresponding to the detection pattern. The determination condition is the condition for determining whether or not it corresponds to the detection pattern. A plurality of detection pattern masters 32 can be set for each camera 50. Also, one detection pattern master 32 can be shared by a plurality of cameras 50. The association between the camera 50 and the detection pattern master 32 can be one-to-one, one-to-many, many-to-one, or many-to-many.

[0021] The determination condition includes one or more detected objects, an area, and a distance. Each detected object is a condition regarding the object to be detected. Each detected object includes the type of the object to be detected and the presence / absence classification of the detection of the object to be detected. The type of the object indicates what the object to be detected is. The types of the object are set, for example, as types such as a person, a mask, and a crane. The presence / absence classification indicates whether the condition is satisfied when detected or whether the condition is satisfied when not detected. The area is a condition regarding the area where the object to be detected is detected. Whether the condition is satisfied is determined based on the presence / absence of detection in the area indicated by the area. When the area is not set, whether the condition is satisfied is determined based on the presence / absence of detection in the entire imaging area of the camera 50. The distance is a condition regarding the distance between two objects. The distance includes a distance reference value and a far / near classification. The distance reference value is the reference value of the distance between two objects. The far / near classification indicates whether the condition is satisfied when the measured distance is farther than the distance reference value or whether the condition is satisfied when the measured distance is closer than the distance reference value. Note that the distance reference value is also set together with the coordinate values indicating the positions of the detected object 1 and the detected object 2. This position is information for indicating which position of the detected object the distance reference value is set for.

[0022] Here, the operation unit 26 accepts the editing of the detection pattern master 32 as necessary. For example, when a new state that is desired to be defined as an unsafe state is created, the operation unit 26 accepts the addition of a detection pattern indicating that state. Further, the operation unit 26 may accept the change of the definition of an existing detection pattern.

[0023] Referring to FIG. 5, the result file 33 according to the first embodiment will be described. In the result file 33, information is set when an object to be warned about is detected. The result file 33 includes a camera number, a detection date and time, a pattern number, a warning sound number, and a video file. The camera number is the camera number of the camera 50 that captured the video data of the detection source for which a warning is to be issued. The detection date and time is the date and time when the object for which a warning is to be issued was detected. The pattern number is the pattern number of the detection pattern that detected the object for which a warning is to be issued. The warning voice number is the warning voice number of the warning voice corresponding to the detection pattern indicated by the pattern number. The video file is the video file of the video data in which the object for which a warning is to be issued was detected.

[0024] Referring to FIG. 6, the operation flow of the monitoring device 10 according to Embodiment 1 will be described. Taking each camera 50 as the target camera 50, the processes from step S101 to step S111 are executed.

[0025] (Step S101: Video acquisition process) The video acquisition unit 21 acquires image data, which is a new frame of the video data captured by the target camera 50, according to the setting file 31. Specifically, the video acquisition unit 21 sets the acquisition destination of the video data to either the target camera 50 or an external storage device according to the video switching flag in the setting file 31. The video acquisition unit 21 acquires image data from the set acquisition destination of the video data.

[0026] (Step S102: Image storage process) The video acquisition unit 21 stores the image data acquired in step S101 in the storage 13 according to the setting file 31. Specifically, when the conditions indicated by the video saving pattern in the setting file 31 are met, the video acquisition unit 21 stores the image data in the destination folder indicated by the video saving folder in the setting file 31. At this time, the video acquisition unit 21 stores the image data while dividing it into different video files every time period indicated by the video saving time in the setting file 31. Also, the video acquisition unit 21 stores the image data while thinning it out as necessary so that the frame rate indicated by the number of video saving frames in the setting file 31 is achieved.

[0027] (Step S103: Object information acquisition process) The object information acquisition unit 22 uses the image data acquired in step S101 as the target image data, and acquires object information indicating the type and position of one or more objects detected from the target image data for each of the objects. Specifically, the object information acquisition unit 22 inputs the target image data into an object detection model. The object detection model is a model that detects objects of a type to be detected using AI technology. The object detection model is, for example, a model configured using a CNN. CNN is an abbreviation for Convolutional Neural Network. The object detection model detects objects of the type to be detected from the input image data, and outputs object information indicating the type and position for each detected object. The object detection model outputs a detection frame surrounding the object as the position. The object information acquisition unit 22 acquires object information indicating the type and position for each object output from the object detection model.

[0028] For each of one or more detection patterns corresponding to the target camera 50 in the detection pattern master 32, the processes from step S104 to step S111 are executed with each as the target detection pattern.

[0029] (Step S104: Region object extraction process) The determination unit 23 determines whether or not the position of the object indicated by the object information acquired in step S103 is included in the region indicated by the area in the target detection pattern for the object indicated by the object information. The determination unit 23 extracts the objects among the objects indicated by the object information whose positions are included in the region indicated by the area in the target detection pattern. Note that when the area in the target detection pattern is not set, the determination unit 23 extracts all the objects indicated by the object information acquired in step S103.

[0030] (Step S105: Object determination process) The determination unit 23 determines whether or not the combination of the type and presence / absence of the object indicated by the object information for the object extracted in step S104 matches the combination of the type and presence / absence classification of each of one or more detected objects in the target detection pattern. When the combination of the type and presence / absence of the object indicated by the object information matches the combination of the type and presence / absence classification of the detected object in the target detection pattern, the determination unit 23 proceeds to step S106. On the other hand, when there is no match, the determination unit 23 ends the process for the target detection pattern.

[0031] Suppose the detection pattern of pattern number 1 shown in FIG. 7 is the target detection pattern. Two detected objects, detected object 1 and detected object 2, are set in the target detection pattern. Detected object 1 is of the type "person" and the presence / absence classification is "present". Detected object 2 is of the type "crane" and the presence / absence classification is "present". In this case, the determination unit 23 determines that when there is an object of the type "person" as the object indicated by the object information and there is an object of the type "crane" as the object indicated by the object information, the combination of the type and presence / absence of the object indicated by the object information matches the combination of the type and presence / absence classification of the detected object in the target detection pattern.

[0032] Suppose the detection pattern of pattern number 2 shown in FIG. 7 is the target detection pattern. Two detected objects, detected object 1 and detected object 2, are set in the target detection pattern. Detected object 1 is of the type "helmet" and the presence / absence classification is "present". Detected object 2 is of the type "mask" and the presence / absence classification is "present". In this case, the determination unit 23 determines that when there is an object of the type "helmet" as the object indicated by the object information and there is an object of the type "mask" as the object indicated by the object information, the combination of the type and presence / absence of the object indicated by the object information matches the combination of the type and presence / absence classification of the detected object in the target detection pattern.

[0033] Assume that the detection pattern with pattern number 3 shown in FIG. 7 is the target detection pattern. Two detection objects, detection object 1 and detection object 2, are set for the target detection pattern. Detection object 1 is a helmet in terms of type and is not present in terms of presence / absence classification. Detection object 2 is a person in terms of type and is present in terms of presence / absence classification. In this case, when there is no object of the type helmet as the object indicated by the object information and there is an object of the type person as the object indicated by the object information, the determination unit 23 determines that the combination of the type and presence / absence of the object indicated by the object information matches the combination of the type and presence / absence classification of the detection objects in the target detection pattern.

[0034] (Step S106: Measurement determination process) The determination unit 23 determines whether the distance in the target detection pattern is set. When the distance in the target detection pattern is set, the determination unit 23 instructs the measurement unit 24 to measure the distance and advances the process to step S107. On the other hand, when the distance in the target detection pattern is not set, the determination unit 23 advances the process to step S109 assuming that an unsafe state or the like has been detected. At this time, the determination unit 23 stores the pattern number of the target detection pattern and the time when an unsafe state or the like was detected in the memory 12.

[0035] When the target detection pattern is the detection patterns with pattern numbers 1 and 2 shown in FIG. 7, since the distance is set, the process advances to step S107. On the other hand, when the target detection pattern is the detection pattern with pattern number 3 shown in FIG. 7, since the distance is not set, the process advances to step S109. That is, in the case of the detection pattern with pattern number 3, as shown in FIG. 8, it is determined that an unsafe state has occurred because no helmet was detected and a person was detected, that is, a person not wearing a helmet was detected.

[0036] (Step S107: Measurement process) The measurement unit 24 measures the distance between an object of the type of detection object 1 and an object of the type of detection object 2 in the target detection pattern. When there are a plurality of at least either the object of the type of detection object 1 or the object of the type of detection object 2, the measurement unit 24 measures the distance for all combinations of the object of the type of detection object 1 and the object of the type of detection object 2. The distance between two objects is, for example, the distance between predetermined positions of the detection frames of the two objects respectively. In Embodiment 1, it is the distance between the upper left points of the detection frames of the two objects respectively. Note that the distance between two objects may be the shortest distance between the detection frames of the two objects. In this case, it is set in the setting of the distance reference value of the detection pattern master 32 shown in FIG. 4 that the shortest distance is used as a reference by the detection frames of the two objects. Specifically, it is determined in the detection pattern master 32 that the distances between the four corner points determined by the detection frame of one object and the four corner points determined by the detection frame of the other object are calculated 16 times, and the shortest distance among them is used as the calculation result in step S107. The method for measuring the distance will be described later.

[0037] (Step S108: Distance determination process) The determination unit 23 compares the distance measured in step S107 with the distance reference value in the target detection pattern, and determines whether the condition for the far - near classification is satisfied. When the determination unit 23 satisfies the condition for the far - near classification, it determines that an unsafe state or the like has occurred. At this time, the determination unit 23 stores the pattern number of the target detection pattern and the time when an unsafe state or the like is detected in the memory 12. Suppose that the detection pattern with pattern number 1 shown in FIG. 7 is the target detection pattern. In this case, as shown in FIG. 9, when the distance measured in step S107 is closer than the distance reference value, the determination unit 23 determines that the condition is satisfied. That is, when the distance between the crane and the person is close, the condition is satisfied, and it is determined that an unsafe state has occurred. Assume that the detection pattern with pattern number 2 shown in Fig. 7 is the target detection pattern. In this case, as shown in Fig. 10, when the distance measured in step S107 is farther than the distance reference value, the determination unit 23 determines that the condition is satisfied. That is, when the distance between the helmet and the mask is far, it is determined that the condition is satisfied and an unsafe state has occurred.

[0038] (Step S109: Threshold determination process) The determination unit 23 determines whether an unsafe state has occurred for more than the threshold number of times in the setting file 31. Specifically, the determination unit 23 determines whether, within the past reference time, it has been determined that an unsafe state has occurred for more than the threshold number of times for the target detection pattern in the setting file 31. If the determination unit 23 determines that an unsafe state has occurred for more than the threshold number of times in the setting file 31, it is considered that the unsafe state is confirmed, and the process proceeds to step S110. On the other hand, if the determination unit 23 determines that an unsafe state has not occurred for more than the threshold number of times in the setting file 31, the process for the target detection pattern ends.

[0039] (Step S110: Result storage process) The determination unit 23 stores the detection result in the result file 33. Specifically, the determination unit 23 writes the camera number of the target camera 50, the current date and time, the pattern number of the target detection pattern, the warning voice number of the warning voice corresponding to the target detection pattern, and the identification information of the video file where the image data was stored in step S102 into the result file 33.

[0040] (Step S111: Notification process) The notification unit 25 notifies workers, etc. that an unsafe state has occurred. Specifically, the notification unit 25 outputs the warning voice corresponding to the target detection pattern to the alarm device 60. Thereby, the alarm device 60 is lit and a warning voice is output from the alarm device 60.

[0041] Referring to FIG. 11, a method for measuring the distance in the measurement process (step S107 in FIG. 6) in Embodiment 1 will be described. First, the measurement unit 24 measures the distance between two objects in the target image data. The distance here is, for example, the number of pixels. In Embodiment 1, the distance between two objects is the distance between the upper left points of the detection frames of the two objects respectively. Therefore, in FIG. 11, the measurement unit 24 measures the number of pixels between the upper left points of the detection frames of the two objects respectively. Here, it is assumed that the distance between the two objects is 40 pixels.

[0042] Next, the measurement unit 24 corrects the measured distance between the objects by using the ratio of the size of the detected object to the reference size corresponding to the type of the detected object. Here, it is assumed that the reference size of the detected object 1 in the detection pattern of the target is 30 pixels in width and 40 pixels in height. On the contrary, it is assumed that the size of the detected object 1 is 40 pixels in width and 55 pixels in height. First, the measurement unit 24 calculates the ratio as follows. (1) Diagonal distance of the reference size = √(30 2 + 40 2 ) = 50 (2) Diagonal distance of the detected object size = √(40 2 + 55 2 ) = 68.00··· (3) Ratio = 68.00 / 50 = 1.36··· Next, the measurement unit 24 corrects the distance between the two objects in the target image data by the ratio. (4) Corrected distance = 40 / 1.36··· = 29.41··· Based on such calculation, 40 pixels are corrected to 29.41 pixels. In the distance determination process (step S108 in FIG. 6), the corrected distance and the distance reference value are compared, and it is determined whether the conditions for the far - near classification are satisfied.

[0043] Here, the ratio was calculated from the reference size of the detected object 1 and the size of the detected object 1. However, the ratio may also be calculated from the reference size of the detected object 2 and the size of the detected object 2. Further, the ratio used for correction may be calculated using the ratio between the reference size of the detected object 1 and the size of the detected object 1 and the ratio between the reference size of the detected object 2 and the size of the detected object 2. For example, the average value of the ratio between the reference size of the detected object 1 and the size of the detected object 1 and the ratio between the reference size of the detected object 2 and the size of the detected object 2 may be used as the ratio for correction.

[0044] ***Effects of Embodiment 1*** As described above, the monitoring device 10 according to Embodiment 1 determines whether or not the object information indicating the type of one or more objects detected from the image data satisfies the detection pattern indicated by the detection pattern master 32 provided so as to be editable. Thereby, it is determined whether or not it is an unsafe state or the like, and it is determined whether or not to notify. The detection pattern master 32 is provided so as to be editable, and the detection pattern can be changed by editing on the user side. Therefore, the detection conditions can be easily changed.

[0045] The monitoring device 10 according to Embodiment 1 defines a detection pattern for determining whether or not it is an unsafe state or the like by combining the presence or absence of objects for each type, the distance between objects of a specified type, and the area where the objects are detected. Thereby, it is possible to set a detection pattern for appropriately determining various situations.

[0046] The monitoring device 10 according to Embodiment 1 does not directly use the distance in the target image data for determination, but corrects the distance in the target image data using the reference size of the object for each type. Thereby, the error in the distance due to the way the object is imaged is corrected, and it is possible to accurately determine whether or not it is an unsafe state or the like.

[0047] The monitoring device 10 according to Embodiment 1 can switch the acquisition source of video data from the camera 50 to an external storage device. As a result, it is possible for the user to perform retraining of the object detection model using past video data. For example, when there is a type of object that one wants to newly detect, it is also possible to train the object detection model so that the object of that type can be detected. As a result, it is also possible to set a detection pattern using the object of that type. When the monitoring device 10 acquires video data as real-time video from the camera 50, it is difficult to verify the determination of pattern detection, but when using a video file, it is possible to confirm the validity of the detection pattern and the like.

[0048] ***Other configurations*** <Modification Example 1> In Embodiment 1, only the distance between Detection Object 1 and Detection Object 2 was set as the condition using the distance. However, conditions may be set using the distances between three or more detection objects. For example, conditions may be set using the distance between Detection Object 1 and Detection Object 2 and the distance between Detection Object 2 and Detection Object 3. As a specific example, conditions may be set according to which of the distance between Detection Object 1 and Detection Object 2 and the distance between Detection Object 2 and Detection Object 3 is longer. <Modification Example 2> In Embodiment 1, the processing flow in the management device 10 was described with reference to FIG. 6. Regarding the threshold determination process in step S109 in FIG. 6, it is a process for correctly determining whether a non-safe state has occurred, assuming that misdetection of the object detection model occurs a certain number of times when acquiring object information in step S103. After step S109 and before step S110, step S109a may be added. In step S109a, the determination unit 23 may first add a determination as to whether the time when it becomes YES in step S109 is equal to or longer than a predetermined situation continuation time after the process of step S109 is completed. If it becomes YES in step S109a, it can be correctly determined that a non-safe state has occurred also based on the situation continuation time. If it becomes NO in step S109a, it is determined that a non-safe state has not occurred based on the situation continuation time, and the processing for the target detection pattern ends. If it becomes YES in step S109a, the processing after step S110 is as described in Embodiment 1.

[0049] <Modification Example 3> In Embodiment 1, each functional component was realized by software. However, as Modification Example 3, each functional component may be realized by hardware. The differences from Embodiment 1 will be described for this Modification Example 3.

[0050] When each functional component is realized by hardware, the monitoring device 10 includes an electronic circuit instead of the processor 11, the memory 12, and the storage 13. The electronic circuit is a dedicated circuit that realizes the functions of each functional component, the memory 12, and the storage 13.

[0051] As the electronic circuit, a single circuit, a composite circuit, a programmed processor, a parallel-programmed processor, a logic IC, a GA, an ASIC, or an FPGA is assumed. GA is an abbreviation for Gate Array. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field-Programmable Gate Array. Each functional component may be implemented by a single electronic circuit, or each functional component may be implemented by being distributed across a plurality of electronic circuits.

[0052] <Modification Example 4> As Modification Example 4, some of each functional component may be implemented by hardware, and the other each functional component may be implemented by software.

[0053] The processor 11, the memory 12, the storage 13, and the electronic circuit are referred to as a processing circuit. That is, the functions of each functional component are realized by the processing circuit.

[0054] Also, the "section" in the above description may be read as "circuit", "step", "procedure", "process", or "processing circuit".

[0055] Embodiment 2. Embodiment 2 is different from Embodiment 1 in that a notification is made when conditions are satisfied simultaneously for a set of a plurality of detection patterns. In Embodiment 2, this different point will be described, and the description of the same points will be omitted.

[0056] ***Description of Configuration*** Referring to FIG. 12, the configuration of the monitoring device 10 according to Embodiment 2 will be described. The monitoring device 10 is different from the monitoring device 10 shown in FIG. 2 in that the related pattern master 34 is stored in the storage 13.

[0057] ***Description of Operation*** Referring to FIGS. 13 to 16, the operation of the monitoring device 10 according to Embodiment 2 will be described. The operation procedure of the monitoring device 10 according to Embodiment 2 corresponds to the monitoring method according to Embodiment 2. Also, the program for realizing the operation of the monitoring device 10 according to Embodiment 2 corresponds to the monitoring program according to Embodiment 2.

[0058] Referring to FIG. 13, the related pattern master 34 according to Embodiment 2 will be described. In the related pattern master 34, a plurality of detection patterns corresponding to different cameras 50 are grouped and set as related patterns. In FIG. 13, the detection pattern of pattern number 4 and the detection pattern of pattern number 5 are grouped as related patterns. As shown in FIG. 14, the detection pattern of pattern number 4 is the detection pattern corresponding to the camera 50 of camera number 1. The detection pattern of pattern number 5 is the detection pattern corresponding to the camera 50 of camera number 2.

[0059] Referring to FIG. 15, the operation flow of the monitoring device 10 according to Embodiment 2 will be described. The processing from step S201 to step S209 is the same as the processing from step S101 to step S109 in FIG. 6. The processing from step S212 to step S213 is the same as the processing from step S110 to step S111 in FIG. 6.

[0060] (Step S210: Related presence determination process) The determination unit 23 determines whether a related pattern exists for the target detection pattern. If the target detection pattern is grouped with other detection patterns, a related pattern exists for the target detection pattern. If the determination unit 23 determines that a related pattern exists for the target detection pattern, the process proceeds to step S211. On the other hand, if the determination unit 23 determines that no related pattern exists for the target detection pattern, it is assumed that an unsafe state has been determined, and the process proceeds to step S212.

[0061] (Step S211: Related pattern determination process) The determination unit 23 determines whether, for all the detection patterns in the same group as the target detection pattern, it has been determined that an unsafe state has occurred more than the threshold number of times in the setting file 31 in the process of step S209. When the determination unit 23 determines that an unsafe state has occurred for a threshold number of times or more for all the detection patterns in the same group, it proceeds to step S212 on the assumption that the unsafe state has been confirmed. On the other hand, if this is not the case, the determination unit 23 ends the process for the target detection pattern.

[0062] When ending the process for the target detection pattern, the determination unit 23 stores in the memory 12, as a related pattern result, that for the target detection pattern, it has been determined that an unsafe state has occurred for a threshold number of times or more. By referring to the related pattern result, the determination unit 23 can determine whether or not it has been determined that an unsafe state has occurred for a threshold number of times or more for other detection patterns in the same group. Note that, in order to prevent making an incorrect determination by referring to an old related pattern result, the determination unit 23 may delete the related pattern result for which a certain period of time has elapsed.

[0063] Referring to FIG. 16, the related patterns according to Embodiment 2 will be described. A plurality of detection patterns corresponding to the cameras 50 that photograph the same area from different angles are grouped as related patterns. As a specific example, conditions using the distance between two specified types of objects are shown, and a plurality of detection patterns in which the specified combination shows the same conditions are grouped as related patterns. This is because it may be difficult to accurately determine the distance between two objects using only the image data photographed from one direction.

[0064] As shown in FIG. 16, it is assumed that the camera 50 with camera number 1 photographs area X from the right side, and the camera 50 with camera number 2 photographs area X from the front side. In the image data photographed by the camera 50 with camera number 2, the distance between the HP and the torch is short, but in the image data photographed by the camera 50 with camera number 1, the distance between the HP and the torch is long. HP is an abbreviation for Hand Planer. If the distance between the HP and the torch is close, it is desired to determine that it is in an unsafe state. In this case, if the determination is made only based on the image data captured by the camera 50 of camera number 2, since the distance between the HP and the torch is close, it is determined that it is in an unsafe state. However, in the image data captured by the camera 50 of camera number 1, which is taken from an angle different from that of the camera 50 of camera number 2, the distance between the HP and the torch is far. That is, although the HP and the torch appear to be close from the angle of the camera 50 of camera number 2, in reality, there is a sufficient distance between the HP and the torch. Therefore, if the determination is made only based on the image data captured by the camera 50 of camera number 2, the correct determination cannot be made. Thus, the detection pattern for the camera 50 of camera number 1 and the detection pattern for the camera 50 of camera number 2 are grouped as related patterns. As a result, a notification will be issued only when it is determined that it is close not only based on the image data captured by the camera 50 of camera number 2 but also based on the image data captured by the camera 50 of camera number 1. Consequently, it becomes difficult to make an incorrect determination.

[0065] As another specific example, it is conceivable to group the detection pattern for detecting a button of a specific color corresponding to a certain camera 50 and the detection pattern for detecting a specific device or the like corresponding to another camera 50 as related patterns. Thereby, for example, it becomes possible to detect that an operation is being attempted even though the operation is prohibited by a button of a specific color.

[0066] ***Effect of Embodiment 2*** As described above, the monitoring device 10 according to Embodiment 2 issues a notification when the conditions are simultaneously satisfied for a set of detection patterns corresponding to a plurality of cameras 50. Thereby, it becomes possible to improve the accuracy of determination and to make a complex determination.

[0067] ***Other Configurations*** <Modification Example 5> In Embodiment 2, the processing flow in the management device 10 was described with reference to FIG. 15. Also in the processing shown in FIG. 15, as shown in Modification Example 2 of Embodiment 1, after performing processing for correctly determining whether or not an unsafe state has occurred with respect to the related pattern in step S211, step S211a may be added after step S211 and before step S212. In step S211a, after the processing in step S211 is first completed, the determination unit 23 may add a determination as to whether or not the time that becomes YES in step S211 is equal to or longer than a predetermined situation continuation time. If it becomes YES in step S211a, it is possible to correctly determine that an unsafe state has occurred also due to the situation continuation time. If it becomes NO in step S211a, it is determined that an unsafe state has not occurred due to the situation continuation time, and the processing for the target detection pattern ends. If it becomes YES in step S211a, the processing after step S212 is as described in Embodiment 2.

[0068] Embodiment 3. In Embodiment 3, an editing screen such as the detection pattern master 32 and the related pattern master 34 will be described.

[0069] ***Description of Operations*** With reference to FIGS. 17 to 22, the operations of the monitoring device 10 according to Embodiment 3 will be described. The operation procedure of the monitoring device 10 according to Embodiment 3 corresponds to the monitoring method according to Embodiment 3. Also, the program for realizing the operations of the monitoring device 10 according to Embodiment 3 corresponds to the monitoring program according to Embodiment 3.

[0070] As shown in FIG. 17, the operation unit 26 displays a selection screen for work monitoring, detection pattern setting, and combination setting. Work monitoring is an item for viewing the live video of each camera. Detection pattern setting is an item for editing the detection pattern master 32. Combination setting is an item for editing the related pattern master 34.

[0071] With reference to FIG. 18, the work monitoring screen according to Embodiment 3 will be described. When the operation unit 26 selects operation monitoring on the selection screen, it displays the screen in the upper left corner of FIG. 18. On the upper left screen, the images of a plurality of cameras 50 are displayed small. In FIG. 18, the images of four cameras 50 with camera numbers from 1 to 4 are displayed small. When the Next button is pressed, the operation unit 26 displays the lower left screen. On the lower left screen, the images of a plurality of cameras 50 following the cameras 50 displayed on the upper left screen are displayed small. In FIG. 18, the images of four cameras 50 with camera numbers from 5 to 8 are displayed small. When the Next button is pressed further, the images of a plurality of cameras 50 following the cameras 50 displayed on the lower left screen are displayed small. On the other hand, when the Back button is pressed, it returns to the upper left screen. In a state where the images of a plurality of cameras 50 are displayed small, such as on the upper left or lower left screen, when the live display button for any one of the cameras 50 is pressed, the image of the camera 50 corresponding to the live display button is enlarged and displayed. When the Back button is pressed on the enlarged display screen, it returns to the original screen where the images of a plurality of cameras 50 are displayed small. When the enlarged display screen is clicked or the like, the enlarged screen is displayed in full screen and further enlarged. When a button such as Esc on the keyboard is pressed in the full screen display state, it returns to the enlarged display screen.

[0072] When the operation unit 26 selects detection pattern setting on the selection screen, it displays the detection pattern setting screen of FIG. 19. On the detection pattern setting screen, each detection pattern registered in the detection pattern master 32 is displayed, and each item of each detection pattern can be edited. Also, a new detection pattern can be added by pressing the Add button. Also, an existing detection pattern can be deleted by pressing the Delete button. In the selection screen of FIG. 19, the distance between objects 1-2 is set as the length of the X-axis × the length of the Y-axis, width, height, and distance from any one of the four corners of the rectangle.

[0073] Here, when setting the distance reference value, as shown in FIG. 20, the operation unit 26 displays image data such as when an actual danger is recognized. The displayed image data is selected by the user. Then, by performing a drag operation from the position of the detected object 1 to the position of the detected object 2, the distance from the position of the detected object 1 to the position of the detected object 2 is measured, and the measured distance is set as the distance reference value. Also, the position of the detected object 1 and the position of the detected object 2 are set accordingly. When editing the distance reference value, the operation unit 26 displays the set position of the detected object 1 and the detected object 2 and the distance therebetween. Then, the distance reference value is edited by re-performing a drag operation from the position of the detected object 1 to the position of the detected object 2.

[0074] When combination setting is selected on the selection screen, the operation unit 26 displays the combination setting screen of FIG. 21. On the combination setting screen, the related patterns registered in the related pattern master 34 are displayed for each camera 50 and each detection pattern. When the modification button of the detection pattern to be edited is pressed, the operation unit 26 displays the editing screen that initially displays the current setting content as shown in FIG. 22. On the editing screen of FIG. 22, in addition to editing the related pattern, it is also possible to edit the name of the camera 50 and the URL of the connection destination of the camera 50. The URL is the abbreviation of Uniform Resource Locator. Note that when the add button is pressed in FIG. 21, the operation unit 26 displays the editing screen of FIG. 22 with all fields blank as the initial display. Regarding the detection pattern setting screen shown in FIG. 19, although "near NG" and "far NG" are displayed in the perspective division column, it can be used not only for detecting an unsafe state but also for detecting a safe state. In that case, "near OK" and "far OK" can be selected, and in the detection pattern master 34 shown in FIG. 4, "3: near OK, 4: far OK" are also displayed. Also, for the warning sound number in the detection pattern master 34, a sound for recognizing safety is selected.

[0075] ***Effects of Embodiment 3*** As described above, the monitoring device 10 according to the third embodiment can confirm the live video of each camera 50 and set the detection pattern master 32 and the related pattern master 34. Thereby, it is possible to appropriately perform the operation using the monitoring device 10.

[0076] Hereinafter, various aspects of the present disclosure will be collectively described as appendices. (Appendix 1) An object information acquisition unit that acquires object information about one or more objects detected from the target image data, A determination unit that determines whether or not the object information acquired by the object information acquisition unit satisfies a detection pattern indicated by a detection pattern master provided to be editable, A notification unit that performs notification when it is determined by the determination unit that the detection pattern is satisfied A monitoring device comprising: (Appendix 2) The object information indicates the type of the object, The detection pattern indicates a condition combining the presence or absence of detection of objects of a specified type The monitoring device according to Appendix 1. (Appendix 3) The object information indicates the type and position of the object, The detection pattern indicates a condition using the distance between two specified types of objects The monitoring device according to Appendix 1 or 2. (Appendix 4) The detection pattern indicates a condition using the area where the object is detected The monitoring device according to any one of Appendices 1 to 3. (Appendix 5) The object information acquisition unit acquires object information using the image data acquired by each of a plurality of cameras as the target image data, The detection pattern is provided corresponding to each of the plurality of cameras, The determination unit determines, for each of the plurality of cameras as the target camera, whether the object information about the image data acquired by the target camera satisfies the detection pattern corresponding to the target camera. The monitoring device according to any one of Appendices 1 to 4. (Appendix 6) A plurality of detection patterns corresponding to different cameras are grouped, When it is determined that all the detection patterns included in the same group are satisfied for the grouped detection patterns, the notification unit performs a notification. The monitoring device according to Appendix 5. (Appendix 7) A plurality of detection patterns corresponding to cameras that photograph the same area from different angles are grouped. The monitoring device according to Appendix 6. (Appendix 8) A plurality of detection patterns indicating conditions using the distance between two specified types of objects are grouped. The monitoring device according to Appendix 7. (Appendix 9) The monitoring device further includes a measurement unit that measures the distance between two specified types of objects, and corrects the distance between the objects in the image data by using the ratio between the size of the detected object and the reference size corresponding to the detected type of object, thereby measuring the distance between the objects. The monitoring device according to Appendix 2, comprising the above. (Appendix 10) The monitoring device further includes an operation unit that accepts editing of the detection pattern master. The monitoring device according to any one of Appendices 1 to 9, comprising the above. (Appendix 11) The computer acquires object information about one or more objects detected from the target image data, and the computer determines whether the object information satisfies the detection pattern indicated by the detection pattern master provided to be editable. A monitoring method in which a computer gives a notification when it determines that a detected pattern is satisfied. (Appendix 12) For one or more objects detected from target image data, an object information acquisition process for acquiring object information, A determination process for determining whether the object information acquired by the object information acquisition process satisfies a detection pattern indicated by a detection pattern master provided to be editable, A notification process for giving a notification when it is determined by the determination process that the detection pattern is satisfied A monitoring program that causes a computer to function as a monitoring device that performs the above.

[0077] The embodiments and modifications of the present disclosure have been described above. Some of these embodiments and modifications may be implemented in combination. Also, any one or some of them may be partially implemented. Note that the present disclosure is not limited to the above embodiments and modifications, and various changes can be made as necessary.

Description of Signs

[0078] 100 Monitoring system, 10 Monitoring device, 11 Processor, 12 Memory, 13 Storage, 14 Communication interface, 21 Video acquisition unit, 22 Object information acquisition unit, 23 Determination unit, 24 Measurement unit, 25 Notification unit, 26 Operation unit, 31 Setting file, 32 Detection pattern master, 33 Result file, 34 Related pattern master, 50 Camera, 60 Alarm device.

Claims

1. an object information acquisition unit that acquires object information for one or more objects detected from the target image data; a determination unit that determines whether the object information acquired by the object information acquisition unit satisfies a detection pattern indicated by an editable detection pattern master; a notification unit that issues a notification when the determination unit determines that the detection pattern is satisfied; A monitoring device comprising:

2. the object information indicates a type of the object, The detection pattern indicates a combination of conditions regarding whether or not a specified type of object is detected. The monitoring device according to claim 1.

3. The object information indicates a type and a position of the object, The detection pattern indicates a condition using the distance between two specified types of objects. The monitoring device according to claim 1.

4. The detection pattern indicates a condition using an area in which an object is detected. The monitoring device according to claim 1.

5. the object information acquisition unit acquires object information using image data acquired by each of a plurality of cameras as image data of the object; the detection patterns are provided corresponding to the plurality of cameras, The determination unit determines, with each of the plurality of cameras as a target camera, whether or not the object information for image data acquired by the target camera satisfies the detection pattern corresponding to the target camera. The monitoring device according to claim 1.

6. Multiple detection patterns corresponding to different cameras are grouped together, The notification unit issues a notification when it is determined that the grouped detection patterns satisfy all of the detection patterns included in the same group. The monitoring device according to claim 5.

7. Multiple detection patterns are grouped together, corresponding to cameras capturing the same area from different angles.

7. The monitoring device according to claim 6.

8. A plurality of detection patterns that specify the same two types and indicate a condition using the distance between the specified two types of objects are grouped together.

8. The monitoring device according to claim 7.

9. The monitoring device further comprises: A measurement unit that measures a distance between two specified types of objects, the measurement unit measuring the distance between the objects by correcting the distance between the objects in the image data using a ratio between a size of the detected object and a reference size corresponding to the type of the detected object. The monitoring device according to claim 2 .

10. The monitoring device further comprises: An operation unit for accepting editing of the detection pattern master The monitoring device according to claim 1 .

11. The computer obtains object information for one or more objects detected from the image data of the object; The computer determines whether or not the object information satisfies a detection pattern indicated by an editable detection pattern master; The monitoring method includes providing a notification when the computer is determined to satisfy the detection pattern.

12. an object information acquisition process for acquiring object information for one or more objects detected from the image data of the target; a determination process for determining whether or not the object information acquired by the object information acquisition process satisfies a detection pattern indicated by an editable detection pattern master; a notification process for issuing a notification when it is determined by the determination process that the detection pattern is satisfied; A monitoring program that causes a computer to function as a monitoring device.

Images