Safety monitoring device, safety monitoring method, and program

Abstract

This safety monitoring device (10) comprises an area setting unit (12) which can set multiple notification areas for a safety sensor detection area, a notification type setting unit (13) which can set any notification function type for each notification area, and a moving body recognition analysis unit (16) which recognizes movement of a moving body on the basis of sensor data measured by a safety sensor. The safety monitoring device has a sensor data acquisition unit (a rider data acquisition unit (15)) which, from the safety sensor, acquires three-dimensional data as sensor data. The moving body recognition analysis unit extracts the moving body from the detection area on the basis of the three-dimensional data, and detects progress of the moving body towards each notification area.

Description

[Technical field]

[0001] The present invention relates to a safety monitoring device, a safety monitoring method, and a program. [Background technology]

[0002] For example, at construction sites, factories, warehouses, etc., equipment such as cranes, heavy machinery, and forklifts are in operation. There are multiple dangerous locations where accidents can occur due to contact between these devices and people. Therefore, there is a demand for detection means to ensure safety so that these devices do not come into contact with people. Therefore, for example, the technologies described in Patent Document 1 and Patent Document 2 have been proposed.

[0003] The technology described in Patent Document 1 uses an object detection sensor to detect people by arbitrarily applying various known image processing methods, machine learning-based classifiers, and the like.

[0004] The technology described in Patent Document 2 determines the distance between a person and a crane truck in three-dimensional space. The technology described in Patent Document 2 divides a moving object into multiple cubes (rectangular parallelepipeds) using depth map data, and determines the distance between the two cubes by subtracting the radius of the circumscribed sphere of each cube from the distance between the center points of the two cubes. The technology described in Patent Document 2 then determines the minimum value of the combination of distances between all the cubes as the shortest distance between the objects (shortest distance between a person and a crane truck). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2018 / 084146 [Patent Document 2] JP 2014-21810 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, the conventional techniques described in Patent Documents 1 and 2 are required to accurately grasp the relationship between a person and an object.

[0007] For example, the technology described in Patent Document 1 does not describe a specific human detection technology, so it is less convenient and it is not easy to build a system. Therefore, there is a demand for the technology described in Patent Document 1 to accurately grasp the relationship between people and objects.

[0008] In addition, the technology described in Patent Document 2 may not be able to detect danger, for example, when a person is directly under a crane truck, if the distance between the person and the crane truck is greater than a predetermined distance. In other words, when a person is directly under a crane truck, the person may be in a dangerous area. In this case, even if the distance between the person and the crane truck is greater than a predetermined distance, it may be unsafe. The technology described in Patent Document 2 may overlook unsafety in such a case. Therefore, there has been a demand for the technology described in Patent Document 2 to properly grasp the relationship between the person and the object so that unsafety is not overlooked.

[0009] The present invention has been made in consideration of the problems associated with the above-mentioned conventional technology, and an object of the present invention is to provide a safety monitoring device, a safety monitoring method, and a program that are capable of accurately grasping the relationship between a person and an object. [Means for solving the problem]

[0010] The above-mentioned object of the present invention can be achieved by the following means.

[0011] (1) A safety monitoring device, comprising: an area setting unit capable of setting a plurality of alarm target areas within a detection target area of ​​a safety sensor; an alarm type setting unit capable of setting an arbitrary alarm function type for each of the alarm target areas; and a moving object recognition and analysis unit that recognizes the movement of a moving object based on sensor data measured by the safety sensor. The alarm type setting unit can select, as the type of alarm function, any two or more of a function for issuing an alarm for all moving objects, a function for issuing an alarm only for people, a function for issuing an alarm when the distance between a person and a large object larger than a person is within a certain distance, a function for issuing an alarm when a person enters the direction of travel of the large object, a function for issuing an alarm only for moving objects moving at a certain speed or faster, a function for issuing an alarm only for moving objects moving at a certain speed or slower, a function for combining any two or more functions and taking a logical sum, and a function for combining any two or more functions and taking a logical product. Safety monitoring device. (2) A safety monitoring device having an area setting unit capable of setting multiple alarm target areas within a detection target area of ​​a safety sensor, an alarm type setting unit capable of setting an arbitrary type of alarm function for each of the alarm target areas, and a moving object recognition analysis unit that recognizes the movement of a moving object based on sensor data measured by the safety sensor, wherein the moving object recognition analysis unit extracts point cloud data representing a moving object from three-dimensional data measured by a lidar serving as the safety sensor, creates a bounding box of a size required to surround the point cloud data, and determines minimum and maximum values ​​for the size of a person using any one or all of five pieces of data: the width, depth, and height of the bounding box, as well as the height of the bottom and the height of the top of the bounding box, and determines the size of the moving object as a human being when the size of the moving object is within the determined range as a human determination condition, and identifies the moving object as a human being when the size of the moving object continues to meet the determination condition for an arbitrarily determined certain period of time. (3) A safety monitoring device comprising: an area setting unit capable of setting multiple alarm target areas within a detection target area of ​​a safety sensor; an alarm type setting unit capable of setting an arbitrary type of alarm function for each of the alarm target areas; a moving object recognition analysis unit that recognizes the movement of a moving object based on sensor data measured by the safety sensor; a record management unit that records and manages video data captured of all or part of the detection target area in a memory unit; and an event list processing unit that adds an event as alarm data to an event list and manages it when an alarm is issued, wherein the record management unit constantly records the video data in the memory unit, and when an arbitrary alarm location is specified from the event list, the event list processing unit displays on a display unit the video data for a certain period of time before and after the specified alarm location. (4) A safety monitoring device comprising: an area setting unit capable of setting multiple alarm target areas within a detection area of ​​a safety sensor; an alarm type setting unit capable of setting any type of alarm function for each of the alarm target areas; a moving object recognition analysis unit that recognizes the movement of a moving object based on sensor data measured by the safety sensor; a record management unit that divides video data into files at specified time intervals and records the files in a memory unit; and an event list processing unit that creates and manages an event list that can be selected for each event, wherein the event list processing unit performs control so that, for an event selected by a safety monitor, files containing video within the video playback time range of the event are not automatically deleted, or the files are re-saved in a separate area of ​​the memory unit.

[0012] ( 5 The above-mentioned (1) has a sensor data acquisition unit that acquires three-dimensional data as the sensor data from the safety sensor, and the moving object recognition analysis unit extracts moving objects from the detection target area based on the three-dimensional data and detects the progress of each moving object into the alarm target area. Any one of (4) to (4) A safety monitoring device as described in

[0014] ( 6 The type of the alarm function can be specified for each area via a user interface. 5 ) A safety monitoring device described in any one of claims 1 to 4.

[0016] ( 7 The moving object recognition analysis unit uses any one or more or all of the five pieces of data, namely the width, depth, and height of the bounding box, as well as the height of the bottom surface and the height of the top surface of the bounding box, to identify a large object larger than a person, calculates the distance between the person and the large object by subtracting the radius of the circumscribing circle of the bottom surface or the top surface of the bounding box of the person from the minimum value of the distance between the center of the bottom surface or the top surface of the bounding box of the person and each side of the bottom surface or the top surface of the bounding box of the large object, and issues a warning when the distance between the person and the large object becomes smaller than a certain distance determined arbitrarily. 2 ) A safety monitoring device as described above.

[0017] ( 8 ) the moving object recognition analysis unit issues a warning when the time during which the distance between the person and the large object becomes smaller than the certain distance continues for a certain period of time that is arbitrarily determined. 7 ) A safety monitoring device as described above.

[0018] ( 9 ) The moving object recognition analysis unit stops the warning when the distance between the person and the large object becomes greater than the certain distance.7 ) or ( 8 ) A safety monitoring device as described above.

[0019] ( 10 ) The moving object recognition analysis unit stops the warning when the time during which the distance between the person and the large object is greater than the certain distance continues for an arbitrarily determined certain period or more. 7 ) or ( 8 ) A safety monitoring device as described above.

[0020] ( 11 ) The moving object recognition analysis unit changes the warning level according to the distance between the person and the large object. 7 )~( 10 ) A safety monitoring device described in any one of claims 1 to 4.

[0021] ( 12 ) The moving object recognition and analysis unit calculates the speed of the moving object from the distance that the center position of the bounding box moves in a certain period of time. 2 ) 、(7) ~( 11 ) A safety monitoring device described in any one of claims 1 to 4.

[0022] ( 13 ) The above-mentioned (1) to ( 12 ) A safety monitoring device described in any one of claims 1 to 4.

[0024] (14) The event list processing unit is configured to filter and display the event data for each of the event types in the event list displayed on the display unit. 3 ) A safety monitoring device as described above.

[0025] (15) The present invention further includes a record management unit that records and manages the image data of the detection target area in a storage unit, and the record management unit records the image data before and after a base point in the storage unit according to the type of the alarm function, based on a time point at which an event that is the subject of an alarm occurs. 13) A safety monitoring device described in any one of claims 1 to 4.

[0026] (16) A safety monitoring device as described in any one of (1) to (15) above, wherein the alarm type setting unit can specify, as the alarm function, a process of detecting characteristics of the moving object from video data of the detection target area.

[0027] (17) The safety monitoring device according to (16) above, wherein the process of detecting the moving object includes one of person identification by face recognition, gender determination, and posture determination.

[0028] (18) A safety monitoring device as described in any one of (1) to (17) above, which notifies an external device equipped with a program for processing camera images of warning data detected by a set alarm function.

[0029] (19) The above-mentioned ( 2 ) 、(7) ~( 12 ) A safety monitoring device described in any one of claims 1 to 4.

[0030] (20) A safety monitoring device as described in (19) above, which has an image processing unit that processes the three-dimensional data according to the type of the alarm function, and the image processing unit displays the point cloud data and the bounding box simultaneously on a display unit.

[0031] (21) The safety monitoring device according to (19) or (20) above, wherein the smoothing means calculates the angle of the bottom face of the bounding box by smoothing.

[0032] (22) A safety monitoring device as described in (21) above, in which the angle of the bottom face of the bounding box discretely calculated by the smoothing means can be arbitrarily changed via a user interface.

[0033] (23) A safety monitoring device according to any one of (19) to (22) above, wherein either or both of the smoothing method and the smoothing time value of the smoothing means can be arbitrarily changed via a user interface.

[0034] (24) The safety monitoring device according to any one of (19) to (23) above, wherein the smoothing means smoothes the center coordinates of the bottom surface of the bounding box.

[0036] ( 25 The event list processing unit has a function of automatically deleting the file based on a predetermined rule, and the predetermined rule is either that a certain amount of free space in the recording means is secured or that a certain time has passed. 4 ) A safety monitoring device as described above.

[0037] ( 26 The event list processing unit acquires sensor data including one or both of a start time and an end time of an event that has occurred in the detection target area from the safety sensor, and determines the video playback time range of the event list. 4 ) or ( 25 ) A safety monitoring device as described above.

[0038] ( 27 The event list processing unit has a function of additionally registering any event that has occurred in the past as a new event in the event list. 4 ) 、(25) ~( 26 ) A safety monitoring device described in any one of claims 1 to 4.

[0039] ( 28 ) The event list processing unit has a function of editing the start time and end time of the event. 4 ) 、(25) ~( 27 ) A safety monitoring device described in any one of claims 1 to 4.

[0040] ( 29) a video processing unit that performs processing on the video data of the detection target area according to the type of the alarm function, the video processing unit creating a display screen that simultaneously displays the video and the event list on a display unit, and when a specific event is designated from the event list by a safety monitor via a graphical user interface, displaying a video including a video playback time range of the designated event on the display unit ( 4 ) 、(25) ~( 28 ) A safety monitoring device described in any one of claims 1 to 4.

[0041] ( 30 ) The event list allows the user to add an importance mark to each event to select whether or not to automatically delete the file via the graphical user interface. 29 ) A safety monitoring device as described above.

[0042] ( 31 ) The event list processing unit performs control not to automatically delete files including an event for which no automatic deletion is selected at the timing of selecting whether or not to automatically delete the file and a period from a certain time before the start time to a certain time after the end time of an event to which the importance mark has been added in advance. 30 ) A safety monitoring device as described above.

[0043] ( 32 ) the event list processing unit performs control to erase the file of the event when the event for which the automatic deletion is not selected is changed to the automatic deletion selection, or when the importance mark is removed from the event to which the importance mark has been added in advance. 31 ) A safety monitoring device as described above.

[0044] ( 33The event list processing unit deletes an event of the file from the event list when a file that has been recorded for a predetermined time is automatically deleted, while controlling not to delete from the event list an event for which no automatic deletion is selected at the timing for selecting whether or not to automatically delete the file and an event to which the importance mark has been added in advance. 30 )~( 32 ) A safety monitoring device described in any one of claims 1 to 4.

[0045] ( 34 ) The above-mentioned (a) further includes a safety monitoring unit that monitors the entry of an object into a monitoring area of ​​the safety sensor, and the safety monitoring unit determines a time when the object enters the monitoring area as a start time of the event, and determines a time when the object leaves the monitoring area or a time when a certain time has elapsed since the start time of the event as an end time of the event, and notifies the event list processing unit of the start time and end time of the event. 4 ) 、(25) ~( 33 ) A safety monitoring device described in any one of claims 1 to 4.

[0046] ( 35 2) A safety monitoring method, comprising: an area setting step for setting a plurality of alarm target areas in a detection target area of ​​a safety sensor; an alarm type setting step for setting an arbitrary alarm function type for each of the alarm target areas; and a moving object recognition and analysis step for recognizing the movement of a moving object based on sensor data measured by the safety sensor. In the alarm type setting step, any two or more of a function for issuing an alarm for all moving objects, a function for issuing an alarm only for people, a function for issuing an alarm when the distance between a person and a large object larger than a person is within a certain distance, a function for issuing an alarm when a person enters the direction of travel of the large object, a function for issuing an alarm only for moving objects moving at a certain speed or faster, a function for combining any two or more functions and taking a logical sum, and a function for combining any two or more functions and taking a logical product. Safety monitoring methods. (36) A safety monitoring method comprising: an area setting step capable of setting a plurality of alarm target areas within a detection target area of ​​a safety sensor; an alarm type setting step capable of setting an arbitrary type of alarm function for each of the alarm target areas; and a moving object recognition and analysis step of recognizing the movement of a moving object based on sensor data measured by the safety sensor, wherein the moving object recognition and analysis step extracts point cloud data representing a moving object from three-dimensional data measured by a lidar serving as the safety sensor, creates a bounding box of a size required to surround the point cloud data, and determines minimum and maximum values ​​for the size of a person using any one or all of five pieces of data: the width, depth, and height of the bounding box, as well as the height of the bottom and the height of the top of the bounding box; determines a human size based on the size of the moving object being within the determined range; and identifies the moving object as a human if the size of the moving object continues to meet the determination condition for an arbitrarily determined certain period of time. (37) A safety monitoring method comprising: an area setting step capable of setting a plurality of alarm target areas in a detection target area of ​​a safety sensor; an alarm type setting step capable of setting an arbitrary type of alarm function for each of the alarm target areas; a moving object recognition and analysis step for recognizing the movement of a moving object based on sensor data measured by the safety sensor; a record management step for recording and managing video data of all or part of the detection target area in a memory unit; and an event list processing step for adding an event as alarm data when an alarm is issued to an event list and managing the event, wherein in the record management step, the video data is constantly recorded in the memory unit, and when an arbitrary alarm location is specified from the event list, the video data for a certain period before and after the alarm location specified in the event list processing step is displayed on a display unit. (38) A safety monitoring method comprising: an area setting step capable of setting a plurality of alarm target areas within a detection target area of ​​a safety sensor; an alarm type setting step capable of setting an arbitrary type of alarm function for each of the alarm target areas; a moving object recognition and analysis step for recognizing the movement of a moving object based on sensor data measured by the safety sensor; a record management step for dividing video data into files at predetermined time intervals and recording the files in a memory unit; and an event list processing step for creating and managing an event list that can be selected for each event, wherein in the event list processing step, for an event selected by a safety monitor, either control is performed so that files containing video within the video playback time range of the event are not automatically deleted, or control is performed so that the files are re-saved in another area of ​​the memory unit.

[0047] ( 39) A program for causing a computer to function as a safety monitor that ensures safety in a detection target area of ​​a safety sensor, the program causing the computer to function as an area setting unit capable of setting a plurality of alarm target areas in the detection target area of ​​the safety sensor, an alarm type setting unit capable of setting an arbitrary type of alarm function for each of the alarm target areas, and a moving object recognition and analysis unit that recognizes the movement of a moving object based on sensor data measured by the safety sensor. The alarm type setting unit can select, as the type of alarm function, any two or more of a function for issuing an alarm for all moving objects, a function for issuing an alarm only for people, a function for issuing an alarm when the distance between a person and a large object larger than a person is within a certain distance, a function for issuing an alarm when a person enters the direction of travel of the large object, a function for issuing an alarm only for moving objects moving at a certain speed or faster, a function for issuing an alarm only for moving objects moving at a certain speed or slower, a function for combining any two or more functions and taking a logical sum, and a function for combining any two or more functions and taking a logical product. program. (40) A program causing a computer to function as a safety monitor that ensures safety in a detection area of ​​a safety sensor, the program causing the computer to function as an area setting unit capable of setting multiple alarm target areas in the detection area of ​​the safety sensor, an alarm type setting unit capable of setting an arbitrary type of alarm function for each of the alarm target areas, and a moving object recognition and analysis unit that recognizes the movement of a moving object based on sensor data measured by the safety sensor, the program causing the moving object recognition and analysis unit to extract point cloud data representing a moving object from three-dimensional data measured by a lidar serving as the safety sensor, create a bounding box of a size required to surround the point cloud data, determine minimum and maximum values ​​for each of the five data items of the width, depth, and height of the bounding box, as well as the height of the bottom and the height of the top of the bounding box, determine minimum and maximum values ​​for the size of a person using any one or all of the five data items of the width, depth, and height of the bounding box, and determine a condition for determining that the moving object is a person that the size of the moving object is within the determined range, and identify the moving object as a person if the size of the moving object continues to meet the determination condition for a certain amount of time that is arbitrarily determined. (41) A program causing a computer to function as a safety monitor that ensures safety in a detection area of ​​a safety sensor, the program causing the computer to function as an area setting unit capable of setting multiple alarm target areas in the detection area of ​​the safety sensor, an alarm type setting unit capable of setting an arbitrary type of alarm function for each of the alarm target areas, a moving object recognition analysis unit that recognizes the movement of a moving object based on sensor data measured by the safety sensor, a record management unit that records and manages video data captured of all or part of the detection target area in a memory unit, and an event list processing unit that adds an event as alarm data to an event list and manages it when an alarm is issued, the program causing the record management unit to constantly record the video data in the memory unit, and when an arbitrary alarm location is specified from the event list, causes the computer to function as a display unit with the video data for a certain period before and after the alarm location specified by the event list processing unit. (42) A program that causes a computer to function as a safety monitor that ensures safety in a detection area of ​​a safety sensor, the program causing the computer to function as an area setting unit capable of setting multiple alarm target areas in the detection area of ​​the safety sensor, an alarm type setting unit capable of setting any type of alarm function for each of the alarm target areas, a moving object recognition analysis unit that recognizes the movement of a moving object based on sensor data measured by the safety sensor, a record management unit that divides video data into files at specified time intervals and records the files in a memory unit, and an event list processing unit that creates and manages an event list that can be selected for each event, and causes the event list processing unit to either control not to automatically delete files containing video within the video playback time range of an event selected by a safety monitor, or control to re-save the files in a separate area of ​​the memory unit. Effect of the Invention

[0048] According to the present invention, the relationship between a person and an object can be grasped well. [Brief description of the drawings]

[0049] [Figure 1] 1 is a configuration diagram of a safety monitoring system including a safety monitoring device according to an embodiment. [Diagram 2] 1 is a configuration diagram of a safety monitoring device according to an embodiment. [Diagram 3] FIG. 2 is an explanatory diagram of an area subject to alert and an area excluded from alert. [Figure 4A] FIG. 1 is an explanatory diagram (1) of a display screen of a safety monitoring device according to an embodiment. [Figure 4B] FIG. 2 is an explanatory diagram (2) of the display screen of the safety monitoring device according to the embodiment. [Figure 4C] FIG. 3 is an explanatory diagram (3) of the display screen of the safety monitoring device according to the embodiment. [Diagram 5] FIG. 2 is an explanatory diagram showing an example of an area. [Figure 6] FIG. 11 is an explanatory diagram showing an example of selection of an alarm function via a user interface. [Figure 7] FIG. 1 is an explanatory diagram showing an example of point cloud data (L-shape data) and a bounding box of a bus. [Figure 8]FIG. 11 is an explanatory diagram showing an example of the coordinates and configuration of a bounding box. [Figure 9] FIG. 13 is an explanatory diagram showing an example of the bottom surface of a bounding box. [Figure 10A] This is an explanatory diagram (1) of how to find a rectangle in the (x,z) plane that will be the base of a bounding box. [Figure 10B] This is an explanatory diagram (2) of how to find a rectangle in the (x,z) plane that will be the base of the bounding box. [Figure 10C] This is an explanatory diagram (3) of how to find a rectangle in the (x,z) plane that will be the base of the bounding box. [Figure 11] FIG. 11 is an explanatory diagram illustrating an example of a bounding box. [Figure 12] FIG. 1 is an explanatory diagram showing an example of a determination condition for a person and a large object. [Figure 13A] FIG. 1 is an explanatory diagram (1) showing an example of classification results of detected moving objects and attributes that take continuity into consideration. [Figure 13B] FIG. 13 is an explanatory diagram (2) showing an example of classification results of detected moving objects and attributes that take continuity into consideration. [Figure 14] 11 is a flowchart showing the operation when a person is detected and an alarm is issued. [Figure 15] FIG. 1 is an explanatory diagram showing an example of the distance between a person and a large object as viewed from above. [Figure 16] 13 is a flowchart showing the operation when detecting a person or a large object and issuing an alarm. [Figure 17] 11 is an explanatory diagram showing an example of a warning screen displayed when a person detected by a lidar enters an area in a camera image that is subject to a warning (a no-entry area). FIG. [Figure 18A] 1 is an explanatory diagram (1) of the operation of a safety monitoring device according to an embodiment; [Figure 18B] FIG. 2 is an explanatory diagram (2) of the operation of the safety monitoring device according to the embodiment. [Figure 19] 1 is an explanatory diagram (3) of the operation of the safety monitoring device according to the embodiment. FIG. [Figure 20] FIG. 11 is an explanatory diagram showing an example of angle smoothing processing. [Figure 21] FIG. 11 is an explanatory diagram showing an example of angle smoothing processing. [Figure 22] FIG. 11 is an explanatory diagram showing a setting example of angle smoothing processing. [Figure 23] 4 is an explanatory diagram of the operation of the safety monitoring device according to the embodiment; FIG. [Figure 24] FIG. 2 is an explanatory diagram showing an example of a display screen. [Diagram 25] FIG. 11 is an explanatory diagram illustrating an example of an event list. [Figure 26] FIG. 13 is an explanatory diagram illustrating an example of an event editing screen. [Figure 27] FIG. 2 is an explanatory diagram showing an example of a format for saving video data in a recording means; [Figure 28] FIG. 13 is an explanatory diagram showing an example of a process for leaving video data of an event marked with an important star at the time of automatic deletion. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0050] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that each drawing is merely a schematic illustration to allow a sufficient understanding of the present invention. Therefore, the present invention is not limited to the illustrated examples. In addition, in each drawing, common or similar components are given the same reference numerals, and duplicated explanations thereof will be omitted.

[0051] <Configuration of safety monitoring device> The configuration of a safety monitoring device 10 according to this embodiment will be described below with reference to Fig. 1 and Fig. 2. Fig. 1 is a configuration diagram of a safety monitoring system S including the safety monitoring device 10 according to this embodiment. Fig. 2 is a configuration diagram of the safety monitoring device 10.

[0052] As shown in FIG. 1, the safety monitoring system S includes one or more safety monitoring devices 10, one or more riders 51, one or more surveillance cameras 61, and one or more Web terminals 62.

[0053] The safety monitoring device 10 is a device that monitors the safety of a detection target area (measurement space) of a safety sensor. The safety monitoring device 10 is configured with a personal computer (PC) or the like. The lidar 51 is a three-dimensional sensor that is installed around the measurement space to be used as a safety sensor and detects objects.

[0054] The surveillance camera 61 is an imaging unit that is placed around the measurement space and captures the measurement space. The surveillance camera 61 may be configured to have a setting reception unit 61a, a setting processing unit 61b, and a notification unit 61c, like the surveillance camera 61z. The surveillance camera 61z is an external device equipped with a program (not shown) for performing camera image processing. The setting reception unit 61a is a means for receiving settings for the camera image processing. The setting processing unit 61b is a means for executing the camera image processing received by the setting reception unit 61a on the image data. The setting processing unit 61b may be set to determine the safety of the measurement space. The notification unit 61c is a means for notifying the safety monitoring device 10 or the Web terminal 62 of the image that has been subjected to the camera image processing and the result of the determination of the safety of the measurement space.

[0055] The Web terminal 62 is an external device equipped with a program (not shown) for processing camera images. In this embodiment, the Web terminal 62 is described as having a notification acquisition unit 62a. The notification acquisition unit 62a is a means for acquiring images that have been subjected to camera image processing and a result of a judgment on the safety of the measurement space from the safety monitoring device 10 or the monitoring camera 61z.

[0056] 2, the safety monitoring device 10 includes a control unit 11, a storage unit 30, and a communication unit 39. The safety monitoring device 10 also includes an operation unit 41, a display 42 (display unit), a warning lamp 43, and a speaker 44 (alarm unit).

[0057] The control unit 11 has an area setting unit 12, an alert type setting unit 13, an image acquisition unit 14, a lidar data acquisition unit 15, a moving object recognition analysis unit 16, a result output unit 17, an image processing unit 18, a record management unit 19, an image display processing unit 20, a safety sensor input unit 21, a safety monitoring unit 22, and an event list processing unit 23.

[0058] The area setting unit 12 is a means for setting an area with any attribute (for example, an alarm target area or an alarm excluded area described later) in a detection target area (measurement space) of the lidar 51 which is a safety sensor. The alarm type setting unit 13 is a means for setting an arbitrary alarm function type for each area. The image acquisition unit 14 is a means for acquiring image data of the measurement space. The lidar data acquisition unit 15 is a sensor data acquisition unit that acquires sensor data from a lidar 51 serving as a safety sensor arranged around the measurement space.

[0059] The moving object recognition analysis unit 16 is a means for extracting a moving object from three-dimensional data based on sensor data measured by a safety sensor (lidar 51 in this embodiment), recognizing the motion of the moving object, analyzing the traveling direction, and issuing a warning. The moving object recognition analysis unit 16 has a foreground data extraction unit 16a, a clustering processing unit 16b, a bounding box generation unit 16c, an angle smoothing unit 16d, an image recognition unit 16e, and an alarm analysis unit 16f.

[0060] The foreground data extraction unit 16a is a means for extracting foreground data from input three-dimensional data. The clustering processing unit 16b is a means for grouping data based on the similarity between data by unsupervised learning. The bounding box generating unit 16c is a means for generating a bounding box. Here, the term "bounding box" refers to the smallest rectangular or rectangular parallelepiped area that encloses an object.

[0061] The angle smoothing unit 16d is a means for performing averaging processing using a pre-specified number of frames or processing method. The image recognition unit 16e is a means for recognizing an image from video data, and for determining a person by identifying a face, a gender, a posture, and the like. The alarm analysis unit 16f is a means for performing a process according to the type of alarm function using the result of the moving object detection by the clustering process, and issuing an alarm.

[0062] The result output unit 17 is a means for sending a warning issued when a moving object is detected by the moving object recognition analysis unit 16 to the video processing unit 18 as event data. The image processing unit 18 is a means for creating a display screen to be displayed on the display 42. The image processing unit 18 can perform processing such as superimposing event data on the image data. For example, the image processing unit 18 can perform any camera image processing such as superimposing a person detection frame on the camera image.

[0063] The record management unit 19 is a means for recording and managing video data in a storage unit. The video display processing unit 20 is a means for processing the files divided for each predetermined time period to create a video and displaying it on the display 42.

[0064] The safety sensor input unit 21 is a means for inputting signals output from safety sensors such as a lidar, a camera, a human sensor, a touch sensor, and the like. The safety monitoring unit 22 is a means for monitoring the intrusion of an object into a monitoring area. The event list processing unit 23 is a means for creating and managing an event list that can be selected for each event. The event list includes, as events, notification data, custom data, and the like.

[0065] The storage unit 30 stores a control program Pr, setting record data 31, video record data 32, and processed image data 33.

[0066] The control program Pr is a program that causes a computer to function as the safety monitoring device 10 . The setting record data 31 is data that is set in the safety monitoring device 10 via a user interface. The video record data 32 is video data captured by a surveillance camera 61 . The processed image data 33 is image data that has been subjected to any processing on the video data.

[0067] <Area settings> The setting of areas will be described below with reference to Fig. 3. Fig. 3 is an explanatory diagram of areas subject to alert and areas excluded from alert.

[0068] The area setting unit 12 of the safety monitoring device 10 sets an alarm target area and an alarm excluded area within a detection target area (measurement space).

[0069] The detection target area is a measurement space in which an object is measured by the lidar 51. In other words, the detection target area is an area in which the movement of a moving object existing within the detection range of the lidar 51 is monitored by the safety monitoring device 10.

[0070] An alert target area is an area in which an event is issued (a warning is issued) when an intruder is detected. An event is data indicating that a warning has been issued. The safety monitoring device 10 continues to issue a warning while an intruder is present within the alert target area. Furthermore, the safety monitoring device 10 does not issue a warning even if a moving object is detected outside the alert target area. Multiple alert target areas can be set. The setting of the alert target area is only effective within the detection target area. Note that if an alert target area is set that overlaps with an alert exclusion area, the alert exclusion area takes priority, and the safety monitoring device 10 does not issue a warning.

[0071] An alarm exclusion area is an area in which an event is not issued (a warning is not issued) even if an intruder is detected. An alarm exclusion area is specified when detection of moving objects is intentionally excluded from the monitoring target. The safety monitoring device 10 does not detect an intruder or issue an event (a warning is not issued) within the range specified as the alarm exclusion area, even if the intruder is within the detection target area or the alarm target area. For example, if there is a tree within the detection target area and the tree is swaying due to the wind, the safety monitoring device 10 can prevent false detection by setting the area around the tree as an alarm exclusion area. Multiple alarm exclusion areas can be set.

[0072] As shown in Fig. 3, the area setting unit 12 can set multiple alarm target areas for one detection target area (measurement space). In the example shown in Fig. 3, two alarm target areas A61a and A61b are set for the detection target area (measurement space) indicated by the two-dot chain line. Note that, although only one alarm exclusion area A62 is set in the example shown in Fig. 3, the area setting unit 12 can set multiple alarm exclusion areas for one detection target area (measurement space).

[0073] 3, when a person moves along the walking route R11, a portion where the walking route R11 and the alert-activation exclusion area A62 overlap is set as an exclusion portion B62. A portion where the walking route R11 and the alert-activation target area A61a overlap, but does not overlap with the alert-activation exclusion area A62, is set as an alert-activation portion B61a. A portion where the walking route R11 and the alert-activation target area A61b overlap is set as an alert-activation portion B61b. The safety monitoring device 10 issues an event (issues a warning) when or while a person enters the alert-activation portions B61a and B61b.

[0074] An example of area setting is shown in Fig. 4A to Fig. 4C. Fig. 4A to Fig. 4C are explanatory diagrams of the display screen 100 of the safety monitoring device 10, respectively. The safety monitoring device 10 displays the display screen 100 on the display 42 and accepts the setting of the area by the safety monitor. Here, the "safety monitor" means a person who monitors the safety of each measurement space by the safety monitoring device 10.

[0075] In the example shown in FIG. 4A, the display screen 100 includes a 3D viewer section 101, a setting section 102, a rider selection section 103, an alert target area tab 104, and the like.

[0076] The 3D viewer section 101 is a section for displaying an image of the measurement space captured by the surveillance camera 61 and an image obtained by processing the image. The setting section 102 is a section for setting various settings such as the area and the type of alarm function. The LIDAR selection unit 103 is an instruction unit that selects three-dimensional data (range measurement data) acquired by any LIDAR 51. The alert target area tab 104 is a tab for setting an alert target area.

[0077] In the example shown in FIG. 4B, the screen of the alert target area tab 104 includes a block tab 104a, a valid check box 104b, an area selection section 104c, a display color setting section 104d, and the like.

[0078] The block tab 104a is a tab for setting the specifications of a bounding box to be added to an object that is the target of a report. Here, the term "bounding box" refers to the smallest rectangular or cuboid area that encloses a certain object. The enable check box 104b is a field for specifying whether or not a bounding box is to be added to the object that is the target of the alert. The area selection section 104c is a field for selecting an area to be set as an area for which an alert is to be issued. The display color setting section 104d is a section for setting the alarm target area frame 101a (see FIG. 4C) displayed on the 3D viewer section 101.

[0079] When the safety monitor selects (designates) an area in the area selection section 104c, the safety monitoring device 10 sets the selected area as an area for which an alert is to be issued. At this time, the safety monitoring device 10 changes the color of the area frame 101a (see FIG. 4C) displayed on the 3D viewer section 101 to the color set in the display color setting section 104d.

[0080] <Alarm function> The safety monitoring device 10 has various types of alarm issuing functions. Here, for example, a case where the safety monitoring device 10 uses the following three types of alarm issuing functions will be described as an example. (1) Entry detection: An alert is issued for any moving object that enters an area subject to an alert (a no-entry area). (2) Entry detection (human): An alert is issued only to human moving objects that enter an area subject to an alert (a no-entry area). (3) Proximity detection: An alarm is issued when a person or large object enters an area subject to alarm (no entry area) and the distance between them falls below a predetermined certain amount.

[0081] Hereinafter, an example of using the alarm function will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is an explanatory diagram showing an example of an area. Fig. 6 is an explanatory diagram showing an example of selection of the alarm function by a user interface.

[0082] For example, in the example shown in FIG. 5, three areas, a first area, a second area, and a third area, are set for one measurement space. The first area is an empty area. Therefore, the safety monitor can set any alarm function for the first area depending on the operation. For example, the safety monitor can set "(1) Entry detection" for the first area to set the first area to detect any moving object. The second area is a forklift work area, which is a dangerous area if a person enters it. Therefore, the safety monitor sets "(2) Entry detection (person)" for the second area. The third area is a crane work site, and is an area where it is dangerous for people to get closer to the crane than a certain distance R. Therefore, the safety monitor sets "(3) Proximity detection" for the third area to warn people who are closer than the certain distance R from the crane.

[0083] As shown in FIG. 6, the display screen 100 (see FIG. 4A) of the safety monitoring device 10 is configured so that the setting unit 102 can select one of these three types of alarm functions. Furthermore, the display screen 100 (see FIG. 4A) of the safety monitoring device 10 is configured so that a plurality of areas (no entry areas) for which an alarm is to be issued can be selected, and each area can be independently set to an area with any attribute.

[0084] <Creating a bounding box> The creation of the above-mentioned "bounding box" will be described below with reference to Fig. 7 to Fig. 10C. Fig. 7 is an explanatory diagram showing an example of point cloud data of a bus and a bounding box. Fig. 8 is an explanatory diagram showing an example of the coordinates and configuration of a bounding box. Fig. 9 is an explanatory diagram showing an example of the bottom surface of a bounding box. Figs. 10A to 10C are each an explanatory diagram of a method for obtaining a rectangle on the (x, z) plane that will be the bottom surface of a bounding box.

[0085] The bounding box is created, for example, by the method described in the above-mentioned Patent Document 1 (International Publication No. 2018 / 084146). Fig. 7 shows an example of point cloud data (L-shape shape data) of a bus measured by a rider 51 and a rectangular bounding box formed to surround the point cloud data. Fig. 8 shows an example of the coordinates and configuration of a rectangular bounding box formed along the three-dimensional shape of the bus.

[0086] In the three-dimensional coordinate system (x, y, z), the horizontal plane is (x, z) and the height direction is the y coordinate. A bounding box is made up of the smallest rectangular or cuboid area that encloses an object. The length of the long side of the base is the width W (Width), the length of the narrowest side is the depth D (Depth), and the length from the base to the top is the height H (Height).

[0087] Fig. 9 shows an example of the bottom surface of a bounding box of a bus that is placed at an angle with respect to the X-axis. In the example shown in Fig. 9, the angle between the bus and the X-axis is indicated as angle θ.

[0088] 10A to 10C each show a method for obtaining a rectangle on the (x, z) plane that is the bottom surface of the bounding box. In the figures, pixel points indicated by black circles (hereinafter referred to as "black circle pixel points") represent each pixel of the point cloud that has been subjected to the clustering process.

[0089] 10A to 10C, first, a straight line is drawn passing through each black circle pixel point with a slope of angle θ, and the outermost straight lines are set as the two straight lines in the X-axis direction of the bounding box at angle θ. Next, a straight line is drawn passing through each black circle pixel point with a slope of angle θ+90°, and the outermost straight lines are set as the two straight lines in the Z-axis direction of the bounding box at angle θ. Finally, the rectangle enclosed by these four straight lines is set as the bounding box at angle θ.

[0090] This procedure is repeated for θ=0°, Δθ, 2×Δθ, 3×Δθ, ..., (n-1)×Δθ (where n is the largest integer such that ((n-1)×Δθ)<90°) to find (calculate) the bounding box at each tilt.

[0091] Then, from among the bounding boxes for each obtained inclination, the one with the highest correlation with the point cloud data is determined as the optimal bounding box for this point cloud data. In this manner, the bounding boxes for each monitoring target are created.

[0092] <Relationship between moving objects and bounding boxes> The relationship between a moving object and a bounding box will be described below with reference to Fig. 11 to Fig. 13B. Fig. 11 is an explanatory diagram showing an example of a bounding box. Fig. 12 is an explanatory diagram showing an example of a determination condition for a person and a large object. Fig. 13A is an explanatory diagram showing an example of a classification result of a detected moving object and an attribute (for a large object) that takes continuity into consideration. Fig. 13B is an explanatory diagram showing an example of a classification result of a detected moving object and an attribute (for a person) that takes continuity into consideration.

[0093] As shown in FIG. 11, the safety monitoring device 10 detects a moving object 105 with the lidar 51, and generates a bounding box 105box in the shape of a rectangular parallelepiped region (rectangular parallelepiped region) so as to just surround the moving object 105 (surround it with the minimum volume).

[0094] The safety monitoring device 10 judges whether the moving object 105 enclosed by the bounding box 105box is a person or a large object larger than a person based on the size (width, depth, height) of the bounding box 105box. The judgment is performed based on the judgment conditions shown in FIG. 12, for example.

[0095] 12, the values ​​of width W, depth D, height H, Ht, and Hb of the moving object 105 are shown as the determination conditions. Note that height H is the distance from the top to the bottom of the bounding box 105box. Furthermore, height Ht is the distance from the ground G to the top of the bounding box 105box. Furthermore, height Hb is the distance from the ground G to the bottom of the bounding box 105box.

[0096] For example, when all values ​​of the width W, depth D, height H, Ht, and Hb of a moving object 105 shown in a certain frame in a series of video data meet the criteria for determining a "person" shown below, the safety monitoring device 10 classifies the moving object 105 as a "person." However, the following ranges are merely examples and can be changed depending on the operation. ("Person" criteria) Width W: Range from minimum 0.3 [m] to maximum 1.2 [m]. Depth D: Range from minimum 0.2 [m] to maximum 1.0 [m]. Height H: Range from minimum 0.5 [m] to maximum 2.3 [m]. Height Ht: Range from minimum 0.5 [m] to maximum 3.0 [m]. Height Hb: Range from minimum -1.0 [m] to maximum 0.8 [m].

[0097] Specifically, the safety monitoring device 10 classifies a moving object 105 as a "person" if all the values ​​of the width W, depth D, height H, Ht, and Hb of the moving object 105 captured in a frame in a series of video data are within the following ranges. 0.3[m]≦Width W≦1.2[m] 0.2[m]≦Depth D≦1.0[m] 0.5[m]≦Height H≦2.3[m] 0.5[m]≦Height Ht≦3.0[m] -1.0[m]≦Height Hb≦0.8[m]

[0098] Furthermore, the safety monitoring device 10 classifies the moving object 105 as a "large object" when each value of the width W, depth D, height H, Ht, and Hb of the moving object 105 meets the following criteria for determining a "large object." However, the following ranges are merely examples and can be changed depending on the operation. (Criteria for determining "large object") In each of the items of width W, depth D, height H, Ht, and Hb of the moving object 105, all of the items are equal to or greater than the minimum value of "person", and one or more items are equal to or greater than the maximum value of "person".

[0099] Specifically, the safety monitoring device 10 classifies a moving object 105 as a "large object" if the values ​​of each item in the width W, depth D, height H, Ht, and Hb of the moving object 105 in a certain frame (bounding box 105box) satisfy the following first and second conditions. (First condition) All values ​​for each item must satisfy the following conditions. 0.3[m]≦Width W 0.2[m]≦Depth D 0.5[m]≦Height H 0.5[m]≦Height Ht -1.0[m]≦Height Hb (Second condition) The value of one or more items must satisfy the following conditions. 1.2[m]≦Width W 1.0[m]≦Depth D 2.3 [m] ≦ Height H 3.0[m]≦Height Ht 0.8[m]≦Height Hb

[0100] In addition, if the values ​​of the width W, depth D, height H, Ht, and Hb of the moving object 105 do not conform to the judgment criteria for a person and the judgment criteria for a large object, the safety monitoring device 10 classifies the moving object 105 as unknown (other).

[0101] The safety monitoring device 10 preferably sets the threshold value large in consideration of a case where multiple people are detected as one object. Also, the safety monitoring device 10 preferably sets the threshold value large in consideration of a case where the moving object 105 appears to be floating above the ground, such as when only the upper body of a person is visible.

[0102] As described above, the safety monitoring device 10 can classify the moving object into three types, namely, "person", "large object", and "unknown (other)", for each frame. However, the moving object detected by the lidar 51 may split or change in size during clustering processing due to measurement variations. Therefore, a large object may momentarily become a mass the size of a person. This may result in the large object being classified as a "person". Therefore, for example, as shown in FIG. 13A, the safety monitoring device 10 may preferably classify the attribute of the moving object 105 as "large object (large moving object)" when the size of the moving object 105 is maintained for a number of consecutive frames that are arbitrarily determined. This allows the safety monitoring device 10 to prevent the large object from being classified as a "person".

[0103] 13A, the safety monitoring device 10 classifies the attribute of the moving object 105 as "large object (large moving object)" if the size of the moving object 105 is maintained for three consecutive frames. Thereafter, the safety monitoring device 10 maintains the attribute of the moving object 105 as "large object (large moving object)" unless three consecutive frames of the size of another attribute (e.g., "person") occur.

[0104] In addition, a moving object 501 classified as a "person" may momentarily become a block larger than a person (or a block smaller than a person) due to the influence of some kind of noise. In that case, for example, as shown in Fig. 13B, the continuity of the moving object 501 is interrupted, so it is advisable to review the attributes of the moving object 501 only when the continuity occurs for an arbitrarily determined number of consecutive frames (e.g., three frames).

[0105] In addition, in the case of a 10 fps (frames per second) LIDAR 51, one frame time is 0.1 seconds, so three frames are equivalent to 0.3 seconds. Here, the case of three consecutive frames has been described, but the optimal number of frames may be selected depending on the site.

[0106] In addition, the safety monitoring device 10 may set up an area (no entry area) within the measurement range of the lidar 51 for which an alert is to be issued, and issue an alert only when a moving object 501 identified as a "person" enters the area for which an alert is to be issued.

[0107] <Operation of safety monitoring device> The operation of the safety monitoring device 10 will be described below with reference to Fig. 14. Fig. 14 is a flow chart showing the operation when a person is detected and an alarm is issued.

[0108] 14, in step S101, a safety monitor operates the safety monitoring device 10 while looking at the display screen 100 (see FIG. 4A) to set an area with any attribute in the measurement space. In response to this, the area setting unit 12 of the safety monitoring device 10 sets one or more alarm target areas and one or more alarm excluded areas in one measurement space.

[0109] Next, in step S102, the safety monitor operates the safety monitoring device 10 while looking at the display screen 100 (see FIG. 4A) to specify the type of alarm function for each alarm target area. In response to this, the alarm type setting unit 13 of the safety monitoring device 10 sets the type of alarm function for each alarm target area. Thereafter, the safety monitor operates the safety monitoring device 10 to instruct the safety monitoring device 10 to start monitoring the measurement space.

[0110] Then, in step S103, the image acquisition unit 14 of the safety monitoring device 10 acquires image data of each measurement space from each monitoring camera 61.

[0111] Next, in step S105, the LIDAR data acquisition unit 15 of the safety monitoring device 10 acquires distance measurement data (θ(i), φ(j), R) from the LIDAR 51 as three-dimensional data (sensor data).

[0112] Next, in step S110, the foreground data extraction unit 16a of the moving object recognition analysis unit 16 of the safety monitoring device 10 extracts foreground data from the three-dimensional data by a background subtraction method. That is, the foreground data extraction unit 16a extracts foreground data by calculating (three-dimensional data-reference background data).

[0113] Next, in step S115, the clustering processing unit 16b of the moving object recognition analysis unit 16 of the safety monitoring device 10 extracts clusters of point cloud data (moving objects) from the foreground data based on the distance data of the point cloud detected by the three-dimensional data, and performs clustering processing.

[0114] Next, in step S120, the bounding box generation unit 16c of the moving object recognition analysis unit 16 of the safety monitoring device 10 creates a bounding box by creating a rectangular region required to surround the cluster of point cloud data (moving object) that has been clustered.

[0115] Next, in step S125, the alarm analysis unit 16f of the moving object recognition analysis unit 16 of the safety monitoring device 10 checks whether the moving object meets the criteria for determining a person from the size (width, depth, height) of the bounding box. If it is confirmed that the moving object meets the criteria for determining a person, the alarm analysis unit 16f adds (sets) a flag representing a person (person flag) to the moving object. This allows the safety monitoring device 10 to recognize the moving object as a person.

[0116] Next, in step S130, the alarm analysis unit 16f of the moving object recognition analysis unit 16 of the safety monitoring device 10 detects the size of the same moving object in the next frame. Then, if the size of the moving object matches the size of a person for the designated number of consecutive frames, the alarm analysis unit 16f sets the attribute of the moving object to "person."

[0117] Next, in step S135, the alert analysis unit 16f of the moving object recognition analysis unit 16 of the safety monitoring device 10 checks whether or not a moving object has entered the alert target area. If a moving object has entered and the attribute of the moving object is "person", the alert analysis unit 16f issues an alert.

[0118] <How to detect distance> A method for detecting distance will be described below with reference to Fig. 15. Fig. 15 is an explanatory diagram showing an example of the distance between a "person" and a "large object" as viewed from above.

[0119] The safety monitoring device 10 can obtain (calculate) the distance between the classified "person" and "large object" as follows.

[0120] 15, the bottom surface of the bounding box 106box of the person 106 does not become an extremely rectangular shape. Therefore, the safety monitoring device 10 sets a circumscribing circle that surrounds the bounding box 106box to the area of ​​the person 106.

[0121] When the horizontal axis is the X-axis and the vertical axis is the Z-axis, the coordinates of the center point 106o of the bounding box 106box of the person 106 can be represented as (ADx, ADz). And let the radius of the circumscribed circle surrounding the bounding box 106box be AD_R.

[0122] The bottom surface of the large object 105a is a rectangle with four points a, b, c, and d as vertices respectively. The safety monitoring device 10 calculates (obtains) the distances between the center point 106o of the person 106 and the line segment ab of the large object 105a, the distances between the center point 106o of the person 106 and the line segment bc of the large object 105a, the distances between the center point 106o of the person 106 and the line segment cd of the large object 105a, and the distances between the center point 106o of the person 106 and the line segment da of the large object 105a respectively. Then, the safety monitoring device 10 sets the minimum value of the obtained (calculated) values as the distance Dmin between each of the line segments ab, bc, cd, da and the center point 106o of the person 106.

[0123] The safety monitoring device 10 can calculate (obtain) the distance between the person 106 and the large object 105a by (Dmin - AD_R).

[0124] The safety monitoring device 10 may calculate the distances for all combinations between all detected persons 106 and the large object 105a every frame, and issue a warning when the calculated distance is shorter than the specified threshold distance.

[0125] At that time, the safety monitoring device 10 may issue different types of warnings according to the calculated distance. For example, when the calculated distance is the following threshold distances Th1, Th2, Th3 (not shown), the following warnings can be issued. Note that the threshold distances Th1, Th2, Th3 (not shown) are in the relationship of Th1 > Th2 > Th3. (1) When (Dmin - AD_R) < Th1, the safety monitoring device 10 sets the level as the caution level. In this case, the safety monitoring device 10 lights the warning lamp 43 yellow. (2) When (Dmin - AD_R) < Th2, the safety monitoring device 10 sets the level to the warning level. In this case, the safety monitoring device 10 lights the warning lamp 43 red. (3) When (Dmin - AD_R) < Th3, the safety monitoring device 10 sets the level to the danger level. In this case, the safety monitoring device 10 causes the warning lamp 43 to blink red and sounds an alarm from the speaker 44.

[0126] Hereinafter, with reference to FIG. 16, the operation of the safety monitoring device 10 will be described. FIG. 16 is a flowchart showing the operation at the time of detecting and reporting a person and a large object.

[0127] As shown in FIG. 16, the operation at the time of detecting and reporting a person and a large object is different from the operation at the time of detecting and reporting a person shown in FIG. 14 in the following points. (1) After step S120, the determination process of step S121 is performed. (2) When it is determined as "Yes" in the determination of step S121, the process proceeds to step S125. (3) When it is determined as "No" in the determination of step S121, the safety monitoring device 10 performs the processes of step S145 and step S150. (4) Step S135 is deleted, and instead, after step S130 or step S150, the process of step S155 is performed. Except for these differences, the operation at the time of detecting and reporting a person and a large object shown in FIG. 16 and the operation at the time of detecting and reporting a person shown in FIG. 14 are the same.

[0128] Specifically, as shown in FIG. 16, after step S120, in step S121, the reporting analysis unit 16f of the moving object recognition analysis unit 16 of the safety monitoring device 10 determines whether the object is a person. When it is determined in the determination of step S121 that the object is a person ("Yes" case), the process proceeds to step S125. In this case, similar to the operation at the time of detecting and reporting a person shown in FIG. 14, the processes from step S125 to step S130 are performed. After step S130, the process proceeds to step S155.

[0129] On the other hand, if it is determined in step S121 that the object is not a person ("No"), the process proceeds to step S145.

[0130] In step S145, the alarm analysis unit 16f of the moving object recognition analysis unit 16 of the safety monitoring device 10 confirms that the moving object meets the criteria for determining a large object based on the size (width, depth, height) of the bounding box, and adds (sets) a flag (large object flag) representing a large object to the moving object. This allows the safety monitoring device 10 to recognize the moving object as a large object.

[0131] Next, in step S150, the alarm analysis unit 16f of the moving object recognition analysis unit 16 of the safety monitoring device 10 detects the size of the same moving object in the next frame. If the size of the moving object meets the large object determination criteria for the designated number of consecutive frames, the alarm analysis unit 16f sets the attribute of the moving object to "large object." After step S150, the process proceeds to step S155.

[0132] In step S155, the alert analysis unit 16f of the moving object recognition analysis unit 16 of the safety monitoring device 10 checks whether a moving object has entered the alert area. When both a "person" and a "large object" have entered the alert area, the alert analysis unit 16f measures the distance between the "person" and the "large object," and issues a warning that the "person" is in danger if the distance is shorter than a certain distance.

[0133] FIG. 17 is an explanatory diagram showing an example of a warning screen when a person 106 detected by the lidar 51 enters an area (no-entry area) in a camera image that is a target for alert generation. The safety monitoring device 10 projects three-dimensional data acquired from the lidar 51 onto two-dimensional image data (camera image) acquired from the surveillance camera 61. The safety monitoring device 10 is set so that the positional relationship between the three-dimensional data and the image data matches. Then, as shown in FIG. 17, the safety monitoring device 10 detects a moving object, adds a bounding box 106box to the moving object, and recognizes the moving object as a person 106 based on the size of the bounding box 106box. Note that, when the person 106 enters an area (no-entry area) in which an alert generation is target, the safety monitoring device 10 may change the color of the bounding box 106box of the person 106 in the camera image to red or superimpose a red frame on the part of the person 106 in question in order to generate an alert.

[0134] The safety monitoring device 10 can also be configured to constantly record (video record) video data (camera video) acquired from the surveillance camera 61 in the storage unit 30. Such a safety monitoring device 10 can create an alarm list by recording the video data (camera video) at the moment an alarm is issued without erasing it. In this case, the safety monitoring device 10 can specify and play back (display on the display 42) the video data (camera video) at the time of alarm activation for a required period of time based on the alarm activation list.

[0135] Further, the safety monitoring device 10 can be configured to operate as shown in Fig. 18A or 18B. Fig. 18A and Fig. 18B are explanatory diagrams of the operation of the safety monitoring device 10.

[0136] 18A, the safety monitoring device 10 acquires three-dimensional data (distance measurement data) measured by the lidar 51 with the lidar data acquisition unit 15. The lidar data acquisition unit 15 outputs the acquired three-dimensional data (distance measurement data) to the moving object recognition analysis unit 16.

[0137] In the moving object recognition analysis unit 16, the alarm analysis unit 16f detects a moving object based on the alarm function designated for each area. Then, the alarm analysis unit 16f judges whether or not any area is in a situation where an alarm should be issued. If there is an area where an alarm should be issued, the moving object recognition analysis unit 16 outputs a turn-on instruction signal to the alarm lamp so that the lamp is turned on according to the level of the alarm. In response to this, the alarm lamp turns on. The alarm lamp may be composed of multiple lamps and designated for each area.

[0138] Moreover, the moving object recognition analysis unit 16 outputs the result of the moving object detection to the video processing unit 18 via the result output unit 17. In response to this, the video processing unit 18 creates a display screen 100 (see FIG. 4A) to be displayed on the display 42. At that time, the video processing unit 18 may perform any camera image processing, such as superimposing a person detection frame on the camera image.

[0139] 18B, the safety monitoring device 10 may be configured to have an image recognition unit 18a for processing camera images in the image processing unit 18. The image recognition unit 18a performs person determination by facial recognition, gender determination, posture determination, and the like for people recognized by the moving object recognition analysis unit 16, based on the image data (camera images) captured by the surveillance camera 61. This allows the safety monitoring device 10 to perform even more detailed risk level determination.

[0140] <About bounding box smoothing> The safety monitoring device 10 preferably has a bounding box smoothing function. FIG. 19 is a diagram illustrating the operation of the safety monitoring device 10.

[0141] 19, the safety monitoring device 10 outputs three-dimensional data (distance measurement data) acquired from the lidar 51 by the lidar data acquisition unit 15 to the moving object recognition analysis unit 16. In the moving object recognition analysis unit 16, the foreground data extraction unit 16a calculates the difference between the three-dimensional data (distance measurement data) that is the input data and background data acquired in advance, and extracts the foreground data.

[0142] Next, the clustering processing unit 16b extracts point cloud data having a certain amount of mass from the foreground data and sets it as a moving object candidate.Then, the clustering processing unit 16b verifies the continuity of the point cloud data set as the moving object candidate with all frames, regards those with continuity as moving objects, and assigns IDs to them.After this, the bounding box generating unit 16c generates a bounding box for the point cloud data regarded as moving objects.

[0143] The angle θ between frames in the (x,z) plane of the bottom surface of the created bounding box is a discrete value for each Δθ. Therefore, the angle smoothing unit 16d performs averaging processing using a pre-specified number of frames or processing method. At that time, the safety monitor may be able to change the number of frames or the processing method using the operation unit 41.

[0144] Furthermore, the safety monitoring device 10 simultaneously displays the video data (camera video) and the bounding box obtained as point cloud data on the display 42. Therefore, the safety monitoring device 10 can show the safety monitor an approximate outline of the part behind the rider 51 that has no point cloud data, to the extent that the outline can be visually seen by the bounding box.

[0145] For example, a case will be described where the input data from the lidar 51 is 10 fps (frames per second). Fig. 20 is an explanatory diagram showing an example of angle smoothing processing. Fig. 20 shows an example where the angle θ of the bounding box of the same object for each frame is set to Δθ=6° and calculated by a simple average formula of 5 frames.

[0146] The angle θ obtained is one of the discrete angles 0°, 6°, 12°, 18°, ..., 78°, and 84°. In the illustrated example, the angle θ gradually changes from 6° to 12°. In the illustrated example, the angle θ is mixed between 6° and 12° due to the variation in the point cloud. The movement of the bounding box in this state appears jerky to the eye.

[0147] Here, when the angle of the i-th frame is θ(i) and the average angle is θave(i), the simple average equation for five frames is given by the following equation (1). θave(i)=(θ(i-4)+θ(i-3)+θ(i-2)+θ(i-1)+θ(i))÷5 …(1)

[0148] The average angle shown in FIG. 20 is the angle calculated by the above formula (1). The average angle shown in FIG. 20 is an angle of smooth change. Therefore, the movement of the bounding box looks stable and is relatively close to the movement of an actual moving object (such as a bus). However, since the angle data between the past four frames and the current frame is used, the movement of the bounding box is slightly delayed. The smoothness of the movement of the bounding box can be improved by increasing the number of smoothed frames, but the responsiveness decreases. Therefore, it is better to select the number of smoothed frames according to the circumstances of the moving object to be detected. For example, since buses and trucks rarely turn suddenly, a larger number of frames can be selected, and for vehicles that can turn easily, such as forklifts, a smaller number of frames can be selected.

[0149] Fig. 21 is an explanatory diagram showing an example of angle smoothing processing, similar to Fig. 20. However, while Fig. 20 shows an example in which the angle is calculated using a simple average formula of five frames, Fig. 21 shows an example in which the angle is calculated using a weighted average formula of five frames weighted by the following formula (2). The weighted average formula places emphasis on the current frame.

[0150] Here, if the angle of the i-th frame is θ(i) and the average angle is θave(i), the weighted average equation of the five frames weighted by the weighted average equation is given by the following equation (2). θave(i)=(1×θ(i-4)+2×θ(i-3)+3×θ(i-2)+4×θ(i-1)+5×θ(i))÷15 …(2)

[0151] Fig. 22 is an explanatory diagram showing a setting example of angle smoothing processing. Fig. 22 shows an example of the case where averaging processing is set by operating the operation unit 41 through a graphical user interface (GUI) (not shown).

[0152] In the example shown in FIG. 22, the display screen is configured to allow a setting of whether to "perform" or "not perform" the angle smoothing process using a radio button. The display screen is also configured to allow a setting of performing angle smoothing when the aspect ratio (width / depth) of a moving object is less than a certain value. The display screen is also configured to allow a setting of performing angle smoothing when the size (width) of the object to be performed is less than a certain size. Note that the larger the value of the size (width) of the object to be performed is set, the more frequently the angle smoothing unit 16d performs angle smoothing on the moving object. The display screen is also configured to allow a setting of performing angle smoothing when the number of averaging frames is less than a certain number. In the illustrated example, 5 frames is selected as the number of averaging frames, but this value can be freely changed. The display screen is also configured to allow a setting of either simple averaging or weighted averaging to be selected using a radio button.

[0153] In addition, if the center coordinates of the bottom face of the bounding box in the i-th frame are (Cx(i), Cz(i)), the angle smoothing process of Cxave(i) and Czave(i) is performed according to the following equations (3) and (4). Cxave(i)=(Cx(i-4)+Cx(i-3)+Cx(i-2)+Cx(i-1)+Cx(i))÷5 …(3) Czave(i)=(Cz(i-4)+Cz(i-3)+Cz(i-2)+Cz(i-1)+Cz(i))÷5 (4)

[0154] In the above calculation example, the angle smoothing unit 16d performs angle smoothing processing using a simple average, but the angle may be smoothed using a weighted average. The safety monitoring device 10 may perform angle smoothing processing and central coordinate smoothing processing simultaneously using a graphical user interface (not shown). The safety monitoring device 10 may also be provided with a graphical user interface (not shown) independent of the angle.

[0155] <Regarding the exclusion of video data marked as important when automatically deleting recorded videos> The safety monitoring device 10 may be configured to constantly record (store in the storage unit 30) the video data (camera video) acquired from the surveillance camera 61 and delete the video data after a certain period of time has elapsed. Fig. 23 is an explanatory diagram of the operation of such a safety monitoring device 10.

[0156] As shown in FIG. 23, the safety monitoring device 10 converts video data (camera video) acquired from a monitoring camera 61 into manageable data such as MxPEG by the video processing unit 18 and outputs the data to the record management unit 19.

[0157] The record management unit 19 divides the converted data into files (file data) at predetermined time intervals and records them in storage means (storage unit 30 or an external recording medium not shown) so that database management is possible. The record management unit 19 also acquires files from the video processing unit 18 based on instructions from the safety monitor, and outputs the acquired files to the video display processing unit 20. The record management unit 19 also reads out a file of a specified time from the group of files recorded in the storage means (storage unit 30 or an external recording medium not shown), and outputs it to the video display processing unit 20. The video display processing unit 20 creates an image based on the file and displays it on the display 42.

[0158] In addition, the safety monitoring device 10 acquires a detection signal of an object in the measurement space from the safety sensor 50 such as a lidar, a camera, a human sensor, or a touch sensor by the safety sensor input unit 21, and outputs the detection signal to the safety monitoring unit 22. The safety monitoring unit 22 detects the intrusion of a person or a dangerous object based on the detection signal and issues an event. That is, the safety monitoring unit 22 monitors the intrusion of an object into the monitoring area of ​​the safety sensor 50. Then, the safety monitoring unit 22 outputs the event to the event list processing unit 23. At this time, the safety monitoring unit 22 determines the time when the object enters the monitoring area of ​​the safety sensor 50 as the start time of the event, and determines the time when the object leaves the monitoring area or the time when a certain time has elapsed since the start time of the event as the end time of the event. Then, the safety monitoring unit 22 notifies the event list processing unit 23 of the start time and end time of the event.

[0159] In response to this, the event list processing unit 23 creates and updates the event list, and displays, for example, a display screen 110 shown in Fig. 24 on the display 42. Fig. 24 is an explanatory diagram showing an example of the display screen 110. In the example shown in Fig. 24, the display screen 110 is configured to simultaneously display images of each measurement space captured by the surveillance camera 61 and the updated event list. The safety monitor operates the operation unit 41 while looking at the display screen 110. The event list processing unit 23 edits the events in response to the operation of the safety monitor.

[0160] In the example shown in FIG. 24, the display screen 110 is configured so that a video display section 111 for displaying videos is arranged on the left side, and an event list display section 112 for displaying an event list is arranged on the right side.

[0161] In the example shown in Fig. 24, the video display unit 111 is divided into four screens. The safety monitoring device 10 is configured to be able to connect up to four surveillance cameras 61, and displays the video captured from each surveillance camera 61 on any screen of the video display unit 111. The safety monitoring device 10 may use the four screens of the video display unit 111 differently depending on the operation. For example, the safety monitoring device 10 may display a live video from the surveillance camera 61 on the upper left screen of the video display unit 111, and display a recorded and played video from a storage means on the upper right screen of the video display unit 111.

[0162] Furthermore, the safety monitoring device 10 is configured to be able to play back on the video display unit 111 video related to each event in the event list displayed on the event list display unit 112. For example, the safety monitor selects an event he or she wishes to play back from the event list displayed on the event list display unit 112 and double-clicks the event with the mouse or drags it onto the screen he or she wishes to play back on the video display unit 111. This causes the safety monitoring device 10 to play back on the video display unit 111 video related to the selected event.

[0163] Fig. 25 is an explanatory diagram showing an example of an event list. In the example shown in Fig. 25, the event list 120 has, from the left, an importance mark column 121, a time stamp column 122, a camera name column 123, a type column 124, and a description column 125.

[0164] An importance mark indicating that the event is important is added to the importance mark column 121. In the illustrated example, the importance mark has a star shape, but the shape of the importance mark is not limited to a star shape and can be any shape.

[0165] The time stamp column 122 is a column for displaying the start time of the event. The camera name column 123 is a display column for the name of the camera that captured the video. The type column 124 is a display column for the type of event (Intruder, CUSTOM, etc.). The explanation column 125 is a display column for explaining the event. In addition to the above, the event list 120 can be configured to display the end time of the event, the cause of the event, and the like.

[0166] The event list 120 is configured so that the importance mark can be turned on / off by clicking the importance mark column 121 of any event with a mouse. A safety monitor plays back the video of any event, checks the video, and if it is determined that an incident (important event) has occurred in the video being played back, adds an importance mark to the importance mark column 121 of that event. The safety monitoring device 10 stores the video related to an event with an importance mark added so that it is not deleted.

[0167] In the safety monitoring device 10, a safety monitor can arbitrarily select one event from the event list 120 and perform editing operations on the event on an event editing screen 130 shown in Fig. 26, thereby performing editing processing on the event. Fig. 26 is an explanatory diagram showing an example of the event editing screen 130.

[0168] In the example shown in FIG. 26, the event editing screen 130 includes a camera name column 131, a user name column 132, an occurrence date and time column 133, an end date and time column 134, a description column 135, a type column 136, a mark column 137, a cause column 138, a countermeasure column 139, and an importance column 140.

[0169] The camera name column 131 is a display column for the name of the camera that captured the video. The user name column 132 is a column for displaying the user name. The occurrence date and time column 133 is a column for displaying the date and time when the event occurred. The end date and time column 134 is a column for displaying the end date and time of the event. The description field 135 is an input field for explaining the event. The type column 136 is an input column for the type of event (Intruder, CUSTOM, etc.). The mark column 137 is a display column for the type of important mark. The cause column 138 is an input column for the cause of the incident that occurred. The countermeasure column 139 is an input column for countermeasures against the incident that has occurred. The importance column 140 is an input column for the importance of the incident that has occurred.

[0170] The event editing screen 130 shown in FIG. 26 reflects the detailed data of the event to which the important mark shown in FIG. 25 has been added. In the example shown in FIG. 26, the occurrence date and time (start time) of the event displayed in the occurrence date and time column 133 is 12:05:41 on January 17, 2020. In addition, the end date and time (end time) of the event displayed in the end date and time column 134 is 12:11:20 on January 17, 2020. When playing back the video data of this event, the safety monitor may play back the video by taking a certain amount of time (for example, 10 seconds before and after) longer than the occurrence date and time (start time) and end date and time (end time) of the event. For example, the safety monitor may edit the occurrence date and time column 133 and the end date and time column 134 on the event editing screen 130 to set the occurrence date and time of the event to 12:05:31 on January 17, 2020 and the end date and time of the event to 12:11:30 on January 17, 2020, and play back the video. The fixed time period (for example, 10 seconds before or after the date and time change) that is applied to the change can be changed as desired.

[0171] The safety monitor can edit the occurrence date and time (start time of the event) in the occurrence date and time field 133 and the end date and time (end time of the event) in the end date and time field 134 on the event edit screen 130 .

[0172] Although not shown, the safety monitoring device 10 can limit the end time of an event to a time up to "the start time of the event plus the maximum event time" by setting a maximum event time.

[0173] The safety monitoring device 10 can be configured to operate as follows. For example, a monitoring area is set in advance in the rider 51, and the time when a person enters the monitoring area is set as the occurrence date and time (start time) of the event, and the time when the person leaves the monitoring area is set as the end date and time (end time) of the event. Then, the safety monitoring device 10 can acquire the occurrence date and time data and end date and time data of the event from the rider 51 when registering an event, and record and manage the acquired date and time data in the storage unit 30.

[0174] Furthermore, when a moving object stays in a monitored area for a long time, the safety monitoring device 10 can limit the end time of the event to a time up to "the start time of the event plus the maximum time of the event." Furthermore, the safety monitoring device 10 may be configured so that other items can also be edited from the event editing screen 130. Furthermore, the safety monitoring device 10 may be configured so that, by using the event editing screen 130, a video of any time can be played back and a new event can be created.

[0175] Fig. 27 is an explanatory diagram showing an example of a format in which video data is saved in a storage means. Fig. 27 shows an example in which SQLite, which is one of the open source relational database management systems (RDBMS: Relational Database Management System), is used for file management.

[0176] As shown in FIG. 27, in the storage unit 30, a database management file called "archive.db" is created under the "Video folder," and date folders indicating the dates (for example, "2020 / 1 / 17 folder," "2020 / 1 / 18 folder," "2020 / 1 / 19 folder," etc.) are created one per day.

[0177] Under each date folder, for example, a video data file compressed in H.264 format is created. In the example shown in FIG. 27, for example, "video_20200117_000000.mp4" and "video_20200117_001000.mp4" are created as video data files. "video_20200117_000000.mp4" is a video data file from 0:00:00 on January 17, 2020 for 10 minutes. Also, "video_20200117_001000.mp4" is a video data file from 0:10:00 on January 17, 2020 for 10 minutes. In this embodiment, it is described that video data files are created at 10-minute intervals in this way. As a result, 144 files (24 hours x (6 files / 1 hour)) are created per day.

[0178] In the case of a normal XGA format (1024 x 768 pixels), the amount of data for 10 minutes of recorded data is about 500 MB, and the amount of data for one month of recorded data is about 2 TB. Therefore, if the safety monitoring device 10 uses a large-capacity HDD (Hard Disk Drive) as the recording means (storage unit 30 or external recording media not shown), it can constantly store about 30 days of recorded data.

[0179] In this embodiment, the safety monitoring device 10 constantly stores 30 days of recorded data (video data files) and automatically deletes, once a day, any one-day recorded data (video data files) that has been recorded for more than 30 days. At this time, the safety monitoring device 10 automatically deletes from the event list any events that have been issued for more than 30 days.

[0180] However, the safety monitoring device 10 does not erase but stores the recorded data (video data files) relating to the events to which the important marks shown in FIG. 25 and FIG. 26 have been added.

[0181] For example, if the recording data shown in FIG. 25 has a time stamp of 12:05:41 on Jan. 17, 2020, and an important mark is not added, the recording data will be automatically deleted when the storage period of 30 days has expired. However, the recording data shown in FIG. 25 has a time stamp of 12:05:41 on Jan. 17, 2020, and an important mark is added. Therefore, even if the timing for automatic deletion arrives (that is, even if the storage period of 30 days from recording has expired), this recording data will continue to be stored without being deleted. In addition, events corresponding to such recording data with an important mark added will remain in the event list without being deleted.

[0182] When automatically deleting recorded data once a day, the safety monitoring device 10 checks events marked with an important mark, calculates the playback time of the event marked with an important mark, and saves recorded data (video data file) including that time without deleting it. For example, in this case, for recorded data with a timestamp of 12:05:41 on Jan. 17, 2020 as shown in FIG. 25, the safety monitoring device 10 saves recorded data (video data) including 12:05:31 to 12:11:30 on Jan. 17, 2020 without deleting it.

[0183] In the example shown in Fig. 27, the recorded data including 12:05:31 to 12:11:30 on January 17, 2020 is two files, "Video_20200117_120000.mp4" and "Video_20200117_121000.mp4." As shown in Fig. 28, even when the recorded data is automatically deleted, the safety monitoring device 10 leaves two files related to the event marked with an important mark and automatically deletes the other files.

[0184] In addition, if the video data file is not deleted when the automatic deletion of recorded data occurs, and the importance mark is subsequently removed from the event through editing, the video data file will be deleted when the automatic deletion of recorded data occurs thereafter. In this case, the event whose importance mark has been removed will also be deleted from the event list.

[0185] <Main features of the safety monitoring device> (1) As shown in FIG. 2, the safety monitoring device 10 of this embodiment is configured to include an area setting unit 12 capable of setting multiple alarm target areas within the detection target area of ​​the safety sensor, an alarm type setting unit 13 capable of setting any type of alarm function for each alarm target area, and a moving object recognition analysis unit 16 that recognizes the movement of a moving object based on sensor data measured by the safety sensor.

[0186] At construction sites, manufacturing sites, and the like, various control areas are assumed, such as areas where heavy machinery (e.g., cranes) work, areas where work vehicles such as forklifts move around, and areas where all moving objects must not enter for vibration suppression, security, and the like. The safety monitoring device 10 according to this embodiment treats such various control areas as alarm target areas, and can set multiple alarm target areas for the detection target areas of the safety sensors. In addition, the safety monitoring device 10 can set any type of alarm function for each alarm target area. In other words, the safety monitoring device 10 can independently set different alarm functions for each of the multiple alarm target areas. Therefore, the safety monitoring device 10 can realize an optimal monitoring method. In addition, the safety monitoring device 10 can properly grasp the relationship between people and objects.

[0187] (2) As shown in Fig. 4A, the safety monitoring device 10 according to this embodiment has a sensor data acquisition unit (lidar data acquisition unit 15) that acquires three-dimensional data (distance measurement data) as sensor data from a lidar 51 serving as a safety sensor. The moving object recognition analysis unit 16 may be configured to extract moving objects from a detection target area based on the three-dimensional data and detect the progress of the moving objects into each of the alert target areas.

[0188] The safety monitoring device 10 according to this embodiment can set multiple areas to the same measurement accuracy conditions by using a safety sensor (e.g., a lidar 51) that can measure three-dimensional data. Therefore, the safety monitoring device 10 can ensure stable detection accuracy. Furthermore, when the conditions to be monitored change, the safety monitoring device 10 can respond by simply changing the area settings, without making physical changes such as reattaching the sensor.

[0189] (3) The alarm type setting unit 13 of the safety monitoring device 10 of this embodiment may preferably be configured to be able to select as the type of alarm function any two or more of the following: a function to issue an alarm for all moving objects, a function to issue an alarm only to people, a function to issue an alarm when the distance between a person and a large object larger than a person is within a certain distance, a function to issue an alarm when a person enters the direction of a large object, a function to issue an alarm only to moving objects moving at a certain speed or above, a function to issue an alarm only to moving objects moving at a certain speed or below, a function that combines any two or more functions and takes a logical AND, and a function that combines any two or more functions and takes a logical AND.

[0190] The safety monitoring device 10 according to the present embodiment can use various types of alarm functions depending on the operation. For example, in an area where heavy machinery (e.g., a crane) moves, it is preferable to issue an alarm when a person approaches the heavy machinery. In an area where a work vehicle such as a forklift moves, it is preferable to monitor so that a worker does not enter. In an area where heavy machinery running on rails moves, it is preferable to issue an alarm when a person enters in the direction of travel. In an area where all moving objects should not enter for vibration suppression, security, etc., it is preferable to monitor the movement of all moving objects. In some areas, it is preferable to monitor so that an alarm is issued when a moving object enters the area at a speed exceeding the speed limit. The safety monitoring device 10 can realize the issuance of an appropriate alarm in such an area. In some areas, the condition for issuing an alarm is set to a logical AND condition of a person and a large object (e.g., a car) at a certain speed or more, so that the safety monitoring device 10 can be set to issue an alarm when a large object (e.g., a car) enters the area at a speed exceeding the speed limit. In addition, by setting the condition for issuing a warning as a logical OR condition of the speeds of a person and a large object, the safety monitoring device 10 can be set to issue a warning when a large object enters the area.

[0191] (4) As shown in FIGS. 5 and 6, in the safety monitoring device 10 according to this embodiment, the type of alarm function may be configured so that it can be specified for each area via a user interface.

[0192] At construction sites, manufacturing sites, and the like, the environment changes from moment to moment depending on the progress of work. For this reason, it is preferable that the area settings and alarm function selection can be arbitrarily and easily changed depending on the situation at the site. The safety monitoring device 10 according to this embodiment can realize an environment in which the settings can be arbitrarily and easily changed via a user interface.

[0193] (5) As shown in Figures 12 to 14, the moving object recognition analysis unit 16 of the safety monitoring device 10 of this embodiment extracts point cloud data representing a moving object from three-dimensional data measured by a lidar 51 as a safety sensor, creates a bounding box of a size required to surround the point cloud data, and determines minimum and maximum values ​​for each of the five pieces of data, namely the width, depth, and height of the bounding box, as well as the height of the bottom and the height of the top of the bounding box, as the size of the moving object to be within the determined range as a condition for determining that the moving object is a person.If the size of the moving object continues to meet the determination condition for an arbitrarily determined certain period of time, the moving object may be identified as a person.

[0194] The safety monitoring device 10 according to this embodiment detects people from the size of the bounding box, and can perform detection in a simple manner without using known image processing techniques or machine learning-based techniques.

[0195] The point cloud data of the moving object measured by the lidar 51 is processed as one mass (object) by clustering processing. At that time, since a large object (for example, a car, etc.) often has an uneven shape, it may be split into multiple masses and recognized without being stably recognized as one object. In this case, the size of one of the split masses may be the same as the size of a person. Therefore, if the attribute of the moving object is determined simply based on the size of a person, there is a possibility that the large object may be misclassified as a person. In contrast, the safety monitoring device 10 according to the present embodiment can determine the moving object (mass) as a person when the state in which the size of the moving object (mass) is determined to be the size of a person continues for a certain period of time because the state in which the large object is split into multiple masses and recognized often disappears in a short period of time. Even if the large object is misclassified as a person based on the size of the moving object (mass), such a safety monitoring device 10 can significantly reduce the misclassification of the large object as a person by determining whether the moving object (mass) is a person or not based on both the size of the moving object (mass) and the duration of that size.

[0196] Moreover, in construction sites, manufacturing sites, and the like, workers walk on the ground. In such environments, the phenomenon in which a large object is split into multiple blocks and recognized, and the size of one of the split blocks becomes the size of a person, often occurs at a location higher than the ground. Therefore, the safety monitoring device 10 according to this embodiment adds a condition that the height of the bottom surface of the bounding box is equal to or lower than a certain height. Furthermore, since a person is at most 2 m tall, the safety monitoring device 10 according to this embodiment can prevent a moving object other than a human from being mistakenly recognized as a human by determining that the object is not a human if the height of the top surface of the bounding box is higher than an arbitrarily determined certain height.

[0197] In addition, since the safety monitoring device 10 according to the present embodiment detects people using the size of a bounding box, it is possible to include moving objects other than people by adjusting this size. For example, the safety monitoring device 10 can adjust the size of the bounding box so that it includes a person carrying luggage, a group of people, a small car, an object carried by a crane, and the like.

[0198] (6) As shown in Fig. 12, the moving object recognition analysis unit 16 of the safety monitoring device 10 according to this embodiment identifies a large object larger than a person by using any one or more or all of the following five pieces of data: the width W, depth D, and height H of the bounding box, and the height Hb and top height Ht of the bounding box. As shown in Fig. 15, the moving object recognition analysis unit 16 may calculate the distance between the person and the large object as the distance (Dmin-AD_R) obtained by subtracting the radius of the circumscribing circle of the bottom or top of the person's bounding box from the minimum value of the distance between the center point 106o of the bottom or top of the person's bounding box 106box and each side of the bottom or top of the large object's bounding box 105box, and issue a warning when the distance between the person and the large object becomes smaller than a certain distance determined arbitrarily.

[0199] When calculating the distance between a person and a large object, there may be a three-dimensional distance where a dangerous object such as a crane truck is hanging above the person's head. In such a case, even if the distance between the person and the crane truck is greater than a predetermined distance, it may still be unsafe. It is desirable for the safety monitoring device 10 according to this embodiment not to overlook such a case. Therefore, the moving object recognition analysis unit 16 of the safety monitoring device 10 according to this embodiment may calculate the distance using the bottom (or top) of each bounding box. As shown in FIG. 15, the moving object recognition analysis unit 16 calculates the distance from the center to each line segment of the bottom of the large object's bounding box 105box using the center point 106o of the bottom of the person's bounding box 106box as a reference, and calculates the distance (Dmin-AD_R) obtained by subtracting the radius of the circumscribing circle of the bottom of the person's bounding box 106box from the minimum value as the distance between the person and the large object. This allows the safety monitoring device 10 to calculate all combinations of distances between multiple people and large objects in the area where an alert is to be issued (no entry area), and issue a warning when any of these combinations falls below a certain distance. Such a safety monitoring device 10 can perform thorough monitoring so as not to miss any danger, even when a dangerous object is hanging above a person's head, such as the above-mentioned crane truck, and the danger is easily overlooked.

[0200] (7) As shown in Figures 15 to 18B, the moving object recognition analysis unit 16 of the safety monitoring device 10 of this embodiment may be configured to issue a warning when the distance between a person and a large object remains smaller than a certain distance for an arbitrarily determined certain period of time.

[0201] At construction sites, manufacturing sites, and the like, when a person approaches a large object and remains within a certain distance from the object, multiple warnings may be issued in succession. Therefore, the safety monitoring device 10 according to this embodiment can prevent multiple warnings from being issued in succession by issuing a warning after a certain period of time has elapsed.

[0202] (8) The moving object recognition and analysis unit 16 of the safety monitoring device 10 according to this embodiment may be configured to stop the warning when the distance between a person and a large object becomes greater than a certain distance.

[0203] When a person approaches a large object and then moves away from the large object and the distance between the person and the large object becomes greater than a certain distance, the person has retreated to a safety zone. Therefore, the safety monitoring device 10 according to this embodiment may stop issuing a warning when the distance between the person and the large object becomes greater than a certain distance.

[0204] (9) The moving object recognition analysis unit 16 of the safety monitoring device 10 of this embodiment may be configured to stop the warning when the time during which the distance between a person and a large object remains greater than a certain distance continues for an arbitrarily determined period of time or more.

[0205] When a person approaches a large object and then moves away from the object and remains within a certain distance, multiple warnings may be issued in succession before the person retreats to a safety zone. Therefore, the safety monitoring device 10 according to this embodiment may stop issuing warnings when the time during which the distance between the person and the large object remains greater than the certain distance continues for an arbitrarily set period of time or more.

[0206] (10) The moving object recognition and analysis unit 16 of the safety monitoring device 10 according to this embodiment may be configured to change the warning level depending on the distance between a person and a large object.

[0207] The safety monitoring device 10 according to this embodiment can change the warning level, such as a caution level, a warning level, a danger level, etc., according to the approaching distance when a person approaches an object. In this case, the safety monitoring device 10 can change the warning lamp from yellow to red when the warning level changes from the caution level to the warning level, and can sound a warning sound (buzzer sound) from the speaker 44 when the warning level changes from the warning level to the danger level.

[0208] (11) The moving object recognition and analysis unit 16 of the safety monitoring device 10 according to this embodiment may be configured to calculate the speed of a moving object from the distance that the center position of a bounding box moves in a certain period of time.

[0209] The safety monitoring device 10 according to this embodiment can determine the speed of a moving object from the distance that the center position of the bounding box moves in a certain period of time (e.g., 0.5 seconds), thereby determining the timing at which a moving object enters an area.

[0210] (12) The safety monitoring device 10 according to the present embodiment may be configured to include a video processing unit 18 that superimposes the position of a moving object detected by a safety sensor (for example, the lidar 51) on video data.

[0211] For example, when the moving object is a person who is the target of a warning, the safety monitoring device 10 according to this embodiment can make the safety monitor quickly and visually recognize the danger by superimposing a warning marker on the person as the position of the moving object. This allows the safety monitor to take some kind of action quickly.

[0212] (13) As shown in Fig. 2, the safety monitoring device 10 according to this embodiment has a record management unit 19 that records and manages video data captured of all or part of the detection target area in a storage unit 30, and an event list processing unit 23 that adds an event including the alert data to an event list when an alert is issued and manages the event. The record management unit 19 constantly records the video data in the storage unit 30. The safety monitoring device 10 may be configured such that, when an arbitrary alert location is specified from the event list, the event list processing unit 23 displays video data for a certain period of time before and after the specified alert location on the display 42 (display unit).

[0213] The safety monitoring device 10 according to this embodiment records and saves the time when the warning was issued and creates an event list so that the playback point can be specified, allowing the safety monitor to understand the behavior of the person for whom the warning was issued. The safety monitoring device 10 can then play back video data from the time before and after the specified event by specifying an arbitrary event. This allows the safety monitor to check the video data from the time before and after the specified event and verify the degree of danger. This also allows the safety monitor to provide on-site workers with education on predicting danger.

[0214] (14) The event list processing unit 23 of the safety monitoring device 10 in this embodiment may be configured to filter and display alert data (events including alert data) by type for the event list displayed on the display 42 (display unit).

[0215] The safety monitoring device 10 according to the present embodiment can filter and display the alarm data (events including the alarm data) by type. Therefore, the safety monitoring device 10 can easily extract recorded data that meets the conditions by, for example, filtering by extracting only the function that issues an alert only to people. The safety monitoring device 10 can facilitate the work of checking the video data by the safety monitor by managing and storing the alarm data (events including the alarm data) by alarm type. The safety monitoring device 10 can limit the recording of the camera video to only when an alarm occurs, for example. Furthermore, the safety monitoring device 10 can selectively record a certain period of time before and after the alarm function according to the type of the alarm function, thereby reducing the storage capacity. The safety monitoring device 10 can facilitate the work of checking the video data by the safety monitor by managing and storing the video data by alarm type.

[0216] (15) As shown in Fig. 2, the safety monitoring device 10 according to this embodiment has a record management unit 19 that records and manages video data of a detection target area in a storage unit 30. As shown in Fig. 24 and Fig. 25, the record management unit 19 may be configured to record video data before and after a base point, which is the time point at which an event that is the subject of a warning occurs, in the storage unit 30, depending on the type of alarm function.

[0217] The safety monitoring device 10 according to this embodiment records video data before and after a base point, which is the time point when an event that is the subject of a warning occurs, in the storage unit 30. Therefore, the safety monitoring device 10 can manage video data for each event that occurs. Furthermore, by managing and storing video data for each event that occurs, the safety monitoring device 10 can facilitate the task of checking the video data by the safety monitor.

[0218] (16) The alarm type setting unit 13 of the safety monitoring device 10 according to this embodiment may be configured to be able to designate, as an alarm function, a process for detecting characteristics of a moving object from video data of a detection target area.

[0219] For example, the detailed characteristics of a moving object cannot be obtained by only detecting the moving object using the lidar 51. Therefore, the safety monitoring device 10 according to the present embodiment may be configured to be able to specify a process for detecting the characteristics of a moving object from video data. In this way, the safety monitoring device 10 can obtain detailed characteristics of a moving object, such as a person's face recognition and gender, from the video captured by the camera as an alarm function. Such a safety monitoring device 10 can improve the detection accuracy of object attributes because it can set the conditions for determining the attributes of a moving object in detail.

[0220] (17) The safety monitoring device 10 according to the present embodiment may be configured so that the process of detecting a moving object includes any one of person identification by face recognition, gender determination, and posture determination.

[0221] The safety monitoring device 10 according to this embodiment allows detailed setting of conditions for determining the attributes of a moving object, and therefore allows for improved accuracy in detecting the attributes of an object.

[0222] (18) The safety monitoring device 10 of this embodiment may be configured to notify warning data detected by the configured alarm function to an external device (Web terminal 62, surveillance camera 61z, etc.) equipped with a program for processing camera images.

[0223] The external device (Web terminal 62, surveillance camera 61z, etc.) may manage events and video data independently of the operation of the safety monitoring device 10. In addition, it may be easier to manage events and video data if the program executed by the safety monitoring device 10 and the program displaying the camera image are configured separately. Therefore, the safety monitoring device 10 according to the present embodiment may be configured to notify an external device equipped with a program for performing camera image processing of warning data detected by a set alarm function. In this way, the safety monitoring device 10 can, for example, cause the external device to perform camera image processing and create an image in which an image of warning data is superimposed on the camera image. At that time, the safety monitoring device 10 may convert the position data (three-dimensional data) of the moving object detected by the safety monitoring device 10 into two-dimensional data of the camera image, and cause the external device to create an image so that the position of the camera image corresponds to the position of the image of the warning data.

[0224] (19) As shown in FIGS. 19 to 22, the safety monitor device 10 according to this embodiment may be configured to include a smoothing means (angle smoothing unit 16d) that smoothes the angle of the bounding box.

[0225] If a bounding box is generated for each frame (e.g., every 0.1 seconds), the angle θ obtained varies due to the variation in the distance measurement data, and as a result, there is a possibility that the variation in the distance between two objects (e.g., between a person and a car) may become large when the distance between the two objects is obtained. Therefore, the safety monitoring device 10 according to this embodiment can stabilize the change in the angle θ accompanying the movement of the car by smoothing the angle of the bounding box using a smoothing means (angle smoothing unit 16d). Therefore, the safety monitoring device 10 can stably obtain the distance between two objects (e.g., between a person and a car). Furthermore, when displaying an image showing a moving object on the display 42 (display unit), the safety monitoring device 10 can display an image with visually natural movement.

[0226] (20) The safety monitoring device 10 according to this embodiment has an image processing unit 18 that performs processing on the three-dimensional data according to the type of the alarm function. As shown in Fig. 19 to Fig. 22, the image processing unit 18 of the safety monitoring device 10 according to this embodiment may be configured to simultaneously display the point cloud data and the bounding box on the display 42 (display unit).

[0227] The safety monitoring device 10 according to this embodiment can simultaneously display the point cloud data after the clustering process and the bounding box, allowing the safety monitor to visually recognize what kind of point cloud the actual measurement data is. Furthermore, in the unlikely event that an accident occurs, the safety monitoring device 10 can allow the safety monitor to easily grasp the situation at the accident site by simultaneously visually checking the point cloud data and the bounding box of the accident site.

[0228] (21) As shown in FIGS. 19 to 22, the smoothing means (angle smoothing unit 16d) of the safety monitor device 10 according to this embodiment may be configured to smooth and calculate the angle of the bottom face of the bounding box.

[0229] In generating a bounding box, the angles are discrete, so that increasing the resolution increases the amount of calculations and extends the calculation time. Therefore, the smoothing means (angle smoothing unit 16d) of the safety monitoring device 10 according to this embodiment may be configured to smooth and calculate the angle of the bottom surface of the bounding box. The safety monitoring device 10 can calculate the angle with an appropriate amount of calculations by smoothing the discrete angles of the bottom surface of the bounding box. Therefore, the safety monitoring device 10 can reduce the amount of calculations and shorten the calculation time. The amount of calculations when generating a bounding box is determined by the number of point groups and the number of iterations n of calculations using the discrete angle Δθ. The calculation time is determined by the performance of the CPU (central processing unit) of the personal computer (PC) constituting the safety monitoring device 10.

[0230] (22) As shown in Figures 19 to 22, the angle of the bottom surface of the bounding box discretely calculated by the smoothing means (angle smoothing unit 16d) of the safety monitoring device 10 of this embodiment may be configured to be arbitrarily changed via a user interface.

[0231] The safety monitoring device 10 according to this embodiment provides a user interface so that, for example, an optimal discrete angle Δθ relative to the angle θ of the bottom surface of the bounding box can be determined, and the value of the discrete angle Δθ can be selected.

[0232] (23) As shown in Figures 20 and 21, either or both of the smoothing method and the smoothing time value of the smoothing means (angle smoothing unit 16d) of the safety monitoring device 10 in this embodiment may be configured to be arbitrarily changeable via a user interface.

[0233] The angle smoothing process is performed using data from the present time to the past. In this case, the safety monitoring device 10 according to the present embodiment can select, for example, a method of simply performing an average, a method of performing a weighted average in which a larger weight is assigned to the data closer to the present time, or the like. This allows the safety monitoring device 10 to perform angle smoothing process suitable for the three-dimensional data input from the lidar 51. Note that smoothing a larger amount of data can improve the smoothness of the image, but the responsiveness of the calculation to the change in the input data decreases. Therefore, the safety monitoring device 10 may be configured to select a suitable time according to the input data, taking the responsiveness into consideration.

[0234] (24) The smoothing means (angle smoothing unit 16d) of the safety monitoring device 10 according to this embodiment may be configured to smooth the center coordinates of the bottom surface of the bounding box.

[0235] Point cloud data varies from frame to frame. Therefore, not only the angle but also the center coordinates of the bounding box (center coordinates of the rectangle on the bottom surface) vary. Smoothing the center coordinates of the bounding box is effective in suppressing the variation. Therefore, the smoothing means (angle smoothing unit 16d) of the safety monitoring device 10 according to this embodiment may be configured to smooth the center coordinates of the bottom surface of the bounding box. Note that, although the center coordinates of the bounding box have been described here as the center coordinates of the rectangle on the bottom surface, the center coordinates of the bounding box can be processed as the center coordinates of a rectangular parallelepiped region (rectangular region). In this case, smoothing the center point of the rectangular parallelepiped region (rectangular region) is also effective.

[0236] 23 to 28, the safety monitoring device 10 according to this embodiment has a record management unit 19 that divides video data into files at predetermined time intervals and records the files in the storage unit 30, and an event list processing unit 23 that creates and manages an event list that can be selected for each event. The event list processing unit 23 may be configured to perform control so as not to automatically delete files that include video within the video playback time range of an event selected by a safety monitor, or to perform control so as to re-save the files in another area of ​​the storage unit 30.

[0237] The safety monitoring device 10 according to this embodiment has data in each event that indicates the video playback time range, and can specify the file (recorded data) whose video should be played from the data. The safety monitoring device 10 is configured to determine whether or not the file (recorded data) is one in which an incident (important event) has occurred, and select not to automatically delete the file if it is determined that the file should be left. This allows the safety monitoring device 10 to manage the file so that it does not delete the file in which an incident (important event) has occurred, even if the file (recorded data) is set to be automatically deleted based on a set rule. By leaving the file in which an incident (important event) has occurred, the safety monitoring device 10 can accumulate data on similar incidents, allowing the safety monitor to consider accident prevention measures.

[0238] (26) As shown in Fig. 28, the event list processing unit 23 of the safety monitoring device 10 according to this embodiment has a function of automatically deleting files based on a predetermined rule. The predetermined rule may be configured to be either that a certain amount of free space in the recording means (the storage unit 30 or an external recording medium not shown) is secured or that a certain amount of time has passed.

[0239] The safety monitoring device 10 according to this embodiment can automatically erase files based on a predetermined rule. When automatically erasing files based on a predetermined rule, it is better to manage the safety monitoring device 10 so that a certain amount of free space is secured in a configuration that fully utilizes the capacity of the HDD as a recording means. Also, in a configuration in which the safety monitoring device 10 automatically erases files after a certain time has elapsed within a range where the capacity of the HDD as a recording means is ample, it is clear how much data remains and data can be easily managed, so it may be used according to the purpose.

[0240] (27) As shown in FIG. 26, the event list processing unit 23 of the safety monitoring device 10 of this embodiment may be configured to acquire sensor data including one or both of the start and end times of an event that has occurred in the detection area from the safety sensor and determine the video playback time range of the event list.

[0241] The safety monitoring device 10 according to this embodiment can play files (video data) within a video playback time range determined for each event, and can therefore play files (video data) for a suitable time. The safety monitoring device 10 can set, for example, the time when a person enters an area where a warning is issued as the start time of an event, and the time when the person exits the area where a warning is issued as the end time of the event list. The safety monitoring device 10 can manage whether or not a person who has entered a dangerous area has evacuated from the dangerous area to a safe area. The safety monitoring device 10 according to this embodiment may be configured to record data on either or both of the start and end times of an event, depending on the application.

[0242] (28) As shown in Figures 23 to 28, the event list processing unit 23 of the safety monitoring device 10 in this embodiment may be configured to have a function of additionally registering any event that occurred in the past as a new event in the event list.

[0243] The safety monitoring device 10 according to this embodiment can add any event that occurred in the past as a new event to the event list at the discretion of the safety monitor. For example, even if the safety monitor 10 has missed an event in the past, if the safety monitor judges that an incident (important event) has occurred by looking at the video of the file left in the recording means, the safety monitoring device 10 can add the incident to the event list as a new event. In this case, the safety monitoring device 10 leaves the file (recorded data) in which the incident (important event) has occurred without erasing it.

[0244] (29) As shown in FIGS. 23 to 28, the event list processing unit 23 of the safety monitoring device 10 according to this embodiment may be configured to have a function of editing the start time and end time of an event.

[0245] The safety monitoring device 10 according to this embodiment can arbitrarily edit the start time and end time of an event (i.e., the playback start time and playback end time of a file related to the event). For example, it is possible that an incident (important event) occurs before a person enters a dangerous area, or that a dangerous situation continues to occur even after a person who entered the dangerous area leaves the dangerous area. In such a case, the safety monitoring device 10 can change the start time and / or end time of the event to leave the files for that time period.

[0246] (30) The safety monitoring device 10 according to this embodiment has a video processing unit 18 that performs processing on the video data of the detection target area according to the type of alarm function. As shown in Fig. 23 to Fig. 28, the video processing unit 18 of the safety monitoring device 10 according to this embodiment may be configured to create a display screen that displays a video and an event list simultaneously on the display 42 (display unit), and when a specific event is designated from the event list by a safety monitor via a graphical user interface, to display a video including the video playback time range of the designated event on the display 42 (display unit).

[0247] The safety monitoring device 10 according to this embodiment can play back video of any event based on the event list. A safety monitor can efficiently check the degree of danger at the site by watching the played back video. By simultaneously displaying the video and the event list on the display 42 (display unit), the safety monitoring device 10 can facilitate event management and improve the efficiency of operations such as event display and event editing.

[0248] (31) As shown in Figures 23 to 28, in the safety monitoring device 10 of this embodiment, the event list may be configured so that an importance mark can be added to each event via a graphical user interface to select whether or not to automatically delete the file.

[0249] The safety monitoring device 10 according to this embodiment can add an importance mark to each event. Therefore, the safety manager can easily change the setting of each event from automatic deletion to no automatic deletion by adding an importance mark to each event. This allows the safety manager to easily keep files in which an incident (important event) has occurred.

[0250] (32) As shown in Figures 23 to 28, the event list processing unit 23 of the safety monitoring device 10 of this embodiment may be configured to control so that files that include an event for which no automatic deletion is selected when selecting whether or not to automatically delete a file and an event for which an importance mark has been added in advance, the period from a certain time before the start time to a certain time after the end time, are not automatically deleted.

[0251] The safety monitoring device 10 according to this embodiment can control not to automatically delete files of events for which no automatic deletion has been selected and files of events for which an important mark has been added in advance. The safety monitoring device 10 can add or delete important marks from the current time until the time when the file is to be automatically deleted. The safety monitoring device 10 can also change the time of each event from the current time until the time when the file is to be automatically deleted.

[0252] (33) As shown in Figures 23 to 28, the event list processing unit 23 of the safety monitoring device 10 of this embodiment may be configured to control erasure of an event file when an event for which no automatic deletion was selected is changed to an event for which automatic deletion is selected, or when an event to which an importance mark has been previously added is removed from the event.

[0253] The safety monitoring device 10 according to this embodiment can control erasure of the file of an event when the selection of the event is changed to one with automatic erasure or when the importance mark is removed. This allows the safety monitoring device 10 to increase the free space of the HDD. That is, the safety monitoring device 10 is configured to leave the video data file related to the event with the importance mark attached, without erasing it, even if a predetermined time has passed since the event was recorded. However, when the safety monitor checks the risk of each event based on the event list, even if the event has been marked with the importance mark, it may be determined that the risk is low. In such a case, the safety monitoring device 10 removes the importance mark, thereby enabling the video data file related to the event with the importance mark removed to be erased. This allows the safety monitoring device 10 to increase the free space of the HDD.

[0254] (34) As shown in Figures 23 to 28, the event list processing unit 23 of the safety monitoring device 10 of this embodiment may be configured to delete the event of a file from the event list when a file that has been recorded for a predetermined time is automatically deleted, while controlling so that events for which no automatic deletion was selected when selecting whether or not to automatically delete a file and events to which an importance mark has been added in advance are not deleted from the event list.

[0255] The safety monitoring device 10 according to this embodiment can prevent events for which no automatic deletion has been selected and events marked with an importance mark from being deleted by mistake. Note that the event list and the video data file have a one-to-one correspondence. Therefore, an event corresponding to a video data file is deleted from the event list at the same time as the video data file to be deleted is deleted. This prevents the safety monitor from accidentally deleting an event that should not be deleted.

[0256] (35) The safety monitoring device 10 according to this embodiment has a safety monitoring unit 22 that monitors the entry of an object into a monitoring area of ​​the safety sensor. The safety monitoring unit 22 may be configured to set the time when an object enters the monitoring area as the start time of an event, and set the time when the object leaves the monitoring area or a time when a certain time has elapsed since the start time of the event as the end time of the event, and notify the event list processing unit 23 of the start time and end time of the event.

[0257] The safety monitoring device 10 according to this embodiment can automatically detect the start and end times of an event and notify the event list processing unit 23. The safety monitoring device 10 can monitor dangerous areas in three dimensions, day or night, by using, for example, a lidar 51 as a safety sensor.

[0258] As described above, the safety monitoring device 10 according to this embodiment can adequately grasp the relationship between a person and an object.

[0259] The present invention is not limited to the above-described embodiment, and various changes and modifications can be made without departing from the gist of the present invention.

[0260] For example, the above-described embodiment has been described in detail to clearly explain the gist of the present invention. Therefore, the present invention is not necessarily limited to having all of the components described. In addition, the present invention can be modified by adding other components to certain components, or by replacing some components with other components. In addition, the present invention can be modified by deleting some components.

[0261] Further, for example, in the above-described embodiment, the safety monitoring device 10 monitors the safety of an area based on three-dimensional data, but it is possible to monitor the safety of an area based on two-dimensional data. [Explanation of symbols]

[0262] 10 Safety monitoring device 11 Control section 12 Area setting section 13. Alarm type setting section 14 Video acquisition section 15 LIDAR data acquisition unit (sensor data acquisition unit) 16 Motion Recognition Analysis Unit 16a Foreground data extraction section 16b Clustering processing section 16c Bounding box generator 16d Angle smoothing part (smoothing means) 16e,18a Image recognition section 16f Alert Analysis Department 17 Result output section 18 Video Processing Section 19 Records Management Department 20 Video display processing unit 21 Safety sensor input section 22 Safety Monitoring Department 23 Event list processing section 30 Storage section 31 Setting record data 32 Video recording data 33 Processing image data 39 Communications Department 41 Operation section 42 Display (display unit) 43 Warning Lamp 44 Speaker (sound reporting section) 50 Safety Sensors 51 Rider 61 Surveillance camera (filming section) 61z Surveillance camera (photographing unit, external device) 61a Setting reception section 61b Setting processing section 61c Notification Department 62 Web terminal (external device) 62a Notification acquisition part 100,110 display screen 101 3D Viewer Section 101a Notification area frame 102 Setting section 103 Lidar Selection Section 104 Notification Area Tab 104a Block Tab 104b Enabled checkbox 104c Area selection section 104d Display color setting section 105 Moving Objects 105a large object 105box,106box Bounding box 106 people 106o center point 106r radius 111 Video display unit 112 Event list display section 120 Event List 121 Important Marking Column 122 Timestamp field 123 Camera name field 124 Type column 125 Description 130 Event Editing Screen 131 Camera name field 132 Username field 133 Date and time of occurrence 134 End date and time column 135 Description 136 Type column 137 Mark column 138 Cause column 139 Countermeasures Column 140 Importance column A51 Detection area A61a, A61b Alert Area A62 Exclusion Area B61a, B61b Report location B62 Exclusions D Depth G ground H, Hb, Ht height Pr control program R fixed distance R11 Walking route S Safety monitoring system Su1,Su2,Su3,Su4,Su5,Su6,Su7,Su8 side W width

Claims

1. an area setting unit capable of setting a plurality of alarm target areas within a detection target area of ​​the safety sensor; an alarm type setting unit capable of setting an arbitrary type of alarm function for each of the alarm target areas; a moving object recognition and analysis unit that recognizes a motion of a moving object based on sensor data measured by the safety sensor, The alarm type setting unit is The ability to issue warnings for all moving objects, A function that warns only people, A function that issues a warning when the distance between a person and a large object larger than a person becomes within a certain distance; A function of issuing a warning when a person enters the direction of travel of the large object; A function that issues a warning only to moving objects moving at a certain speed or faster, A function that issues a warning only to moving objects moving at a certain speed or less, A function that combines two or more arbitrary functions and takes a logical OR. Any two or more functions can be selected as the types of the alarm function. Safety monitoring device.

2. An area setting unit capable of setting a plurality of alarm target areas within a detection target area of ​​a safety sensor; an alarm type setting unit capable of setting an arbitrary type of alarm function for each of the alarm target areas; a moving object recognition and analysis unit that recognizes a motion of a moving object based on sensor data measured by the safety sensor, The moving object recognition and analysis unit Extracting point cloud data representing a moving object from the three-dimensional data measured by the lidar as the safety sensor, and creating a bounding box of a size necessary to surround the point cloud data; Using any one or more or all of the five pieces of data, namely the width, depth, and height of the bounding box, as well as the height of the bottom surface and the height of the top surface of the bounding box, determine the minimum and maximum values ​​of each as the size of the person; The size of the moving object is set to be within a determined range as a condition for determining that the moving object is a human, and when the size of the moving object continues to meet the determination condition for a certain period of time that is arbitrarily determined, the moving object is identified as a human. Safety monitoring device.

3. An area setting unit capable of setting a plurality of alarm target areas within a detection target area of ​​a safety sensor; an alarm type setting unit capable of setting an arbitrary type of alarm function for each of the alarm target areas; a moving object recognition and analysis unit that recognizes a motion of a moving object based on sensor data measured by the safety sensor; a record management unit that records and manages video data obtained by capturing all or a part of the detection target area in a storage unit; an event list processing unit that adds an event as the alert data to an event list when the alert is issued, and manages the event list; the recording management unit constantly records the video data in the storage unit, When an arbitrary alarm location is designated from the event list, the event list processing unit displays on a display unit image data for a certain period of time before and after the designated alarm location. Safety monitoring device.

4. An area setting unit capable of setting a plurality of alarm target areas within a detection target area of ​​a safety sensor; an alarm type setting unit capable of setting an arbitrary type of alarm function for each of the alarm target areas; a moving object recognition and analysis unit that recognizes a motion of a moving object based on sensor data measured by the safety sensor; a record management unit that divides the video data into files at predetermined time intervals and records the files in a storage unit; an event list processing unit that creates and manages an event list that can be selected for each event; The event list processing unit performs control for an event selected by a safety observer so as not to automatically delete a file including a video within a video playback time range of the event, or performs control to re-store the file in another area of ​​the storage unit. Safety monitoring device.

5. a sensor data acquisition unit that acquires three-dimensional data as the sensor data from the safety sensor, The moving object recognition analysis unit extracts moving objects from the detection target areas based on the three-dimensional data, and detects the progress of each moving object into the alert target areas. A safety monitoring device according to any one of claims 1 to 4.

6. The type of alarm function can be specified for each area via a user interface. A safety monitoring device according to any one of claims 1 to 5.

7. The moving object recognition and analysis unit Identifying a large object larger than a person using any one or more or all of the five pieces of data: the width, depth, and height of the bounding box, as well as the height of the bottom surface and the height of the top surface of the bounding box; Calculating the distance between the person and the large object by subtracting the radius of the circumscribing circle of the bottom or top surface of the person's bounding box from the minimum distance between the center of the bottom or top surface of the person's bounding box and each side of the bottom or top surface of the large object's bounding box; A warning is issued when the distance between the person and the large object becomes smaller than a certain distance that is arbitrarily determined. The safety monitoring device according to claim 2.

8. The moving object recognition analysis unit issues a warning when the time during which the distance between the person and the large object becomes smaller than the certain distance continues for an arbitrarily set certain period of time. A safety monitoring device according to claim 7.

9. The moving object recognition analysis unit stops the warning when the distance between the person and the large object becomes greater than the certain distance. A safety monitoring device according to claim 7 or 8.

10. The moving object recognition analysis unit stops the warning when the time during which the distance between the person and the large object is greater than the certain distance continues for an arbitrarily determined certain period or more. A safety monitoring device according to claim 7 or 8.

11. The moving object recognition analysis unit changes an alarm level according to a distance between the person and the large object. A safety monitoring device according to any one of claims 7 to 10.

12. The moving object recognition analysis unit calculates the speed of the moving object from the distance that the center position of the bounding box moves in a certain period of time. A safety monitoring device according to any one of claims 2 and 7 to 11.

13. A video processing unit that superimposes the position of a moving object detected by the safety sensor on video data. A safety monitoring device according to any one of claims 1 to 12.

14. The event list processing unit can filter and display the event data for each of the types in the event list displayed on the display unit. The safety monitoring device according to claim 3.

15. Further, a record management unit is provided for recording and managing the image data of the detection target area in a storage unit, The record management unit records the video data before and after a base point, based on a time point at which an event that is the subject of a warning occurs, in the storage unit according to the type of the alarm function. A safety monitoring device according to any one of claims 1 to 13.

16. The alarm type setting unit can specify, as the alarm function, a process of detecting characteristics of the moving object from video data of the detection target area. A safety monitoring device according to any one of claims 1 to 15.

17. The process of detecting a moving object includes one of person identification by face recognition, gender determination, and posture determination.

17. A safety monitoring device according to claim 16.

18. The safety monitoring device notifies an external device having a camera image processing program installed therein of warning data detected by the set alarm function. A safety monitoring device according to any one of claims 1 to 17.

19. A smoothing means for smoothing the angles of the bounding box is provided. A safety monitoring device according to any one of claims 2 and 7 to 12.

20. a video processing unit that processes the three-dimensional data according to a type of the alarm function; The image processing unit displays the point cloud data and the bounding box simultaneously on a display unit.

20. A safety monitoring device according to claim 19.

21. The smoothing means calculates the angle of the bottom surface of the bounding box by smoothing the angle. A safety monitoring device according to claim 19 or 20.

22. The angle of the bottom surface of the bounding box calculated discretely by the smoothing means can be arbitrarily changed via a user interface.

22. A safety monitoring device according to claim 21.

23. Either or both of the smoothing method and the smoothing time value of the smoothing means can be arbitrarily changed via a user interface. A safety monitoring device according to any one of claims 19 to 22.

24. The smoothing means smoothes the center coordinates of the bottom surface of the bounding box. A safety monitoring device according to any one of claims 19 to 23.

25. the event list processing unit has a function of automatically deleting the file based on a predetermined rule, The predetermined rule is either that a certain amount of free space in the recording means is secured or that a certain amount of time has passed. A safety monitoring device according to claim 4.

26. The event list processing unit acquires sensor data including one or both of a start time and an end time of an event that has occurred in the detection target area from the safety sensor, and determines the video playback time range of the event list. A safety monitoring device according to claim 4 or claim 25.

27. The event list processing unit has a function of additionally registering any event that has occurred in the past as a new event in the event list. A safety monitoring device according to any one of claims 4, 25 and 26.

28. The event list processing unit has a function of editing the start time and end time of the event. A safety monitoring device according to any one of claims 4, 25 to 27.

29. a video processing unit that processes the video data of the detection target area according to a type of the alarm function; The video processing unit creates a display screen that simultaneously displays the video and the event list on a display unit, and when a specific event is designated from the event list by a safety monitor via a graphical user interface, displays a video including a video playback time range of the designated event on the display unit. A safety monitoring device according to any one of claims 4, 25 to 28.

30. The event list allows the user to mark each event as important so that the user can select whether or not to automatically delete the file via the graphical user interface.

30. A safety monitoring device according to claim 29.

31. The event list processing unit performs control not to automatically delete files including an event for which no automatic deletion is selected at a timing for selecting whether or not to automatically delete the file and a period from a certain time before the start time to a certain time after the end time of an event to which the importance mark has been added in advance.

31. A safety monitoring device according to claim 30.

32. The event list processing unit performs control to erase a file of an event when the event for which no automatic deletion was selected is changed to the event for which automatic deletion is selected or when the importance mark is removed from the event to which the importance mark has been added in advance.

32. A safety monitoring device according to claim 31.

33. The event list processing unit deletes an event of the file from the event list when a file that has been recorded for a predetermined time is automatically deleted, while controlling not to delete from the event list an event for which no automatic deletion has been selected at a timing for selecting whether or not to automatically delete the file and an event to which the importance mark has been added in advance. A safety monitoring device according to any one of claims 30 to 32.

34. The safety sensor further includes a safety monitoring unit that monitors the intrusion of an object into a monitoring area of ​​the safety sensor, The safety monitoring unit determines a time when the object enters the monitoring area as a start time of the event, and determines a time when the object leaves the monitoring area or a time when a certain time has elapsed from the start time of the event as an end time of the event, and notifies the event list processing unit of the start time and end time of the event. A safety monitoring device according to any one of claims 4, 25 to 33.

35. an area setting step capable of setting a plurality of alarm target areas within a detection target area of ​​the safety sensor; an alarm type setting step for setting an arbitrary type of alarm function for each of the alarm target areas; A moving object recognition and analysis step of recognizing a motion of a moving object based on sensor data measured by the safety sensor, In the alarm type setting step, The ability to issue warnings for all moving objects, A function that warns only people, A function that issues a warning when the distance between a person and a large object larger than a person becomes within a certain distance; A function of issuing a warning when a person enters the direction of travel of the large object; A function that issues a warning only to moving objects moving at a certain speed or faster, A function that issues a warning only to moving objects moving at a certain speed or less, A function that combines two or more arbitrary functions and takes a logical OR. Any two or more functions can be selected as the types of the alarm function. Safety monitoring methods.

36. An area setting step capable of setting a plurality of alarm target areas for a detection target area of ​​a safety sensor; an alarm type setting step for setting an arbitrary type of alarm function for each of the alarm target areas; A moving object recognition and analysis step of recognizing a motion of a moving object based on sensor data measured by the safety sensor, In the moving object recognition and analysis step, Extracting point cloud data representing a moving object from the three-dimensional data measured by the lidar as the safety sensor, and creating a bounding box of a size necessary to surround the point cloud data; Using any one or more or all of the five pieces of data, namely the width, depth, and height of the bounding box, as well as the height of the bottom surface and the height of the top surface of the bounding box, determine the minimum and maximum values ​​of each as the size of the person; The size of the moving object is set to be within a determined range as a condition for determining that the moving object is a human, and when the size of the moving object continues to meet the determination condition for a certain period of time that is arbitrarily determined, the moving object is identified as a human. Safety monitoring methods.

37. An area setting step capable of setting a plurality of alarm target areas within a detection target area of ​​a safety sensor; an alarm type setting step for setting an arbitrary type of alarm function for each of the alarm target areas; a moving object recognition and analysis step of recognizing a motion of a moving object based on sensor data measured by the safety sensor; a recording and managing step of recording and managing video data of all or a part of the detection target area in a storage unit; an event list processing step of adding an event as the alert data to an event list when the alert is issued, and managing the event list; In the recording management step, the video data is constantly recorded in the storage unit, When an arbitrary alarm location is designated from the event list, video data for a certain period of time before and after the alarm location designated in the event list processing step is displayed on a display unit. Safety monitoring methods.

38. An area setting step capable of setting a plurality of alarm target areas for a detection target area of ​​a safety sensor; an alarm type setting step for setting an arbitrary type of alarm function for each of the alarm target areas; a moving object recognition and analysis step of recognizing a motion of a moving object based on sensor data measured by the safety sensor; a recording management step of dividing the video data into files at predetermined time intervals and recording the files in a storage unit; An event list processing step of creating and managing a selectable event list for each event, In the event list processing step, for an event selected by a safety observer, a control is performed to prevent automatic deletion of a file including a video within a video playback time range of the event, or to re-store the file in another area of ​​the storage unit. Safety monitoring methods.

39. A program that causes a computer to function as a safety monitor that ensures safety in a detection target area of ​​a safety sensor, The computer, an area setting unit capable of setting a plurality of alarm target areas within a detection target area of ​​the safety sensor; an alarm type setting unit capable of setting an arbitrary type of alarm function for each of the alarm target areas; a moving object recognition and analysis unit that recognizes the movement of a moving object based on sensor data measured by the safety sensor; The alarm type setting unit, The ability to issue warnings for all moving objects, A function that warns only people, A function that issues a warning when the distance between a person and a large object larger than a person becomes within a certain distance; A function of issuing a warning when a person enters the direction of travel of the large object; A function that issues a warning only to moving objects moving at a certain speed or faster, A function that issues a warning only to moving objects moving at a certain speed or less, A function that combines two or more arbitrary functions and takes a logical OR. Any two or more of the functions can be selected as the types of the alarm function. program.

40. A program for causing a computer to function as a safety monitor for ensuring safety in an area to be detected by a safety sensor, comprising: The computer, an area setting unit capable of setting a plurality of alarm target areas within a detection target area of ​​the safety sensor; an alarm type setting unit capable of setting an arbitrary type of alarm function for each of the alarm target areas; a moving object recognition and analysis unit that recognizes the movement of a moving object based on sensor data measured by the safety sensor; The moving object recognition and analysis unit, extracting point cloud data representing a moving object from the three-dimensional data measured by the lidar serving as the safety sensor, and creating a bounding box of a size necessary to surround the point cloud data; Using any one or more or all of the five data items of the width, depth, and height of the bounding box, as well as the height of the bottom surface and the height of the top surface of the bounding box, a minimum value and a maximum value of each of the data items are determined as the size of a person; The size of the moving object is set to be within a determined range as a condition for determining that the moving object is a human, and when the size of the moving object continues to meet the condition for a certain period of time that is arbitrarily determined, the moving object is identified as a human. program.

41. A program for causing a computer to function as a safety monitor for ensuring safety in an area to be detected by a safety sensor, comprising: The computer, an area setting unit capable of setting a plurality of alarm target areas within a detection target area of ​​the safety sensor; an alarm type setting unit capable of setting an arbitrary type of alarm function for each of the alarm target areas; a moving object recognition and analysis unit that recognizes a motion of a moving object based on sensor data measured by the safety sensor; a record management unit that records and manages video data obtained by capturing all or a part of the detection target area in a storage unit; functioning as an event list processing unit that, when a warning is issued, adds an event as the issued data to an event list and manages the event; The recording management unit constantly records the video data in the storage unit, When an arbitrary alarm location is specified from the event list, video data for a certain period of time before and after the alarm location specified by the event list processing unit is displayed on a display unit. program.

42. A program for causing a computer to function as a safety monitor for ensuring safety in an area to be detected by a safety sensor, comprising: The computer, an area setting unit capable of setting a plurality of alarm target areas within a detection target area of ​​the safety sensor; an alarm type setting unit capable of setting an arbitrary type of alarm function for each of the alarm target areas; a moving object recognition and analysis unit that recognizes a motion of a moving object based on sensor data measured by the safety sensor; a record management unit that divides the video data into files at predetermined time intervals and records the files in a storage unit; an event list processing unit that creates and manages a selectable event list for each event; The event list processing unit is caused to perform either of control for not automatically deleting a file including a video within a video playback time range of an event selected by a safety observer, or control for re-saving the file in another area of ​​the storage unit. program.

Images