Safety support systems, safety support methods, and programs

Abstract

To make it easier for safety managers to check the site conditions. [Solution] The safety support system 100 includes an object detection unit 13 that detects moving images of objects that meet detection conditions from among multiple video sources MI10 of each detection area 200 captured by multiple imaging devices (cameras 110) registered in a database 30; an alarm condition setting unit 12 that registers detection conditions; a detection moving image extraction unit 17 that extracts and saves moving images of objects that meet detection conditions as detection moving images MI11 from among the video sources detected by the detection conditions; and an extracted moving image viewing unit 18 that displays multiple detection moving images of the same detection area on a display device 3a in synchronization with the time of imaging.

Description

Technical Field

[0001] The present invention relates to a safety support system, a safety support method, and a program.

Background Art

[0002] For example, at the work site of a manufacturing factory, a plurality of vehicles such as forklifts are running, and a plurality of workers are moving. Since vehicles such as forklifts have many blind spots, they may not be able to visually recognize workers. Therefore, there is a possibility of contact accidents between vehicles and workers. At such a work site in a manufacturing factory, there is a demand for preventing contact accidents between vehicles and workers and providing teaching materials suitable for preventing contact accidents. Therefore, for example, a recognition device described in Patent Document 1 and an image processing device described in Patent Document 2 have been proposed.

[0003] The recognition device described in Patent Document 1 is "a recognition device for a signal of a vehicle's direction indicator, including a first detection unit that sequentially performs vehicle detection for each frame in an input frame sequence, where the frame sequence includes at least one frame period; a tracking unit that performs vehicle tracking based on the vehicle detection results of each frame and obtains a vehicle tracking result; a second detection unit that performs lamp detection based on the vehicle detection results of each frame and obtains a lamp detection result; a third detection unit that performs detection of the display direction of the vehicle based on the vehicle detection results of each frame and obtains a vehicle display direction detection result; a matching unit that performs pairing of lamps based on the vehicle tracking result, the lamp detection result, and the vehicle display direction detection result and obtains a lamp pairing result; an extraction unit that extracts luminance for the area where the lamp is located based on the lamp pairing result and obtains a luminance extraction result; and a fourth detection unit that performs detection of a signal of the vehicle's direction indicator based on the luminance extraction result and obtains a detection result of the vehicle's direction indicator signal."

[0004] Furthermore, the image processing apparatus described in Patent Document 2 is described as "an image processing apparatus having a posture information acquisition unit that extracts a first person appearing in image data and acquires posture information relating to the posture of the first person; a feature information acquisition unit that acquires feature information relating to the features of the image data; and an identification unit that identifies the action being performed by the first person and the relationship between the first person and a first item that is the target of the action being performed by the first person, based on the posture information and the feature information." [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2021-149950 [Patent Document 2] Japanese Patent Publication No. 2022-83232 [Overview of the project] [Problems that the invention aims to solve]

[0006] However, the prior art described in Patent Documents 1 and 2 does not take into consideration how to make it easier for safety managers responsible for safety at the work site to confirm on-site conditions such as the circumstances under which a contact accident occurred (or nearly occurred). Therefore, in this respect, there is room for improvement in terms of convenience in the prior art described in Patent Documents 1 and 2.

[0007] This invention was made to solve the aforementioned problems, and its main objective is to provide a safety support system, a safety support method, and a program that make it easier for safety managers to check on-site conditions. [Means for solving the problem]

[0008] To achieve the above objective, the present invention provides a safety support system comprising: an object detection unit that detects moving images showing objects corresponding to detection conditions from among multiple video sources of each detection area captured by multiple imaging devices registered in a database; an alarm condition setting unit that registers the detection conditions; a detection moving image extraction unit that extracts and saves moving images showing objects corresponding to the detection conditions from among the video sources detected by the detection conditions as detection moving images; and an extracted moving image viewing unit that displays multiple detection moving images of the same detection area on a display device in synchronization with the time of imaging. Other methods will be described later. [Effects of the Invention]

[0009] According to the present invention, it is possible to make it easier for safety managers to check the on-site conditions. [Brief explanation of the drawing]

[0010] [Figure 1] This is a diagram showing the overall configuration of the safety support system according to the embodiment. [Figure 2] This is a hardware configuration diagram showing an example of a computer that implements a safety support system. [Figure 3] This is an explanatory diagram showing an example of an application program used in a safety support system. [Figure 4] This is a flowchart illustrating the general operation of the safety support system. [Figure 5] This is an operation diagram illustrating the general operation of the safety support system. [Figure 6A] This is a flowchart showing the object detection operation of the safety support system. [Figure 6B] This is a flowchart showing the detection and video extraction operation of the safety support system. [Figure 6C] This is a flowchart showing the operation of viewing extracted video footage from the safety support system. [Figure 7] This is an operation diagram (1) illustrating the general operation of the video source management unit. [Figure 8]It is an operation explanatory diagram (2) showing the schematic operation of the video source management unit. [Figure 9] It is an operation explanatory diagram (1) showing the schematic operation of the transmission condition setting unit. [Figure 10] It is an operation explanatory diagram (2) showing the schematic operation of the transmission condition setting unit. [Figure 11] It is an operation explanatory diagram showing the schematic operation of the object detection unit. [Figure 12] It is an operation explanatory diagram (1) showing the schematic operation of the alarm transmission unit. [Figure 13] It is an operation explanatory diagram (2) showing the schematic operation of the alarm transmission unit. [Figure 14] It is an operation explanatory diagram (1) showing the schematic operation of the transmission status monitoring unit. [Figure 15] It is an operation explanatory diagram (2) showing the schematic operation of the transmission status monitoring unit. [Figure 16] It is an operation explanatory diagram showing the schematic operation of the detection information recording unit. [Figure 17] It is an operation explanatory diagram showing the schematic operation of the detected moving image extraction unit. [Figure 18] It is an operation explanatory diagram (1) showing the schematic operation of the extracted moving image viewing unit. [Figure 19] It is an operation explanatory diagram (2) showing the schematic operation of the extracted moving image viewing unit. [Figure 20] It is an operation explanatory diagram (3) showing the schematic operation of the extracted moving image viewing unit. [Figure 21] It is an explanatory diagram (1) showing an example of the viewing screen. <​​​​​​​​​​​​​​​​​​This is a flowchart illustrating the general operation of posture determination. [Figure 29] This is an explanatory diagram showing an example of prediction based on the direction of movement. [Figure 30A] This is an explanatory diagram (1) illustrating an example of a situation that is highly likely to be applicable. [Figure 30B] This is an explanatory diagram (2) illustrating an example of a situation that is highly likely to be applicable. [Figure 30C] This is an explanatory diagram (3) illustrating an example of a situation that is highly likely to be applicable. [Figure 30D] This is an explanatory diagram (4) illustrating an example of a situation that is highly likely to be applicable. [Modes for carrying out the invention]

[0011] Hereinafter, embodiments of the present invention (hereinafter referred to as "these embodiments") will be described in detail with reference to the drawings. Note that each figure is merely a schematic representation to the extent necessary for a thorough understanding of the present invention. Therefore, the present invention is not limited to the illustrated examples. Furthermore, in each figure, common or similar components are denoted by the same reference numerals, and their redundant descriptions are omitted.

[0012] <Configuration of the safety support system> The overall configuration of the safety support system 100 according to this embodiment will be described below with reference to Figure 1. Figure 1 is a diagram showing the overall configuration of the safety support system 100 according to this embodiment. The safety support system 100 according to this embodiment is a system that analyzes and monitors the state of the detection area. Here, the safety support system 100 will be described as being used to issue an alarm when there is a possibility of contact between a vehicle (object) such as a forklift and a worker (person) in a manufacturing plant work area, and to verify the circumstances of the alarm event that triggered the alarm.

[0013] In the example shown in Figure 1, the safety support system 100 according to this embodiment includes a video source management unit 11, an alarm condition setting unit 12, an object detection unit 13, an alarm alarm unit 14, an alarm status monitoring unit 15, a detection information recording unit 16, a detection video extraction unit 17, and an extracted video viewing unit 18. These components are placed in the monitoring device 1 or the support computer group 6 by installing program 99a from the storage medium 160 into the monitoring device 1 and program 99b into the support computer group 6. The storage medium 160 stores program 161, which includes programs 99a and 99b. Programs 99a and 99b may also be installed in the monitoring device 1 or the support computer group 6 from a server located remotely via a network.

[0014] The video source management unit 11 is a component that registers and manages the captured video images (video source MI10) of each detection area 200 captured by multiple cameras 110 in the database 30. The alarm condition setting unit 12 is a component that registers detection conditions for detecting circumstances that require alarm activation.

[0015] The object detection unit 13 is a component that detects moving images in which objects that meet the detection conditions are captured from among the video sources MI10 of each detection area 200, which are captured by multiple cameras 110 (imaging devices) registered in the database 30. The alarm issuing unit 14 is a component that issues an alarm to the alarm device 3c, display device 3a, etc.

[0016] The alarm status monitoring unit 15 is a component that monitors the alarm status of each detection area 200. The detection information recording unit 16 is a component that registers detection information corresponding to the detection conditions in the database 30.

[0017] The detection video extraction unit 17 is a component that extracts and saves, as a detection video MI11, a video image showing an object that matches the detection conditions, from the video source MI10 of each detection area 200 detected according to the detection conditions. The extracted video viewing unit 18 is a component that displays multiple detected video images MI11 from the same detection area 200 on the display device 3a in a row, synchronized with the imaging time.

[0018] The safety support system 100 comprises a monitoring device 1, a group of support computers 6, a database 30, and multiple cameras 110. However, the safety support system 100 may also be configured to include multiple monitoring devices 1 and databases 30. Note that "database" may be written as "DB" in the drawings. Furthermore, the configuration shown in Figure 1 is merely an example, and the configuration of the safety support system 100 can be arbitrarily changed according to the operation.

[0019] Monitoring device 1 is a terminal device used by safety managers who manage safety at work sites. In this embodiment, monitoring device 1 is described as being composed of a personal computer. In this embodiment, it is described as being configured in monitoring device 1 as having a video source management unit 11, an alarm condition setting unit 12, an alarm alarm unit 14, an alarm status monitoring unit 15, and an extracted video viewing unit 18. In this embodiment, it is described as being connected to an input device 2 such as a keyboard or mouse, and an output device 3 such as a display device 3a, a printer 3b, or an alarm 3c.

[0020] The support computer group 6 is a computer system that assists in improving safety at the work site. In this embodiment, the support computer group 6 is described as having a configuration comprising multiple edge computers 7, a file server 8, and a workstation 9. However, the safety support system 100 may also have a configuration comprising multiple file servers 8 and workstations 9.

[0021] The edge computer 7 is a computer positioned around the file server 8 and / or workstation 9. In this embodiment, the edge computer 7 is described as having an object detection unit 13 that detects objects from the video source MI 10 captured by the camera 110. It is preferable that one edge computer 7 be provided for each camera 110 in order to improve the detection processing speed of objects (vehicles such as forklifts) and people (workers). However, it is also possible to provide one edge computer 7 for several cameras 110. In this case, it is preferable that the edge computer 7 has one object detection unit 13 (i.e., a dedicated object detection unit 13) for each camera 110. Furthermore, it is preferable that the object detection unit 13 includes an object detection engine 13a for detecting objects (e.g., vehicles such as forklifts) and a person detection engine 13b for detecting people (e.g., workers).

[0022] File server 8 is a server that records video source MI10, detection video MI11, digest video MI12, combined video MI13, system configuration file FI11, etc. Video source MI10 is the video footage of each detection area 200 captured by each camera 110. Detection video MI11 is the video footage from video source MI10 (captured video footage) that matches the detection conditions. Digest video MI12 is a video extracted from detection video MI11, containing only the parts that match the detection conditions. Combined video MI13 is a video created by combining detection video MI11 (or digest video MI12) captured by multiple cameras 110 capturing the same detection area 200. System configuration file FI11 contains various information related to the safety support system 100.

[0023] Workstation 9 is a computer that performs various processes related to safety at the work site. In this embodiment, workstation 9 has a detection information recording unit 16 and a detection video extraction unit 17, and will be described as performing the processing of recording detection information to the database 30 and the processing of extracting detection videos.

[0024] The database 30 is a recording means for recording video sources MI10 (captured video images) and various other information for each detection area 200. In this embodiment, the database 30 is described as a database server.

[0025] Camera 110 is an imaging device that is fixedly installed in each detection area 200 and captures moving images of each detection area 200. In this embodiment, camera 110 is configured as a network camera connected to the edge computer 7 via a network. In this embodiment, the manufacturing plant is provided with work sites for detection areas 200a, 200b, ..., 200n. Cameras 110a, 110b, ..., 110n are fixedly positioned in the work sites corresponding to each detection area 200a, 200b, ..., 200n.

[0026] The computers constituting the monitoring device 1 and support computer group 6 of the safety support system 100 are implemented by a computer 900 with a configuration such as that shown in Figure 2. Figure 2 is a hardware configuration diagram showing an example of a computer 900 that implements the safety support system 100 described above. The computer 900 includes a CPU 901, ROM 902, RAM 903, SSD 904, input / output interface 905 (labeled as input / output I / F (Interface) in Figure 2), communication interface 906 (labeled as communication I / F in Figure 2), and media interface 907 (labeled as media I / F in Figure 2). The computer 900 may be equipped with an HDD (Hard Disk Drive) instead of the SSD 904, or it may be equipped with an HDD in addition to the SSD 904.

[0027] The CPU 901 operates based on programs stored in the ROM 902 or SSD 904. The ROM 902 stores boot programs executed by the CPU 901 when the computer 900 starts up, as well as programs related to the computer 900's hardware. The recording medium 912 corresponds to the storage medium 160 in Figure 1. The input device 910 corresponds to input device 2 in Figure 1. The output device 911 corresponds to output device 3 in Figure 1.

[0028] Figure 3 is an explanatory diagram showing an example of an application program used in the safety support system 100. As shown in Figure 3, the following application program is used as an example in the safety support system 100. Note that "application program" may be written as "app" in the diagram. (1) Video source management application program AP11 (2) Alarm condition setting application program AP12 (3) Object detection application program AP13 (4) Alarm activation application program AP14 (5) Alarm status monitoring application program AP15 (6) Detection information recording application program AP16 (7) Detection and video extraction application program AP17 (8) Extracted video viewing application program AP18

[0029] The video source management application program AP11 is an application program that makes the computer function as a video source management unit 11 (Figure 1). The video source management application program AP11 (i.e., the video source management unit 11) records the following information, for example, in the database 30. Camera information • Object detection information / Object tracking information / Skeleton setting information Area Information ·Movement speed calculation setting information • Vector drawing settings information • Detection posture setting file

[0030] The alarm condition setting application program AP12 is an application program that causes the computer to function as the alarm condition setting unit 12 (Figure 1). The alarm condition setting application program AP12 (i.e., the alarm condition setting unit 12) records the following information, for example, in the database 30. • Alarm triggering conditions (judgment conditions) setting information, such as detection area, object type, and judgment formula. • Position information of each video source when creating a grid video. • Alarm system activation command information

[0031] The object detection application program AP13 is an application program that causes the computer to function as an object detection unit 13 (Figure 1). The object detection application program AP13 (i.e., the object detection unit 13) outputs object detection result information, including, for example, the following information, to the alarm notification application program AP14 and the detection information recording application program AP16. • Object detection coordinates • Types of objects detected • Number of objects detected • Object movement information • Speed ​​of movement of an object • Information on the area where the detected object entered. • ID information assigned to the object • Object trajectory information • Skeletal information if the object is a person

[0032] The alarm activation application program AP14 is an application program that causes the computer to function as an alarm activation unit 14 (Figure 1). The alarm activation application program AP14 (i.e., the alarm activation unit 14) outputs an alarm activation command to the alarm device 3c based on the object detection result information output from the object detection application program AP13, causing the alarm device 3c to activate an alarm. The alarm activation application program AP14 also outputs alarm status information to the alarm status monitoring application program AP15. Furthermore, the alarm activation application program AP14 outputs the following information to the detection information recording application program AP16. • Alarm information, such as which alarm conditions were met. • Start and end time information for issuing alerts

[0033] The alarm status monitoring application program AP15 is an application program that causes the computer to function as an alarm status monitoring unit 15 (Figure 1). The alarm status monitoring application program AP15 (i.e., the alarm status monitoring unit 15) monitors the alarm status of each alarm device 3c based on the alarm status information output from the alarm alarm application program AP14.

[0034] The detection information recording application program AP16 is an application program that causes the computer to function as a detection information recording unit 16 (Figure 1). The detection information recording application program AP16 (i.e., the detection information recording unit 16) records object detection result information output from the object detection application program AP13 and alarm information output from the alarm issuance application program AP14 in the database 30.

[0035] The detection video extraction application program AP17 is an application program that makes the computer function as a detection video extraction unit 17 (Figure 1). The detection video extraction application program AP17 (i.e., the detection video extraction unit 17) reads information such as the start-end time of alarm activation and the alarm activation condition setting information for creating grid video from the database 30. Here, "grid video" refers to video displayed in a video display area 60g that is divided into multiple grids, as shown in Figure 22, for example. The detection video extraction application program AP17 also records video information (such as storage location) for each extracted video source and extracted grid video information (such as storage location) in the database 30.

[0036] The extracted video viewing application program AP18 is an application program that makes the computer function as the extracted video viewing unit 18 (Figure 1). The extracted video viewing application program AP18 (i.e., the extracted video viewing unit 18) reads from the database 30 the number of alarms (daily, monthly, and yearly information) and video information of the moment of alarm. The extracted video viewing application program AP18 also displays information that matches the detection conditions, including the video source M10, on the display device 3a based on the information read from the database 30.

[0037] <Operation of the safety support system> First, the general operation of the safety support system 100 will be explained with reference to Figures 4 and 5. Figure 4 is a flowchart illustrating the general operation of the safety support system 100. Figure 5 is an operation diagram illustrating the general operation of the safety support system 100. Details of the operation of the safety support system 100 will be described later.

[0038] Here, we will explain assuming that the detection conditions (for example, alarm conditions for detecting events that require an alarm) are registered in the safety support system 100.

[0039] As shown in Figures 4 and 5, first, the safety support system 100 continuously records the on-site conditions (multiple video sources MI10) of each detection area 200 using multiple cameras 110 (step S10). At this time, the video source management unit 11 of the safety support system 100 records the continuously recorded on-site conditions (multiple video sources MI10) in the database 30 (step S10).

[0040] Next, the object detection unit 13 of the safety support system 100 performs real-time detection of objects (vehicles) and people (workers) based on the detection conditions (alarm conditions) (step S20). In step S20, object detection processing, object tracking processing, and skeleton detection processing are performed. Details of these processes will be described later.

[0041] Next, the detection information recording unit 16 of the safety support system 100 records the detection information detected in step S20 in the database 30 (step S30). Examples of the detection information recorded at this time include the following: • Detection area • Detection label • Detection coordinates (x, y) • Number of detections • Detection time

[0042] Next, the detection video extraction unit 17 of the safety support system 100 automatically extracts detection videos based on the detection information at any given time (step S40). At this time, the detection video extraction unit 17 extracts, based on the detection information, a video showing an object that matches the detection condition (alarm condition) from among multiple video sources MI10 (site conditions) recorded in the database 30, and uses it as a detection video MI11.

[0043] Next, in the safety support system 100, the operator of the monitoring device 1 (safety manager) operates the monitoring device 1 to check the detection status for the current day / previous day and earlier in a timely manner (step S50). At this time, the extracted video viewing unit 18 of the safety support system 100 displays the detection video for the date and time specified by the operator on the display device 3a.

[0044] Next, in the safety support system 100, the operator (safety manager) operates the monitoring device 1 to output a report containing the detection status and a digest video of the site for a specified period (step S60). At this time, the extracted video viewing unit 18 of the safety support system 100 displays, for example, the report, daily / monthly / yearly summaries, and digest video MI12 (Figure 1) on the display device 3a.

[0045] Next, the operation of the safety support system 100 will be explained with reference to Figures 6A to 6C. Figure 6A is a flowchart showing the object detection operation of the safety support system 100. Figure 6B is a flowchart showing the detection video extraction operation of the safety support system 100. Figure 6C is a flowchart showing the extracted video viewing operation of the safety support system 100.

[0046] In the safety support system 100, the object detection operation shown in Figure 6A is performed first. As shown in Figure 6A, during the object detection operation, the video source management unit 11 of the safety support system 100 first sets up the detection settings, receives video sources from each camera 110 and registers them in the database 30 (step S105).

[0047] Next, the alarm condition setting unit 12 of the safety support system 100 sets the alarm conditions (step S110).

[0048] Next, the object detection unit 13 of the safety support system 100 detects the trajectory and posture of an object if it is a person, based on the continuously recorded on-site conditions (multiple video sources MI10).

[0049] Next, the detection information recording unit 16 of the safety support system 100 records the detection information in the database 30 (step S120).

[0050] Next, the alarm activation unit 14 of the safety support system 100 determines whether or not the detected object or person meets the alarm activation conditions (step S125).

[0051] If the determination in step S125 determines that the detected object or person does not meet the alarm conditions ("NO"), the alarm unit 14 discards the detection information (step S130).

[0052] On the other hand, if the determination in step S125 determines that the detected object or person meets the alarm conditions ("YES"), the alarm unit 14 outputs an alarm command to the alarm unit 3c, causing the alarm unit 3c to issue an alarm (step S135).

[0053] After step S135, the alarm status monitoring unit 15 of the safety support system 100 reflects the alarm status in its own memory unit (step S140).

[0054] After step S140, the detection information recording unit 16 of the safety support system 100 records the detection information in the database 30 (step S145). The detection information recording unit 16 also outputs the detection information to the detection video extraction unit 17. Alternatively, the detection video extraction unit 17 queries the detection information recording unit 16 for detection information at any time.

[0055] Following the object detection operation shown in Figure 6A, the detection video extraction operation shown in Figure 6B is performed. As shown in Figure 6B, in the detection video extraction operation, first, the detection video extraction unit 17 of the safety support system 100 is activated by selecting the extraction period (step S205).

[0056] Next, the detection video extraction unit 17 determines whether or not detection information exists within the specified period (step S210).

[0057] If the determination in step S210 determines that no detection information exists within the specified period ("NO"), the detection video extraction unit 17 terminates the extraction process (step S215).

[0058] On the other hand, if the determination in step S210 determines that detection information exists within the specified period ("YES"), the detection video extraction unit 17 determines whether or not to grid-combine each video source (step S220). Here, "grid-combine" refers to the process of combining multiple video images in order to display multiple video images side by side in the video display area 60g divided into four sections, as shown in Figure 22. Hereafter, the video image created by combining multiple video images through grid-combine will be referred to as a "grid video image".

[0059] If the determination in step S220 is that each video source should not be grid-combined ("NO"), the detection video extraction unit 17 terminates the process without creating a grid video (step S225).

[0060] On the other hand, if the determination in step S220 is made to combine each video source into a grid ("YES"), the detection video extraction unit 17 extracts video images from each video source that show objects that match the detection conditions (alarm conditions) (step S230). As a result, the detection video extraction unit 17 creates a grid video image.

[0061] After step S220, the detection video extraction unit 17 records the extracted information in the database 30 (step S235). The detection video extraction unit 17 also outputs the detection information to the extracted video viewing unit 18. Alternatively, the extracted video viewing unit 18 queries the detection video extraction unit 17 for detection information at any time.

[0062] After the detection video extraction operation shown in Figure 6B, the extracted video viewing operation shown in Figure 6C is performed at any time. As shown in Figure 6C, in the extracted video viewing operation, first, the extracted video viewing unit 18 of the safety support system 100 is activated by specifying a viewing period (step S305). Next, the extracted video viewing unit 18 determines whether or not extracted videos exist within the specified period (step S310).

[0063] If the determination in step S310 determines that no extracted video images exist within the specified period ("NO"), the extracted video viewing unit 18 displays the number of detected items as 0 on the display device 3a (step S315). Next, the extracted video viewing unit 18 transitions the display screen of the display device 3a to the report output screen (step S320) and prompts the operator (safety manager) to specify the report output period (step S325). Next, the extracted video viewing unit 18 determines whether or not extracted video images exist within the specified period (step S330).

[0064] If the determination in step S330 is that no extracted video exists within the specified period ("NO"), the process returns to step S315. On the other hand, if the determination in step S330 is that extracted video exists within the specified period ("YES"), the extracted video viewing unit 18 prompts the operator (safety manager) to select the report to be output for each "year," "month," and "day" (step S335). After step S330, the extracted video viewing unit 18 outputs the report to the display device 3a (step S340).

[0065] Furthermore, if the determination in step S310 above determines that there are extracted videos within the specified period ("YES"), the extracted video viewing unit 18 displays the search results on the display device 3a (step S345).

[0066] Next, the extracted video viewing unit 18 transitions the display screen of the display device 3a to the digest video creation screen (step S350), allowing the operator (safety manager) to select the video to be used to create the digest video (step S355). Here, the "digest video" is explained as corresponding to the "grid video" described above.

[0067] Next, the extracted video viewing unit 18 prompts the operator (safety manager) to specify or select the length (time) of the video to be created (step S360).

[0068] Next, the extracted video viewing unit 18 prompts the operator (safety manager) to select the quality of the digest video (step S365).

[0069] Next, the extracted video viewing unit 18 specifies the save location for the output digest video (step S370), creates the digest video, and saves the digest video to the specified save location (step S375).

[0070] The extracted video viewing unit 18 may, depending on the operation, proceed from step S345 to step S320.

[0071] (Operation of the video source management unit) Next, we will provide a supplementary explanation of the operation of the video source management unit 11 with reference to Figures 7 and 8. Figures 7 and 8 are explanatory diagrams illustrating the general operation of the video source management unit 11, respectively.

[0072] As shown in Figure 7, the video source management unit 11 references the system configuration file FI11 for database connection information and obtains the database connection information from the system configuration file FI11. The system configuration file FI11 records information such as the following, for example. (1) Address information of the database storage location (2) Database name (3) Server connection method (4) User identifier / password (5) Network camera firmware access information

[0073] The video source management unit 11 records video source setting information such as the following: (1) Camera MAC / IP address / video source name (2) Camera resolution (3) Firmware access password (4) Number of video frames (fps) to be acquired every second (5) Object detection / tracking / skeleton configuration file path (6) Object tracking / skeleton detection weight file path (7) Object tracking algorithm file path (8) Path to the configuration file for detecting high-risk posture

[0074] The video source management unit 11 records video source setting information in the database 30. The video source management unit 11 also retrieves previously set video source setting information from the database 30.

[0075] As shown in Figure 8, the operator of monitoring device 1 (safety manager) performs the following actions by operating monitoring device 1. (1) Operate the monitoring device 1 to set the video source setting information on the video source setting screen 51. (2) Operate the monitoring device 1 to set the detection area, area name, detection threshold, etc. on the judgment condition setting screen 52. The video source management unit 11 then registers the configured information in the database 30 (reflecting it in the video source setting information registered in the database 30).

[0076] (Operation of the alarm condition setting unit) Next, we will provide a supplementary explanation of the operation of the alarm condition setting unit 12 with reference to Figures 9 and 10. Figures 9 and 10 are explanatory diagrams illustrating the general operation of the alarm condition setting unit 12, respectively.

[0077] As shown in Figure 9, the alarm condition setting unit 12 references the database connection information in the system configuration file FI11 and obtains the database connection information from the system configuration file FI11. The system configuration file FI11 records information such as the following, for example. (1) Address information of the database storage location (2) Database name (3) Server connection method (4) User identifier / password (5) Network camera firmware access information (6) Alarm condition setting file path (7) Default alarm system command information (8) Default network camera connection information

[0078] The alarm condition setting unit 12 records alarm condition setting information such as the following: (1) Detection area coordinate information (2) Detection area name information (3) Information on the type / number of objects to be detected in a specific area (4) Information on setting judgment conditions using logical formulas (5) Grid animation creation function ON / OFF information (6) Position information of each video source when creating a grid video (acquired position information) (7) Alarm activation command (http command) (8) Text information explaining the content of the alert

[0079] The alarm condition setting unit 12 records the alarm condition setting information in the database 30. The alarm condition setting unit 12 also retrieves the video source, detection area information, and previously set alarm condition setting information from the database 30.

[0080] As shown in Figure 10, the operator of monitoring device 1 (safety manager) performs the following actions by operating monitoring device 1. (1) Operate the monitoring device 1 and specify a task on the judgment task registration screen 41a. (2) Operate the monitoring device 1 to add an alarm condition on the processing method editing screen 41b. (3) Operate the monitoring device 1 to set the alarm conditions on the area label setting screen 41c. (4) Operate the monitoring device 1 to set the AND / OR judgment condition expression on the processing method editing screen 41b. (5) Operate the monitoring device 1 to specify the grid joining position on the area label screen 41d. The alarm condition setting unit 12 then registers the set information in the database 30 (reflecting it in the alarm condition setting information registered in the database 30).

[0081] (Operation of the object detection unit) Next, we will provide a supplementary explanation of the operation of the object detection unit 13 with reference to Figure 11. Figure 11 is an explanatory diagram showing the schematic operation of the object detection unit 13.

[0082] As shown in Figure 11, the object detection unit 13, upon startup, references the object detection setting information in the system configuration file FI11 and obtains the object detection setting information from the system configuration file FI11. The object detection unit 13 also obtains the on-site conditions (video source MI10) that are continuously recorded from the camera 110. Then, in step S115 of Figure 6A, the object detection unit 13 uses the continuously recorded on-site conditions (video source MI10) to detect the trajectory and posture of an object if it is a person. The object detection unit 13 then outputs the detection information to the alarm issuing unit 14.

[0083] Then, in step S125 of Figure 6A, the alarm activation unit 14 determines whether or not the detected object or person meets the alarm activation conditions.

[0084] If the determination in step S125 determines that the detected object or person does not meet the alarm conditions ("NO"), the alarm unit 14 discards the detection information (step S130).

[0085] On the other hand, if the determination in step S125 determines that the detected object or person meets the alarm conditions ("YES"), the alarm unit 14 outputs an alarm command to the alarm unit 3c, causing the alarm unit 3c to issue an alarm (step S135).

[0086] After step S135, the alarm issuing unit 14 determines whether or not it has been continuously issuing alarms at regular intervals (step S136). If the determination in step S136 is that it has been continuously issuing alarms at regular intervals ("YES"), the process returns to step S135. On the other hand, if the determination in step S136 is that it has not been continuously issuing alarms at regular intervals ("NO"), the alarm issuing unit 14 outputs an alarm stop command to the alarm device 3c, causing the alarm device 3c to stop issuing alarms (step S137).

[0087] Next, the detection information recording unit 16 of the safety support system 100 records the detection information in the database 30 (step S145).

[0088] (Operation of the alarm system) Next, we will provide a supplementary explanation of the operation of the alarm issuing unit 14 with reference to Figures 12 and 13. Figures 12 and 13 are explanatory diagrams showing the general operation of the alarm issuing unit 14, respectively.

[0089] As shown in Figure 12, the alarm activation unit 14, upon startup, references the system configuration file FI11 for alarm activation condition setting information and obtains the alarm activation condition setting information from the system configuration file FI11.

[0090] When the alarm unit 14 receives detection information from the object detection unit 13, it determines in step S125 whether the detected object or person meets the alarm conditions. If the determination in step S125 is "NO", the alarm unit 14 discards the detection information (step S130). On the other hand, if the determination in step S125 is "YES", the alarm unit 14 outputs an alarm command to the alarm unit 3c, causing the alarm unit 3c to issue an alarm (step S135). The alarm unit 14 also displays the alarm on the display device 3a. The alarm unit 14 also transmits the alarm content to the alarm status monitoring unit 15.

[0091] After step S135, the alarm issuing unit 14 repeatedly performs the determination process in step S136 (determining whether or not alarms are being continuously issued at regular intervals) until it determines that the result is "NO". If the determination in step S136 is "NO", the alarm issuing unit 14 outputs an alarm stop command to the alarm device 3c, causing the alarm device 3c to stop issuing alarms (step S137). The alarm issuing unit 14 also displays "Alarm Stopped" on the display device 3a. The alarm issuing unit 14 also transmits the alarm information to the detection information recording unit 16. The alarm information transmitted at this time includes, for example, information on which alarm conditions were met and alarm start-end time information.

[0092] The detection information recording unit 16, upon receiving alarm information from the alarm issuing unit 14, registers the alarm information in its own storage area, creates detection information from the alarm information, and records the detection information in the database 30 (step S145).

[0093] The alarm issuing unit 14 may also transmit the alarm information to the detection information recording unit 16 in a step prior to the processing in step S137. However, in this case, the alarm start-end time information of the alarm information does not include the alarm end time information. Therefore, when the alarm issuing unit 14 outputs an alarm stop command to the alarm device 3c in step S137, it is preferable that the alarm end time information is transmitted to the detection information recording unit 16, and the detection information recording unit 16, upon receiving the alarm end time information, updates the alarm start-end time information of the alarm information.

[0094] Figure 13 shows an example of the processing steps involved in the operation shown in Figure 12. In the process of step S125 shown in Figure 12, the alarm generation unit 14 detects the status of an object (vehicle) or person (worker) (for example, whether or not they are moving) from the screen of the video source 42a shown in diagram (1) of Figure 13. On the screen of the video source 42a shown in diagram (1) of Figure 13, the object (vehicle) or person (worker) detected by the object detection unit 13 is displayed surrounded by a rectangular frame.

[0095] Next, the alarm unit 14 determines the status of an object (vehicle) or person (worker) (for example, direction of movement, posture, etc.) from the status information 42b shown in diagram (2) of Figure 13.

[0096] Next, in step S135 shown in Figure 12, the alarm issuing unit 14 issues an alarm. At this time, for example, on the screen of the video source 42a shown in diagram (3) of Figure 13, the alarm device 3c lights up a red light and outputs an alarm sound.

[0097] Next, in the process of step S145 shown in Figure 12, for example, as shown in diagram (4) of Figure 13, the detection information recording unit 16 records the detection information in the database 30.

[0098] (Operation of the alarm status monitoring unit) Next, we will provide a supplementary explanation of the operation of the alarm status monitoring unit 15 with reference to Figures 14 and 15. Figures 14 and 15 are explanatory diagrams showing the general operation of the alarm status monitoring unit 15, respectively.

[0099] As shown in Figure 14, when the alarm status monitoring unit 15 starts up, it refers to the alarm condition setting information in the database 30 and obtains the alarm condition setting information from the database 30.

[0100] The alarm status monitoring unit 15 receives the alarm content from the alarm issuing unit 14 during the processing of step S135. Then, in the processing of step S140 shown in Figure 6A, the alarm status monitoring unit 15 executes the following processing.

[0101] First, the alarm status monitoring unit 15 updates the status of the detection task that is currently issuing an alarm to "ON" (step S141). Next, the alarm status monitoring unit 15 repeatedly checks whether the alarm is continuing to be issued until it is determined to be "NO" (step S142). Then, if it is determined to be "NO" in step S142, the alarm status monitoring unit 15 updates the status of the detection task that is currently issuing an alarm to "OFF" (step S143). At this time, as shown in Figure 15, the alarm status monitoring unit 15 updates the ON / OFF information of the alarm status information 42c.

[0102] (Operation of the detection information recording unit) Next, we will provide a supplementary explanation of the operation of the detection information recording unit 16 with reference to Figure 16. Figure 16 is an operation diagram illustrating the schematic operation of the detection information recording unit 16.

[0103] As shown in Figure 16, the detection information recording unit 16, upon startup, references the object detection setting information in the system setting file FI11 and obtains the object detection setting information from the system setting file FI11. In addition, the detection information recording unit 16 obtains alarm information from the alarm issuing unit 14 when the alarm issuing unit 14 stops (or issues) an alarm to the alarm device 3c.

[0104] The alarm information should ideally include detection information such as the following: • Object detection coordinate information on the detection screen • Information on the type of object detected • Count information of detected objects • Object movement information • Information on the movement speed of objects • Judgment task information that meets the triggering conditions • Confidence / reliability information during object detection • Detected skeletal coordinate information • Information on the area into which the detected object entered. • ID information assigned to the object • Object trajectory information • Skeletal information (if the object is a "person") • Name of the video source where detection occurred • Start and end time information for issuing alerts • Detected rectangular coordinates (bounding box) information • Posture information of the person who made the judgment

[0105] During the processing of step S145, the detection information recording unit 16 registers the alarm information in its own storage area, creates detection information from the alarm information, and records the detection information in the database 30.

[0106] (Operation of the detection and video extraction unit) Next, with reference to Figure 17, we will provide a supplementary explanation of the operation of the detection and video extraction unit 17. Figure 17 is an explanatory diagram showing the schematic operation of the detection and video extraction unit 17.

[0107] As shown in Figure 17, the detection video extraction unit 17, upon startup, references the alarm activation condition setting information in the system configuration file FI11 and obtains the alarm activation condition setting information from the system configuration file FI11.

[0108] The detection video extraction unit 17 searches the database 30 for the presence or absence of alarm events at any given time, and obtains alarm start-end time information and alarm condition setting information for creating grid video from the database 30.

[0109] Next, the detection video extraction unit 17 queries the database 30 (or file server 8) (or file server 8) to see if data corresponding to the alarm start-end time exists in the video storage folder FL11 located in the database 30 (or file server 8).

[0110] The database 30 (or file server 8) that receives the query determines whether or not continuous recording data exists for the time period to be used for extraction (step S405). If the determination in step S405 determines that continuous recording data does not exist for the time period to be used for extraction ("NO"), the database 30 (or file server 8) displays an error on the display device 3a via the monitoring device 1 (step S410). On the other hand, if the determination in step S405 determines that continuous recording data exists for the time period to be used for extraction ("YES"), the database 30 (or file server 8) sends the video data to be used for extraction to the detected video extraction unit 17. In response, the detected video extraction unit 17 executes the video extraction process. The following processes are executed in the video extraction process.

[0111] First, the detection video extraction unit 17 targets the total number of alarms issued within the specified period and extracts video footage based on the recorded alarm start-end time information (step S450). At this time, the detection video extraction unit 17 causes the database 30 to record the extracted information. The extracted information recorded at this time may include, for example, the following information. • File path of the extracted video • Thumbnails of extracted videos • Video start time ~ end time • Video frame rate (fps), etc.

[0112] After step S450, the detection video extraction unit 17 determines whether the grid video creation function is enabled in the alarm task to be extracted (step S455). If the determination in step S455 determines that the grid video creation function is enabled ("YES"), the detection video extraction unit 17 extracts video, that is, creates grid video (step S460). At this time, the detection video extraction unit 17 extracts video using the video source to be used for extraction set in the alarm condition setting unit 12 and the joining position of the grid video.

[0113] On the other hand, if the determination in step S460 determines that the grid video creation function is not enabled ("NO"), the detection video extraction unit 17 terminates processing without creating a grid video (step S470).

[0114] (Operation of the extracted video viewing unit) Next, we will provide a supplementary explanation of the operation of the extracted video viewing unit 18 with reference to Figures 18 to 20. Figures 18 to 20 are explanatory diagrams showing the general operation of the extracted video viewing unit 18, respectively.

[0115] As shown in Figure 18, the extracted video viewing unit 18 can be activated at any time (step S505). After this, the operator (safety manager) selects the alarm conditions to be viewed (step S510).

[0116] The extracted video viewing unit 18 then determines whether or not there is corresponding detection data in the database 30 (step S515). If the determination in step S515 determines that there is no corresponding detection data ("NO"), the extracted video viewing unit 18 displays "No matching data" on the display device 3a (step S520). On the other hand, if the determination in step S515 determines that there is corresponding detection data ("YES"), the extracted video viewing unit 18 executes video viewing processing. The following processes are executed in the video viewing processing.

[0117] First, a summary screen is displayed on the display device 3a to allow the operator (safety manager) to specify the video source to be viewed (step S525). Here, we will explain assuming that the summary screen has the same configuration as the viewing screen 60 shown in Figure 21. The operator (safety manager) specifies the video source to be viewed from the summary screen (step S530), and then selects the date and time to be viewed (step S535).

[0118] Then, the extracted video viewing unit 18 selects the video for the time period to be viewed from the timeline display area (timeline display area 60i in the example shown in Figure 21) of the summary screen (summary screen), and plays the selected video (step S540).

[0119] Furthermore, if the operator (safety manager) presses, for example, the digest video display instruction button 60e shown in Figure 21 during step S530 or step S535, the process proceeds to step S605 (start digest video creation) shown in Figure 19.

[0120] Furthermore, if the operator (safety manager) presses, for example, the report output instruction button 60f shown in Figure 21 during step S530 or step S535, the process proceeds to step S705 (start of report output) shown in Figure 20, which is shown in Figure 19.

[0121] Figure 19 shows an example of the process for creating a digest video. As shown in Figure 19, when the operator (safety manager) presses, for example, the digest video display instruction button 60e shown in Figure 21, the extracted video viewing unit 18 starts creating the digest video (step S605). At this time, the extracted video viewing unit 18 displays, for example, the viewing screen 60 shown in Figure 21 on the display device 3a. The viewing screen 60 will be described later.

[0122] The operator (safety manager) who views the viewing screen 60 specifies the period to be viewed (step S610). Then, the extracted video viewing unit 18 activates the digest video creation function (step S615).

[0123] The operator (safety manager) specifies the period for which the video source to be used for the digest video was acquired and performs a search (step S620). The extracted video viewing unit 18 then prioritizes and extracts detection videos with a high degree of relevance according to the degree of relevance (step S625). The degree of relevance is, for example, the abnormal speed of movement of an object (vehicle) or the length of the alarm duration. The extracted video viewing unit 18 filters all video sources by the degree of relevance to extract detection videos with a high degree of relevance. At this time, the extracted video viewing unit 18 can also extract a specific video from all detection videos, depending on the operation (step S630).

[0124] After step S625 (or step S630), the extracted video viewing unit 18 selects videos from the search results to be used for the digest video (step S635). Then, the extracted video viewing unit 18 calculates the video length of the combined video when all the selected videos are combined and displays it on the display device 3a (step S640).

[0125] The operator (safety manager), after viewing the calculated video duration, inputs the desired length (duration) of the combined video (step S645). The extracted video viewing unit 18 then calculates the speed-up ratio of the combined video that will result in a video duration closest to the specified length (duration) (step S650). The extracted video viewing unit 18 then calculates the estimated video duration of the combined video, which has changed due to the speed-up, and displays it on the display device 3a (step S655).

[0126] Next, the operator (safety manager) specifies the quality of the combined video to be output using a CFR value (step S665). The operator (safety manager) also specifies the destination folder path (step S670). Then, the extracted video viewing unit 18 outputs the digest video to the display device 3a (step S675).

[0127] Figure 20 shows an example of the process when outputting a report. As shown in Figure 20, when the operator (safety manager) presses, for example, the report output instruction button 60f shown in Figure 21, the extracted video viewing unit 18 starts outputting the report (step S705). At this time, the extracted video viewing unit 18 displays, for example, the viewing screen 60 shown in Figure 21 on the display device 3a.

[0128] The operator (safety manager) who views the viewing screen 60 specifies the period to be viewed (step S710). Then, the extracted video viewing unit 18 activates the report output function (step S715).

[0129] The operator (safety manager) specifies the period to be output as the report (step S720). The extracted video viewing unit 18 then extracts alarm events with a high degree of relevance according to their degree of relevance (step S725). The degree of relevance is, for example, an abnormal speed of movement of an object (vehicle) or the length of the alarm duration. The extracted video viewing unit 18 filters all video sources by the degree of relevance to extract alarm events with a high degree of relevance. At this time, the extracted video viewing unit 18 can also display the total number of alarms without filtering, depending on the operation (step S730).

[0130] After step S725 (or step S730), the extracted video viewing unit 18 determines whether the relevant detection data exists for one day or more (step S735). If the determination in step S735 determines that the relevant detection data does not exist for one day or more ("NO"), the extracted video viewing unit 18 displays "No relevant data" on the display device 3a (step S740). On the other hand, if the determination in step S735 determines that the relevant detection data exists for one day or more ("YES"), the extracted video viewing unit 18 activates the "daily" report output function (step S745).

[0131] After step S745, the extracted video viewing unit 18 determines whether or not there is data corresponding to more than one year (step S750). If the determination in step S750 determines that there is data corresponding to more than one year ("YES"), the extracted video viewing unit 18 enables the "annual" report output function (step S755). On the other hand, if the determination in step S750 determines that there is no data corresponding to more than one year ("NO"), the extracted video viewing unit 18 disables the "annual" report output function (step S760).

[0132] After step S760, the extracted video viewing unit 18 determines whether or not there is data corresponding to one month or more (step S765). If the determination in step S765 determines that there is data corresponding to one month or more ("YES"), the extracted video viewing unit 18 enables the "monthly" report output function (step S770). On the other hand, if the determination in step S765 determines that there is no data corresponding to one month or more ("NO"), the extracted video viewing unit 18 deactivates the "monthly" report output function (step S775).

[0133] After one of steps S755, S770, or S775, the operator (safety manager) selects one or all of the following: "annual," "monthly," or "daily" (step S780). The extracted video viewing unit 18 then outputs a report corresponding to the selected period to the display device 3a (step S785).

[0134] <Viewing screen configuration> Next, the configuration of the browsing screen 60 will be described with reference to Figures 21 and 22. Figures 21 and 22 are explanatory diagrams showing examples of the browsing screen 60, respectively.

[0135] In the examples shown in Figures 21 and 22, the viewing screen 60 includes a display video source selection field 60a, a calendar display field 60b, a date selection button 60c, and a detection count display field 60d. The viewing screen 60 also includes a digest video display instruction button 60e, a report output instruction button 60f, a video display field 60g, and a timeline display field 60i.

[0136] The display video source selection field 60a is for selecting the video source to display. The calendar display field 60b is a field that allows users to select and display alarm events by specifying either the year, month, or day. The date selection button 60c is used to specify the date on which the video to be displayed in the video display area 60g was taken. The detection count display field 60d is a field that displays the number of alarm events detected.

[0137] The digest video display instruction button 60e is a button that instructs the display of a digest video. The report output instruction button 60f is a button that instructs the system to output a report. The video display area 60g is a section that displays multiple digest videos, each containing only the portion of the detected video that matches the detection criteria. The timeline display area 60i is a field that allows users to select and display alarm events by specifying a time. By selecting a video within the time frame displayed on the timeline in the timeline display area 60i, the video within that time frame can be displayed on the display device 3a.

[0138] When the viewing screen 60 is first displayed, the report output instruction button 60f and the timeline display area 60i should be deactivated. The report output instruction button 60f and the timeline display area 60i are then activated, for example, when the operator (safety manager) selects (clicks) the date selection button 60c.

[0139] If the report output instruction button 60f is selected, the viewing screen 60 transitions to the report output screen (not shown). Also, if a day is selected using the day selection button 60c, the timeline is displayed in the timeline display field 60i. Note that the video display field 60g remains blank until a video is selected on the timeline.

[0140] Furthermore, in the example shown in Figure 22, the video display area 60g is divided into four grids, and a digest video 60h corresponding to each of the four grids is displayed.

[0141] Furthermore, in the example shown in Figure 22, the timeline display area 60i displays three selectable detection time zones 60j. Each detection time zone 60j represents the time period during which the video source related to the alarm event was recorded. By selecting one of the detection time zones 60j, the operator (safety manager) displays multiple digest videos 60h corresponding to the selected time zone in the video display area 60g. For example, when the operator (safety manager) hovers their mouse over a detection time zone 60j displayed on the timeline in the timeline display area 60i, a thumbnail is displayed.

[0142] When the operator (safety manager) double-clicks in an empty area of ​​the thumbnail display area (including the timeline), they can widen the time interval displayed on the timeline (zoom down).

[0143] Additionally, when the operator (safety manager) double-clicks the thumbnail, the video will be displayed in the video display area 60g.

[0144] Furthermore, when the operator (safety manager) single-clicks on the detection time period 60j displayed on the timeline in the timeline display area 60i, a digest video 60h is displayed in the video display area 60g. Each digest video 60h is a video based on a detection video MI11 extracted from video sources MI10 captured in the same detection area 200 in synchronization with the imaging time.

[0145] <Judgment details> Next, with reference to Figure 23, the determination process of the object detection unit 13 will be explained. Figure 23 is an explanatory diagram showing an example of the determination process. In the example shown in Figure 23, the object detection unit 13 performs the following five-level determination from Lv.1 to Lv.5. (Level 1: Object Detection) (Level 2: Multiple Conditional Judgment (Logical Expression, Quantity)) (Level 3: Object Tracking) (Level 4: Skeleton detection (posture estimation)) (Level 5: Prediction based on direction of movement)

[0146] (Level 1: Object Detection) Figure 24 is an explanatory diagram showing an example of object detection. As shown in Figure 24, in Level 1 "object detection," the object detection unit 13 detects the intrusion of an object into a specific area from among multiple video sources 42a.

[0147] (Level 2: Multiple Conditional Judgment (Logical Expression, Quantity)) Figure 25 is an explanatory diagram showing an example of object detection using multiple condition judgment. The operator (safety manager) can freely define the judgment conditions using logical expressions. As shown in Figure 25, in Level 2's "multiple condition judgment (logical expression, quantity)," the object detection unit 13 detects one or more objects (vehicles) or people (workers) from multiple video sources 42a that have captured the same detection area 200, based on the defined judgment conditions. In Level 1, an alarm was issued for all detections that met simple conditions, but in Level 2, it becomes possible to set complex judgment conditions by introducing a proprietary algorithm. In Level 2, it becomes possible to specify the number of individuals for detection labels, and to perform AND and OR judgments combining multiple labels. It also becomes possible to detect alarm events that cannot be detected with only one angle. However, in Level 2, only alarm events in specified (assumed) scenes are detected, making it difficult to detect unknown alarm events.

[0148] (Level 3: Object Tracking) Figure 26 is an explanatory diagram illustrating an example of object tracking. The object detection unit 13 tracks an object and detects whether or not an alarm event will occur, depending on the likelihood of an alarm event occurring (how likely it is to occur). The likelihood of an alarm event occurring is determined by, for example, the example shown in Figure 26. In the example on the left of Figure 26, only a person 400, who is a worker, is visible in the video source 42a. The person 400 moves only a small amount. Therefore, the example on the left of Figure 26 has a low probability of an alarm event occurring (i.e., the likelihood of an alarm event occurring is low). On the other hand, in the example on the right of Figure 26, not only the person 400 but also a vehicle 300, such as a forklift, is visible in the video source 42a. The vehicle 300 moves only a large amount. Therefore, the example on the left of Figure 26 has a high probability of an alarm event occurring (i.e., the likelihood of an alarm event occurring is high). As shown in Figure 26, in Level 3 "object tracking," the object detection unit 13 detects a large moving object (vehicle 300) as a target for alarm activation. The object detection unit 13 then tracks the object to determine whether or not an alarm event will occur. The alarm activation unit 14 of the safety support system 100 activates an alarm when there is a high probability that an alarm event will occur. However, the alarm activation unit 14 of the safety support system 100 may also activate an alarm when the object detection unit 13 detects a large moving object. In Level 2, it is difficult to detect unknown alarm events, but in Level 3, alarm events other than those specified (assumed) can also be detected, making it possible to detect alarm events other than unknown ones.

[0149] (Level 4: Skeleton detection (posture estimation)) Figure 27 is an explanatory diagram showing an example of skeletal detection (posture estimation). Depending on their posture, person 400 may have difficulty noticing vehicle 401. Figure 27 shows the normal posture, downward-looking posture, and forward-leaning posture of person 400 as detected by the object detection unit 13 from skeletal detection images 43a, 43b, and 43c. "Downward-looking posture" is a posture in which person 400 (worker) is likely to be looking away. "Forward-leaning posture" is a posture in which person 400 (worker) is likely to be working in a posture that makes it difficult to see ahead. From the normal posture to the downward-looking posture and then the forward-leaning posture, it becomes more difficult for person 400 (worker) to notice vehicle 401. In other words, the more the posture changes from a normal posture to a downward-looking posture or a forward-leaning posture, the higher the probability of an alarm event occurring (i.e., the likelihood of an alarm event occurring). In Level 4's "skeleton detection (posture estimation)," the object detection unit 13 detects whether or not an alarm event will occur by performing this skeletal detection (posture estimation). The alarm issuing unit 14 of the safety support system 100 issues an alarm when there is a high probability that an alarm event will occur. The safety support system 100 can reduce the occurrence of contact or collision between people and vehicles by detecting and issuing an alarm when a person 400 (worker) is in a posture that makes it difficult to notice the vehicle 401, such as a downward-looking posture or a forward-leaning posture.

[0150] Figure 28 is a flowchart illustrating the general operation of posture determination. Here, we will explain the definition of posture types based on the posture determination results as follows. (1) If the pose is outside the definition despite the absence of feature points, the type of pose shall be designated as "out-of-definition pose". (2) If a person is detected but all feature points are missing, the posture type will be set to "total loss". (3) If some of the posture determination conditions cannot be determined due to the absence of some feature points, the type of posture will be set to "missing". (4) If a feature point is missing but the defined condition is met, the operator (safety manager) shall be instructed to input the "posture name" as the type of posture. (5) If all characteristic points are present and the defined condition is met, the operator (safety manager) shall be instructed to input the "posture name" as the type of posture.

[0151] As shown in Figure 28, when the object detection unit 13 detects a person (step S805), it determines whether or not a feature point has been detected (step S810).

[0152] If the determination in step S810 is that no feature points have been detected ("NO"), the object detection unit 13 sets the posture type to "total loss" (step S815). On the other hand, if the determination in step S810 is that feature points have been detected ("YES"), the object detection unit 13 determines whether or not the feature points used for posture determination are missing (step S820).

[0153] If the determination in step S820 determines that the feature points used for posture determination are missing ("NO"), the object detection unit 13 sets the posture type to "missing" (step S825). On the other hand, if the determination in step S820 determines that the feature points used for posture determination are not missing ("YES"), the object detection unit 13 determines whether or not the posture determination conditions are met (step S830).

[0154] If the determination in step S830 indicates that the posture determination conditions are not met ("NO"), the object detection unit 13 sets the posture type to "undefined posture" (step S835). On the other hand, if the determination in step S830 indicates that the posture determination conditions are met ("YES"), the object detection unit 13 (or alarm condition setting unit 12) prompts the operator (safety manager) to input a new "posture name" as the posture type (step S840).

[0155] (Level 5: Prediction based on direction of movement) Figure 29 is an explanatory diagram showing an example of prediction based on direction of movement. In Level 5, "Prediction based on direction of movement," the object detection unit 13 can predict the likelihood of an alarm event occurring by predicting the direction of movement of the vehicle. For example, in the upper example of Figure 29, the object detection unit 13 detects that the vehicle is moving in reverse. Also, in the lower example of Figure 29, the object detection unit 13 detects that the vehicle and a person are moving towards the intersection of the passageway. By making such predictions based on direction of movement, the object detection unit 13 detects whether or not an alarm event will occur. The alarm issuing unit 14 of the safety support system 100 issues an alarm when there is a high probability that an alarm event will occur.

[0156] Figures 30A to 30D are explanatory diagrams illustrating examples of situations that are highly likely to occur (situations in which an alarm event is likely to occur). Figure 30A shows a situation where a worker is walking near a vehicle traffic area with both hands full of luggage, making it impossible to check their surroundings. Figure 30B shows a situation where a forklift (vehicle) is entering the factory premises, and a worker has their back to the forklift near the vehicle traffic area. Figure 30C shows a situation where a worker is walking near a forklift (vehicle) while it is moving within the vehicle traffic area of ​​the factory premises, and the worker is distracted by looking at their smartphone or work instructions. Figure 30D shows a situation where two workers, while talking, are walking and looking away, and are unaware of the approaching forklift (vehicle). The safety support system 100 should be configured so that the occurrence of these situations can be detected by the object detection unit 13.

[0157] <Main features of the safety support system, safety support methods, and program> (1) As shown in Figure 1, the safety support system 100 according to this embodiment includes an alarm condition setting unit 12, an object detection unit 13, a detected video image extraction unit 17, and an extracted video image viewing unit 18. The alarm condition setting unit 12 is a component for registering detection conditions. The object detection unit 13 is a component for detecting video images in which objects corresponding to the detection conditions are shown from among multiple video sources MI10 of each detection area 200 captured by multiple cameras 110 (imaging devices) registered in the database 30. The detected video image extraction unit 17 is a component for extracting and saving video images in which objects corresponding to the detection conditions are shown from among the video sources MI10 detected by the detection conditions as detected video images MI11. The extracted video image viewing unit 18 is a component for displaying multiple detected video images MI11 of the same detection area 200 side by side on the display device 3a in synchronization with the time of imaging.

[0158] The safety support system 100 according to this embodiment displays multiple detection video MI11 of the same detection area 200 on the display device 3a in a manner synchronized with the time of imaging. Such a safety support system 100 allows safety managers to view video images of the same detection area 200 taken from multiple directions (i.e., multiple video images of the site situation). Therefore, the safety support system 100 makes it easier for safety managers to check the site situation.

[0159] (2) As shown in step S640 of Figure 19, in the safety support system 100 described in item (1) above, the detection video extraction unit 17 may create a combined video MI13. The combined video MI13 is created by combining multiple detection video MI11 in synchronization with the imaging time, or by combining multiple digest video MI12, which are obtained by extracting only the parts corresponding to the detection conditions from multiple detection video MI11. The extracted video viewing unit 18 displays the combined video MI13 on the display device 3a.

[0160] In this embodiment, the safety support system 100 displays the combined video MI13 created by the detection video extraction unit 17 on the display device 3a using the extracted video viewing unit 18. This allows the safety support system 100 to display multiple detection video MI11 of the same detection area 200 side by side on the display device 3a in synchronization with the imaging time. Therefore, the safety support system 100 makes it easier for safety managers to check the situation on site.

[0161] (3) As shown in step S20 of Figure 4, in the safety support system 100 described in item (1) above, the detection conditions may include object detection conditions such as the type and number of objects, object tracking conditions such as the speed and direction of movement of objects, and skeletal detection conditions such as the posture of a person.

[0162] The safety support system 100 according to this embodiment can distinguish and detect people and vehicles. Furthermore, the safety support system 100 can track objects by detecting the speed and direction of movement of people and vehicles. It can also detect postures that are likely to indicate a person is distracted or working in a position that makes it difficult to see ahead. Therefore, the safety support system 100 can efficiently issue alarms when there is a possibility of an alarm event occurring.

[0163] (4) As shown in Figure 1, the safety support system 100 described in item (1) above may further include an alarm issuing unit 14 that issues an alarm. As shown in Figure 29, the object detection unit 13 estimates the object movement trajectory, which represents the direction and speed of movement of the object, and the person movement trajectory, which represents the direction and speed of movement of the person, based on the detection conditions, and determines whether or not there is a possibility of contact between the object and the person. The alarm issuing unit 14 may issue an alarm if the object detection unit 13 determines that there is a possibility of contact between the object and the person.

[0164] The safety support system 100 according to this embodiment can estimate the object's movement trajectory, which represents the direction and speed of movement of an object, and the person's movement trajectory, which represents the direction and speed of movement of a person, and can determine whether or not there is a possibility of contact between the object and the person. Therefore, the safety support system 100 can efficiently issue an alarm when there is a possibility of an alarm event occurring.

[0165] (5) As shown in Figure 1, the safety support system 100 described in item (1) above may further include an alarm issuing unit 14 that issues an alarm. As shown in Figure 27, the object detection unit 13 detects a person and an object moving in close proximity to the person based on the detection conditions, detects the person's posture, and determines whether or not the person may not notice the object. The alarm issuing unit 14 may issue an alarm when the object detection unit 13 determines that the person may not notice the object.

[0166] The safety support system 100 according to this embodiment can detect the posture of a person when it detects an object moving in close proximity to the person, and can determine whether or not the person may not notice the object. Therefore, the safety support system 100 can issue an alarm more efficiently when there is a possibility of an alarm event occurring.

[0167] (6) As shown in Figures 6A and 6B, in the safety support system 100 described in item (1) above, suppose the object detection unit 13 detects the occurrence of an alarm event that requires the issuance of an alarm (step S115 in Figure 6A). In this case, it is preferable for the detection video extraction unit 17 to extract and save the detection video MI11 related to the alarm event from among the multiple video sources MI10 (step S235 in Figure 6B).

[0168] In this embodiment, the safety support system 100 pre-extracts and saves the detection video MI11 related to the alarm event when the alarm event occurs. Therefore, the safety support system 100 can efficiently play back the detection video MI11 related to the alarm event in a short amount of time.

[0169] (7) As shown in the viewing screen 60 in Figure 21, in the safety support system 100 described in item (6) above, the detection video extraction unit 17 should extract and save the detection video MI11 related to the alarm event in a state that allows it to be displayed at any time interval.

[0170] The safety support system 100 according to this embodiment stores the detected video MI11 related to alarm events in a state that allows it to be displayed at arbitrary intervals. Therefore, the safety support system 100 can specify an interval and display the detected video MI11 related to alarm events on the display device 3a.

[0171] (8) As shown in Figure 6C, in the safety support system 100 described in item (6) above, the extracted video viewing unit 18 may display data related to alarm events as a report on the display device 3a (step S340).

[0172] The safety support system 100 according to this embodiment can display data related to alarm events as a report on the display device 3a. Therefore, the safety support system 100 can suitably present data related to alarm events to the operator (safety manager).

[0173] (9) As shown in Figure 21, in the safety support system 100 described in item (1) above, the extracted video viewing unit 18 creates a viewing screen 60 that displays alarm events that match the detection conditions. The viewing screen 60 may include a calendar display field 60b, a timeline display field 60i, and a video display field 60g. The calendar display field 60b is a display field for selecting alarm events by specifying either the year, month, or day. The timeline display field 60i is a display field for selecting alarm events by specifying the time. The video display field 60g is a display field that displays multiple digest video images MI12 side by side, extracted from multiple detected video images MI11, showing only the parts that match the detection conditions.

[0174] The safety support system 100 according to this embodiment can display a user-friendly viewing screen 60 for safety managers, thereby improving convenience.

[0175] (10) As shown in Figures 6A to 6C, the safety support method according to this embodiment includes a video source management step (step S105), an alarm condition setting step (step S110), and an object detection step (step S115). The safety support method according to this embodiment also includes a detected video image extraction step (steps S230, S235) and an extracted video image viewing step (step S375). The video source management step (step S105) is a step of registering and managing multiple video sources MI10 of each detection area 200 captured by multiple cameras 110 (imaging devices) in a database 30. The alarm condition setting step (step S110) is a step of registering detection conditions. The object detection step (step S115) is a step of detecting video images in which an object corresponding to the detection conditions is shown from among the multiple video sources MI10 registered in the database 30. The detection video extraction process (steps S230, S235) is a process of extracting and saving, as detection video MI11, a video image containing an object that matches the detection conditions, from the video source MI10 detected according to the detection conditions. The extracted video viewing process (step S375) is a process of displaying multiple detection video MI11 from the same detection area 200 on the display device 3a in a sequence, synchronized with the time of imaging.

[0176] The safety support method according to this embodiment makes it easier for safety managers to check the on-site situation.

[0177] (11) The program according to this embodiment is a program that causes a computer to execute the video source management step, the alarm condition setting step, the object detection step, the detected video image extraction step, and the extracted video image viewing step described in item (10) above.

[0178] The program according to this embodiment can implement the safety support system 100 configured in item (1) above on a computer.

[0179] As described above, the safety support system 100 according to this embodiment makes it easier for safety managers to check the on-site situation.

[0180] The present invention is not limited to the embodiments described above, and includes various modifications. For example, the embodiments described above are described in detail for the purpose of clearly illustrating the present invention, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace some of the configurations of the embodiments with other configurations, and it is also possible to add other configurations to the configurations of the embodiments. In addition, it is possible to add, delete, or replace some of the configurations of each configuration with other configurations.

[0181] Furthermore, each of the above-mentioned configurations, functions, processing units, processing means, etc., may be implemented in hardware, in whole or in part, for example, by designing them as integrated circuits. Alternatively, each of the above-mentioned configurations, functions, etc., may be implemented in software by having the processor interpret and execute programs that implement each function. Information such as programs, tables, files, etc., that implement each function can be stored in memory, a storage device such as a hard disk or SSD (Solid State Drive), or a storage medium such as an IC card, SD card, or DVD.

[0182] Furthermore, the safety support system 100 may be configured to include an AI learning unit (not shown) that automatically learns detection conditions based on the video source MI 10 registered in the database 30. The AI ​​learning unit (not shown) is provided inside or outside the monitoring device 1. The AI ​​learning unit (not shown) automatically learns detection conditions based on the video source MI 10 registered in the database 30 and provides the learned detection conditions to the alarm condition setting unit 12 of the monitoring device 1. The alarm condition setting unit 12 registers the detection conditions learned by the AI ​​learning unit (not shown) to the edge computer 7, file server 8, workstation 9, database 30, etc. As a result, the object detection unit 13 detects objects (vehicles) or people (workers) based on the learned detection conditions. In this configuration, the safety support system 100 can automatically learn detection conditions and use the learned detection conditions to detect objects (vehicles) or people (workers), thereby automatically improving detection accuracy.

[0183] Furthermore, the safety support system 100 may also enlarge and display the specified object (vehicle) or person (worker) displayed on the viewing screen 60 by specifying the object (vehicle) or person (worker). This makes it even easier for safety managers to check the site conditions. [Explanation of Symbols]

[0184] 1 Monitoring device 2,910 Input devices 3,911 Output device 3a Display device 3b Printing device 3c alarm 6. Support Computer Group 7 Edge Computers 8 File Server 9 Workstations 11. Video Source Management Department 12 Alarm Condition Setting Unit 13 Object detection unit 13a Engine for object detection 13b Human detection engine 14 Alarm dispatcher 15. Alarm Status Monitoring Department 16 Detection Information Recording Unit 17 Detection and video extraction unit 18 Extracted video viewing section 30 databases 60 Browsing screen 60a Display video source selection field 60b Calendar display area 60°C day selection button 60d Detected items display field 60e Digest video display instruction button (corresponding video display instruction button) 60f Report Output Command Button 60g Video display area (Relevant video display area) 60h digest video (relevant video) 60i Timeline Display Area 60j detection time period 99a,99b,161 Programs 100 Safety Support Systems 110, 110a, 110b, ... Camera (imaging device) 160 Storage medium Detection area: 200, 200a, 200b, ... AP11 Video Source Management Application Program AP12 Alarm Activation Condition Setting Application Program AP13 Object Detection Application Program AP14 Alarm Activation Application Program AP15 Alarm Status Monitoring Application Program AP16 Detection Information Recording Application Program AP17 Detective Video Extraction Application Program AP18 Extracted Video Viewing Application Program FI11 System Configuration File FL11 Video storage folder FI11 System Configuration File MI10 video source (recorded video footage) MI11 detection video (extracted video) MI12 Digest Video MI13 Combined Video

Claims

1. An object detection unit detects moving images containing objects that meet the detection criteria from among multiple video sources of each detection area captured by multiple imaging devices registered in a database, A notification condition setting unit for registering the aforementioned detection conditions, A detection video extraction unit extracts and saves, as a detection video, a video source detected by the aforementioned detection conditions, a video in which an object corresponding to the aforementioned detection conditions is shown, and saves it. It includes an extracted video viewing unit that displays multiple detected video images of the same detection area side by side on a display device in synchronization with the time of imaging, A safety support system characterized by the following features.

2. In the safety support system according to claim 1, The detection video extraction unit creates a combined video by combining multiple detection videos in synchronization with the capture time, or by combining multiple digest videos in which only the portion corresponding to the detection condition is extracted from multiple detection videos. The extracted video viewing unit causes the combined video to be displayed on the display device. A safety support system characterized by the following features.

3. In the safety support system according to claim 1, The aforementioned detection conditions include object detection conditions, object tracking conditions, and skeleton detection conditions. A safety support system characterized by the following features.

4. In the safety support system according to claim 1, It is further equipped with an alarm issuing unit that issues alarms, The object detection unit estimates the object's movement trajectory, which represents the direction and speed of the object's movement, and the person's movement trajectory, which represents the direction and speed of the person's movement, based on the detection conditions, and determines whether or not there is a possibility of contact between the object and the person. The alarm issuing unit issues an alarm when the object detection unit determines that there is a possibility of contact between the object and the person. A safety support system characterized by the following features.

5. In the safety support system according to claim 1, It is further equipped with an alarm issuing unit that issues alarms, The object detection unit, based on the detection conditions, detects a person and an object moving in close proximity to the person, detects the person's posture, and determines whether or not the person may not notice the object. The alarm issuing unit issues an alarm when the object detection unit determines that the person may not notice the object. A safety support system characterized by the following features.

6. In the safety support system according to claim 1, When the object detection unit detects the occurrence of an alarm event requiring the issuance of an alarm, the detection video extraction unit extracts and saves a detection video related to the alarm event from among the multiple video sources. A safety support system characterized by the following features.

7. In the safety support system described in claim 6, The detection video extraction unit extracts and stores the detection video related to the alarm event in a state that allows it to be displayed at arbitrary intervals. A safety support system characterized by the following features.

8. In the safety support system described in claim 6, The extracted video viewing unit displays the data related to the alarm event as a report on the display device. A safety support system characterized by the following features.

9. In the safety support system according to claim 1, The extracted video viewing unit creates a viewing screen that displays alarm events that match the detection conditions. The aforementioned viewing screen is A calendar display field where you can select an alarm event by specifying either the year, month, or day, A timeline display area where you can select an alarm event by specifying the time, Includes a video display area that simultaneously displays multiple digest videos, each containing only the portion of the detected video that matches the detection criteria, from among multiple detected videos. A safety support system characterized by the following features.

10. A video source management process involves registering and managing multiple video sources from each detection area, captured by multiple imaging devices, in a database. The process involves setting alarm conditions to register detection conditions, An object detection step of detecting a video image containing an object that meets the detection conditions from among multiple video sources registered in the database, A detection video extraction step involves extracting and saving, as a detection video, a video source detected by the aforementioned detection conditions, a video in which an object corresponding to the aforementioned detection conditions is shown, and saving it. Includes an extracted video viewing step which involves displaying multiple detected video images of the same detection area on a display device in a sequence, synchronized with the time of acquisition. A safety support method characterized by the following features.

11. On the computer, A video source management process involves registering and managing multiple video sources from each detection area, captured by multiple imaging devices, in a database. The process involves setting alarm conditions to register detection conditions, An object detection step of detecting a video image containing an object that meets the detection conditions from among multiple video sources registered in the database, A detection video extraction step involves extracting and saving, as a detection video, a video source detected by the aforementioned detection conditions, a video in which an object corresponding to the aforementioned detection conditions is shown, and saving it. A program for performing an extracted video viewing step, which involves displaying multiple detected video images of the same detection area on a display device in a sequence, synchronized with the time of acquisition.

Images