Work detection system, information processing device, program for information processing device, terminal device, program for terminal device, and work detection method.

Abstract

To provide a work detection system capable of improving work safety without being constrained by photographing conditions of a work area and without needing additional devices or worker wearing equipment, an information processing device, a program for the information processing device, a terminal device, a program for the terminal device, and a work detection method.SOLUTION: This work detection system comprises: a photographing device that captures an image of a work area; and an information processing device that inputs the image captured by the photographing device to a machine learning model in which the image of the work area is input data and image data of objects to be monitored included in the image is output data so as to extract an object to be monitored from the image captured by the photographing device, determines an object to be warned about from among the objects to be monitored on the basis of the position and the size of the extracted object to be monitored, and determines the danger of the object to be monitored with respect to the object to be warned about according to the position of the object to be warned about.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a work detection system, an information processing device, a program for the information processing device, a terminal device, a program for the terminal device, and a work detection method.

Background Art

[0002] As technologies for improving the safety of crane work, the technologies described in Patent Documents 1 and 2 are known. Specifically, Patent Document 1 describes an apparatus including a camera installed on the main body of a crane for imaging the crane work area from directly above, and an infrared projector provided near the camera. The apparatus described in Patent Document 1 detects the reflected light of infrared rays from a mark made of a retroreflective material attached to the helmet of an operator with the camera, and confirms the safety of the operator by specifying the position of the operator in the crane work area. On the other hand, Patent Document 2 describes a system including a camera that photographs the lower part of a suspended load attached to a crane, and a distance meter attached near the camera for measuring the distance to the work surface. The system described in Patent Document 2 adjusts the display size of a danger area on a monitor according to the distance between the camera, whose height changes, and the work surface, and determines whether to issue an alarm signal by discriminating an operator based on the color of the helmet.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] The technologies described in Patent Documents 1 and 2 have limitations regarding the conditions for photographing the crane work area, specifically that the crane work area must be photographed from directly above. Furthermore, additional measuring equipment is required to determine the worker's position, incurring significant costs. Moreover, since worker position determination is performed using equipment worn by the worker, such as markers or helmet colors, additional work such as pre-work inspections of the equipment is necessary to eliminate the effects of forgotten, missing, or soiled equipment.

[0005] The present invention was made to solve the above problems, and its objective is to provide a work detection system, an information processing device, a program for the information processing device, a terminal device, a program for the terminal device, and a work detection method that can improve work safety without imposing restrictions on the shooting conditions of the work area and without requiring the installation of additional equipment or equipment worn by workers. [Means for solving the problem]

[0006] The work detection system according to the present invention comprises: an imaging device that captures images of a work area; and an information processing device that, by inputting the images captured by the imaging device to a machine learning model that takes the images of the work area as input data and image data of objects to be monitored contained in the images as output data, extracts objects to be monitored from the images captured by the imaging device, determines objects to be alerted from among the objects to be monitored based on the position and size of the extracted objects to be monitored, and determines the danger posed by the objects to be alerted to the objects to be alerted according to the position of the objects to be alerted.

[0007] The information processing device may determine the object to be alerted based on the relative positional relationship of the monitored objects.

[0008] The information processing device may determine the object to be alerted based on the posture, possessions, and accessories of the monitored object.

[0009] The aforementioned work area is preferably a crane work area where loads are transported by crane hooks.

[0010] The information processing device according to the present invention takes an image of a work area as input data and image data of objects to be monitored contained in the image as output data to a machine learning model, inputs an image of a work area, extracts objects to be monitored from the image of the work area, determines which objects to be alerted from among the objects to be monitored based on the position and size of the extracted objects to be monitored, and determines the risk posed by the objects to be alerted to the objects to be alerted according to the position of the objects to be alerted.

[0011] The program for the information processing device according to the present invention takes an image of a work area as input data and image data of objects to be monitored contained in the image as output data, and when the image of the work area is input to a machine learning model, it extracts objects to be monitored from the image of the work area, determines which objects to be alerted from among the objects to be monitored based on the position and size of the extracted objects to be monitored, and causes the computer to execute a process to determine the danger posed by the objects to be alerted to the objects to be alerted to, according to the position of the objects to be alerted to.

[0012] The terminal device according to the present invention takes an image of the work area as input data and image data of objects to be monitored contained in the image as output data to a machine learning model, inputs an image of the work area, extracts objects to be monitored from the image of the work area, determines which objects to be alerted from among the objects to be alerted based on the position and size of the extracted objects to be alerted, and executes a warning for the objects to be alerted according to the determination result of an information processing device that determines the danger posed by the objects to be alerted to the objects to be alerted based on the location of the objects to be alerted.

[0013] The program for the terminal device according to the present invention takes an image of the work area as input data and image data of objects to be monitored contained in the image as output data as input data to a machine learning model. By inputting an image of the work area, the program extracts objects to be monitored from the image of the work area, determines which objects to be alerted from among the objects to be monitored based on the position and size of the extracted objects to be monitored, and causes a computer to execute a process to issue a warning to the objects to be alerted according to the determination result of an information processing device that determines the danger posed by the objects to be alerted to based on the position of the objects to be alerted to.

[0014] The terminal device according to the present invention performs a warning on the object to be warned in response to an instruction from the information processing device according to the present invention.

[0015] The work detection method according to the present invention includes a shooting step of taking an image of a work area, and a step of inputting the image taken in the shooting step into a machine learning model that takes the image of the work area as input data and image data of objects to be monitored included in the image as output data, thereby extracting objects to be monitored from the image taken in the shooting step, determining an object to be alerted from among the objects to be monitored based on the position and size of the extracted objects to be monitored, and determining the risk of the objects to be alerted to the objects to be alerted according to the position of the objects to be alerted. [Effects of the Invention]

[0016] According to the work detection system, information processing device, program for the information processing device, terminal device, program for the terminal device, and work detection method of the present invention, it is possible to alert workers solely by processing information using image data of the work area. Therefore, it is possible to improve work safety without imposing restrictions on the shooting conditions of the work area and without requiring the installation of additional equipment or equipment worn by workers. [Brief explanation of the drawing]

[0017] [Figure 1]FIG. 1 is a block diagram showing the configuration of a crane operation detection system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram for explaining the characteristics of the crane operation performed at the site shown in FIG. 1. [Figure 3] FIG. 3 is a flowchart showing the flow of a crane operation detection process according to an embodiment of the present invention. [Figure 4] FIG. 4 is a flowchart showing the flow of an object detection process for alerting attention shown in FIG. 3. [Figure 5] FIG. 5 is a diagram for explaining the object detection process for alerting attention shown in FIG. 4.

MODE FOR CARRYING OUT THE INVENTION

[0018] Hereinafter, with reference to the drawings, the configuration and operation of a crane operation detection system according to an embodiment of the present invention will be described. Note that although this embodiment applies the present invention to crane operations, the application range of the present invention is not limited to crane operations, and the present invention can be applied to all operations around equipment that moves mechanically.

[0019] 〔Configuration〕 First, referring to FIG. 1, the configuration of a crane operation detection system according to an embodiment of the present invention will be described.

[0020] FIG. 1 is a block diagram showing the configuration of a crane operation detection system according to an embodiment of the present invention. As shown in FIG. 1, a crane operation detection system 1 according to an embodiment of the present invention is a system for suppressing the occurrence of being sandwiched or collided with a suspended load by an operator during a crane operation, and includes an information processing device 2 and a photographing device 3 as main components. However, the information processing device 2 and the photographing device 3 may be configured as one device.

[0021] The information processing device 2 and the imaging device 3 are configured to communicate information with each other via a telecommunications line. Examples of telecommunications lines include the Internet, a local area network, or a combination of the Internet and a local area network. The communication method can be wired communication, wireless communication, or a combination of wired and wireless communication. The information processing device 2 and the imaging device 3 may also be configured to communicate information directly with each other without using a telecommunications line. In this case as well, the communication method can be wired communication, wireless communication, or a combination of wired and wireless communication. In this case, for wired communication, signal lines connecting the communication units or USB (Universal Serial Bus) can be used, and for wireless communication, Bluetooth® can be used.

[0022] Information processing device 2 is composed of a known information processing device such as a computer and is installed in a control room where a manager responsible for managing the execution of crane operations is stationed. Information processing device 2 functions as an object detection unit 21, a warning target determination unit 22, and a hazard determination unit 23 by having its internal CPU (Central Processing Unit) and other processing units execute computer programs. The functions of each of these units will be described later. Each unit may be distributed among other information processing devices, and information may be exchanged between each unit via telecommunication lines. Information processing device 2 is equipped with an object information database (object information DB) 24. The object information DB 24 stores object information used by the warning target determination unit 22 when determining an object to be warned about. Examples of object information include information regarding the specified dimensions of an object (or average height if the object is a worker).

[0023] The imaging device 3 is composed of a known imaging device such as a CCD (Charge Coupled Device) camera and is installed at any position on the site IM where the crane operation is performed. The imaging device 3 captures still images and videos of the site IM according to control signals from the information processing device 2, and outputs the captured digital image data to the information processing device 2 as monitoring image data. The resolution of the monitoring image data is set to a range in which the crane hook suspending the load, the worker, etc., can be identified from the monitoring image data.

[0024] [Characteristics of crane operations] Next, with reference to Figures 2(a) and (b), we will describe the characteristics of the crane operations performed at the above-mentioned on-site IM.

[0025] Figure 2(a) shows an example of a site IM image taken by the camera 3, and Figure 2(b) is a diagram illustrating the crane work performed at the site IM shown in Figure 2(a). In the example shown in Figure 2(b), the crane work involves rigging, where the eye of the wire W to which the load is tied is attached to the crane hook F to suspend the load, or the eye of the wire W attached to the crane hook F is attached to the load to suspend the load. In this rigging work, a crane supervisor (a person performing tasks specific to rigging) P1, who gives instructions (poses) that are uniformly determined for the construction unit, a load guide (a guide who operates the load) P2, who is holding the necessary jigs (hook rods, guide ropes, etc.) to guide the load, and a worker P3, who is handling the load directly below the crane, are working in the crane work area R1. In addition, there is a worker P4 who is working in area R2, which is adjacent to the crane work area R1, but is not directly involved in the rigging work.

[0026] In this type of lifting operation, the crane hook F to which the wire W is attached, the crane operator P1, the load guide P2, and the workers P3 and P4 become objects that require attention to prevent being caught in or colliding with the suspended load. Therefore, in the crane operation detection system 1 of this embodiment, the information processing device 2 detects objects that require attention from the monitoring image data output from the camera 3 by executing the crane operation detection process shown below, and issues a warning based on the location information of the detected objects that require attention. The operation of the information processing device 2 (crane operation detection method) when executing the crane operation detection process will be explained below with reference to Figures 3 to 5.

[0027] [Crane operation detection processing] Figure 3 is a flowchart showing the flow of the crane operation detection process, which is one embodiment of the present invention. Figure 4 is a flowchart showing the flow of the attention-raising object detection process shown in Figure 3. The flowchart shown in Figure 3 starts when the crane operation detection system 1 is activated, and the crane operation detection process proceeds to step S1. The crane operation detection process is repeatedly executed at predetermined control cycles while the crane operation detection system 1 is activated.

[0028] In step S1, the object detection unit 21 acquires monitoring image data from the imaging device 3 and inputs the acquired monitoring image data into the object detection program to detect the type of monitored object and its image data included in the monitoring image data. The object detection unit 21 then outputs the detected type of monitored object and its image data to the attention target determination unit 22. The object detection program is a machine learning model that uses monitoring image data as input data and the type of monitored object and its image data included in the monitoring image data as output data. This machine learning model can be generated by performing machine learning using historical data of monitoring image data and pairs of the type of monitored object and its image data as training data.

[0029] Any machine learning model is acceptable as long as it provides sufficient detection accuracy for practical use. For example, commonly used neural networks (including deep learning and convolutional neural networks), decision tree learning, random forests, and support vector regression can be used. Ensemble models combining multiple models can also be used. Examples of monitored object types include workers, jigs, crane hooks F, wires W, and suspended loads. The detection accuracy of monitored objects can be improved by machine learning the image data of each type of monitored object, captured under various conditions (orientation and posture of the monitored object, time of capture, etc.). With this, the processing of step S1 is completed, and the crane operation detection process proceeds to the processing of step S2.

[0030] In step S2, the attention target determination unit 22 calculates the coordinate position of each monitored object using the type of monitored object and image data detected by the object detection unit 21. For example, as shown in Figures 5(a) and (b), if the monitored objects are workers P5 and P6, the attention target determination unit 22 extrapolates a rectangular frame onto the image data of workers P5 and P6. The attention target determination unit 22 then calculates the coordinates of the intersection points between the frame lines and the heads of workers P5 and P6 as the coordinate positions of workers P5 and P6 (Xatop, Yatop) and (Xdtop, Ydtop). For example, as shown in Figure 5(c), if the monitored object is a crane hook F, the attention target determination unit 22 extrapolates a rectangular frame containing the image data of the crane hook F onto the image data of the crane hook F. The attention target determination unit 22 then calculates the coordinates of the intersection points between the frame line and the upper and lower ends of the crane hook F as the coordinate positions of the crane hook F (Xctop, Yctop) and (Xcbot, Ycbot). With this, the processing of step S2 is completed, and the crane operation detection process proceeds to the processing of step S3.

[0031] In step S3, the attention target discrimination unit 22 detects an object to be alerted from among the monitored objects detected by the object detection unit 21. Specifically, as shown in Figure 4, first, the attention target discrimination unit 22 determines the object to be alerted based on the state of the monitored object, such as its posture, possessions, and accessories (step S31). For example, if the monitored object is in the form of a worker in a predetermined pose, the attention target discrimination unit 22 determines that the monitored object is a crane operator P1. For example, if the monitored object is in the form of a worker holding a jig, the attention target discrimination unit 22 determines that the monitored object is a load guide P2. This determination process may be performed using a machine learning model trained with image data for each worker classification.

[0032] Next, the attention target determination unit 22 determines the attention target object based on the position information of the monitored object (step S32). For example, as shown in Figure 5(c), the attention target determination unit 22 sets the proximity range of the rectangular crane hook F wl (=Hc×α1), wr (=Hc×α2), ht (=Hc×α3), and hb (=Hc×α4) based on the height Hc of the crane hook F. α1 to α4 are arbitrary coefficients. The attention target determination unit 22 may correct the number of pixels corresponding to each monitored object (the range of the monitored object in the image) based on the number of pixels corresponding to each monitored object and the specified dimensions of the object stored in the object information DB 24, so that the size of each monitored object corresponds to the specified dimensions of the object. Then, the attention target determination unit 22 determines the worker whose coordinate position calculated in step S2 is within the proximity range as the attention target object. For example, if there are multiple workers whose coordinate positions are within the proximity range, the attention target determination unit 22 determines that the multiple workers are workers P4 located in area R2 adjacent to the crane work area R1. For example, if there is one worker whose coordinate position is within the proximity range, the attention target determination unit 22 determines that the worker is worker P3 touching the suspended load directly below the crane hook F. The attention target determination unit 22 may also perform the above processing when the height Hc of the crane hook F is within a predetermined upper and lower limit range. This processing can suppress the misjudgment that a worker is directly below the crane hook F, even though they are actually behind or in front of the crane hook F, due to the shooting angle (overhead viewing angle). With this, the processing of step S3 is completed, and the crane work detection process proceeds to the processing of step S4.

[0033] Returning to Figure 3, in step S4, the hazard determination unit 23 determines whether the monitored object, such as a suspended load, poses a high risk based on the location information of the object to be warned detected in step S3. Specifically, the hazard determination unit 23 determines that the monitored object poses a high risk to the object to be warned if the distance between the object to be warned and the monitored object falls below a predetermined value. If the hazard determination unit 23 determines that the monitored object poses a high risk to the object to be warned, it executes various processes to issue a warning to the object to be warned. Specifically, for on-site personnel, the hazard determination unit 23 issues an alarm, lights up a patrol light (registered trademark), and notifies worker terminal devices (such as mobile terminals and smartphones). For administrators, the hazard determination unit 23 notifies administrator terminal devices (such as mobile terminals and smartphones) and records video when a risk is detected. When the worker terminal device or administrator terminal device receives a notification from the hazard determination unit 23 via the telecommunications line, it issues a warning about the object requiring attention through voice, screen display, SMS, app notification, etc. In this case, the functions of the worker terminal device or administrator terminal device are realized by the computer within the terminal device executing a computer program. With this, the processing in step S4 is completed, and the series of crane operation detection processes are finished.

[0034] As is clear from the above explanation, in the crane work detection process, which is one embodiment of the present invention, the information processing device 2 inputs the image of the crane work area as input data to a machine learning model that takes the image of the crane work area as input data and the image data of the objects to be monitored contained in the image of the crane work area as output data. The device extracts the objects to be monitored from the image of the crane work area, determines which objects to be warned about from among the objects to be monitored based on the position and size of the extracted objects to be monitored, and determines the danger posed by the objects to be warned about to the objects to be warned about according to the position of the objects to be warned about. With this configuration, warnings can be issued to workers solely by information processing using image data of the crane work area, thus improving the safety of crane operations without imposing restrictions on the shooting conditions of the crane work area and without requiring the installation of additional equipment or equipment worn by workers.

[0035] Although embodiments applying the invention made by the present inventors have been described above, the present invention is not limited by the descriptions and drawings that constitute part of the disclosure of the present invention in this embodiment. That is, all other embodiments, examples, and operational techniques made by those skilled in the art based on this embodiment are included in the scope of the present invention. [Explanation of Symbols]

[0036] 1. Crane operation detection system 2. Information Processing Device 3. Imaging device 21 Object detection unit 22 Warning Target Identification Unit 23. Hazard Assessment Department 24. Object Information Database (Object Information DB) F Crane Hook IM Field Site P1 Crane Operator P2 Hanging load guide P3,P4 worker W wire

Claims

1. A camera that takes images of the work area, Equipped with an information processing device, The aforementioned information processing device is A machine learning model takes an image of the work area as input data and the type of monitored object (which is one of the following: worker, jig, crane hook, wire, or suspended load) and its image data as output data. By inputting an image captured by the aforementioned camera, the model detects the type of monitored object and its image data contained in the image captured by the camera. The type of monitored object detected and the image data are used to calculate the coordinate position of each monitored object. Based on the location and size of the detected monitored object, a worker will be selected from among the monitored objects to be designated as a target for attention. A work detection system that determines the risk to a worker, excluding the worker, based on the determined worker's position.

2. The work detection system according to claim 1, wherein the information processing device determines a worker to be designated as an object to be alerted based on the relative positional relationship of the monitored objects.

3. The work detection system according to claim 1, wherein the information processing device determines a worker to be designated as an object to be alerted based on the posture, possessions, and accessories of the object to be monitored.

4. The work detection system according to claim 1, wherein the work area is a crane work area for transporting a suspended load by a crane hook.

5. An information processing device that takes an image of a work area as input data and the type of monitored object and image data of any of the monitored objects (workers, jigs, crane hooks, wires, and suspended loads) contained in the image as output data, detects the type of monitored object and image data contained in the image of the work area by inputting an image of the work area, calculates the coordinate position of each monitored object using the detected type of monitored object and image data, determines which worker should be designated as a cautionary object from among the monitored objects based on the position and size of the detected monitored object, and determines the risk to the worker from the monitored objects excluding the worker according to the position of the determined worker.

6. A program for an information processing device that takes an image of a work area as input data and the type of monitored object (which is one of the following: a worker, a jig, a crane hook, a wire, or a suspended load) and its image data as output data, inputs an image of a work area to a machine learning model, detects the type of monitored object and its image data contained in the image of the work area, calculates the coordinate position of each monitored object using the detected type of monitored object and its image data, determines which worker should be designated as a cautionary object from among the monitored objects based on the position and size of the detected monitored object, and then causes the computer to execute a process to determine the risk to the worker from the monitored objects excluding the worker, according to the position of the determined worker.

7. A terminal device that, when an image of a work area is input to a machine learning model that takes an image of a work area as input data and the type of monitored object (which is one of the following: worker, jig, crane hook, wire, or suspended load) and image data contained in the image as output data, detects the type of monitored object and image data contained in the image of the work area, calculates the coordinate position of each monitored object using the detected type of monitored object and image data, determines which worker will be designated as a caution target object from among the monitored objects based on the position and size of the detected monitored objects, and, according to the determination result of an information processing device that determines the danger of monitored objects other than the worker to the determined worker based on the position of the determined worker, issues a warning to the worker designated as a caution target object.

8. A terminal device program that takes an image of a work area as input data and the type of monitored object (which is one of the following: worker, jig, crane hook, wire, or suspended load) and image data contained in the image as output data, inputs an image of the work area to a machine learning model, detects the type of monitored object and image data contained in the image of the work area, calculates the coordinate position of each monitored object using the detected type of monitored object and image data, determines which worker will be designated as a caution target object from among the monitored objects based on the position and size of the detected monitored objects, and, according to the determination result of an information processing device that determines the danger of monitored objects other than the worker to the determined worker based on the position of the determined worker, causes the computer to execute a process to issue a warning to the worker designated as a caution target object.

9. The shooting step involves taking images of the work area, A machine learning model takes an image of the work area as input data and the type of monitored object (which is one of the following: worker, jig, crane hook, wire, or suspended load) and image data contained in the image as output data. By inputting the image taken in the shooting step, the model detects the type of monitored object and image data contained in the image taken in the shooting step, calculates the coordinate position of each monitored object using the detected type of monitored object and image data, determines which worker will be designated as a cautionary object from among the monitored objects based on the position and size of the detected monitored object, and determines the risk to the worker excluding the worker according to the position of the determined worker. A work detection method, including the following.

Images