Work detection system, information processing device, program for information processing device, terminal device, program for terminal device, and work detection method

The work detection system uses image analysis and machine learning to enhance crane safety by identifying potential dangers without restrictive photographing conditions or additional worker equipment, improving alert accuracy and reducing costs.

JP2024156491AActive Publication Date: 2024-11-06JFE STEEL CORP
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
JP2023070998
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-04-24
Publication Date
2024-11-06
Estimated Expiration
2043-04-24

AI Technical Summary

Technical Problem

Existing crane safety technologies require restrictive photographing conditions and additional equipment for workers, necessitating costly installations and pre-work inspections to ensure proper equipment wear, which is cumbersome and inefficient.

Method used

A work detection system utilizing a photographing device and machine learning model to analyze images of the work area, extracting and monitoring objects based on their position and size, determining potential dangers without additional worker equipment, and issuing alerts through information processing.

Benefits of technology

Enhances crane safety by improving alert accuracy and reducing the need for restrictive photographing conditions and additional worker equipment, ensuring efficient and cost-effective safety enhancements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024156491000001_ABST
    Figure 2024156491000001_ABST
Patent Text Reader

Abstract

To provide a work detection system capable of improving work safety without being constrained by photographing conditions of a work area and without needing additional devices or worker wearing equipment, an information processing device, a program for the information processing device, a terminal device, a program for the terminal device, and a work detection method.SOLUTION: This work detection system comprises: a photographing device that captures an image of a work area; and an information processing device that inputs the image captured by the photographing device to a machine learning model in which the image of the work area is input data and image data of objects to be monitored included in the image is output data so as to extract an object to be monitored from the image captured by the photographing device, determines an object to be warned about from among the objects to be monitored on the basis of the position and the size of the extracted object to be monitored, and determines the danger of the object to be monitored with respect to the object to be warned about according to the position of the object to be warned about.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an activity detection system, an information processing device, a program for an information processing device, a terminal device, a program for a terminal device, and an activity detection method. [Background technology]

[0002] The techniques described in Patent Documents 1 and 2 are known as techniques for improving the safety of crane work. Specifically, Patent Document 1 describes a device that includes a camera that is installed on the body of a crane and captures the crane work area from directly above, and an infrared spotlight installed near the camera. The device described in Patent Document 1 uses a camera to detect infrared light reflected from a mark made of a retroreflective material attached to the worker's helmet, and identifies the worker's position in the crane work area, thereby confirming the worker's safety. Meanwhile, Patent Document 2 describes a system that includes a camera that captures the lower part of a suspended load attached to a crane, and a range finder that is installed near the camera and measures the distance to the work surface. The system described in Patent Document 2 adjusts the display size of a danger area on a monitor according to the camera, whose height changes, and the distance to the work surface, and determines whether or not to issue an alarm signal by identifying the worker based on the color of the helmet. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-127372 A [Patent Document 2] Patent Publication No. 2022-82974 Summary of the Invention [Problem to be solved by the invention]

[0004] The techniques described in Patent Documents 1 and 2 have restrictions on the conditions for photographing the crane work area, in that the crane work area must be photographed from directly above. In addition, it is necessary to install additional measuring equipment to determine the worker's position, which requires a lot of cost. Furthermore, since the worker's position is determined using the worker's equipment, such as a mark or the color of a helmet, additional work such as a pre-work inspection of the equipment is required to eliminate the influence of forgetting to wear the equipment, or missing or dirty equipment.

[0005] The present invention has been made to solve the above-mentioned problems, and its object is to provide a work detection system, an information processing device, a program for an information processing device, a terminal device, a program for a terminal device, and a work detection method that can improve work safety without imposing restrictions on the shooting conditions of the work area, and without requiring the installation of additional equipment or wear by the worker. [Means for solving the problem]

[0006] The work detection system of the present invention comprises an imaging device that captures images of a work area, and an information processing device that inputs the images captured by the imaging device into a machine learning model in which the image of the work area is input data and image data of a monitored object contained within the image is output data, thereby extracting monitored objects from the images captured by the imaging device, determining a target object for attention from among the monitored objects based on the position and size of the extracted monitored object, and determining the dangerousness of the monitored object relative to the target object for attention based on the position of the target object for attention.

[0007] The information processing device may determine an object for which attention should be drawn based on a relative positional relationship of the monitored objects.

[0008] The information processing device may determine an object to which attention should be drawn based on the posture, belongings, and accessories of the monitored object.

[0009] The work area may be a crane work area in which a load is transported by a crane hook.

[0010] The information processing device of the present invention inputs an image of the work area into a machine learning model, which has an image of the work area as input data and image data of the monitored object contained in the image as output data, thereby extracting monitored objects from the image of the work area, determining an object to be warned about from among the monitored objects based on the position and size of the extracted monitored objects, and determining the dangerousness of the monitored object relative to the object to be warned about based on the position of the object to be warned about.

[0011] The program of the information processing device of the present invention causes the computer to execute a process of inputting an image of the work area into a machine learning model in which an image of the work area is used as input data and image data of the monitored object contained in the image is used as output data, thereby extracting monitored objects from the image of the work area, determining an object to be warned about from among the monitored objects based on the position and size of the extracted monitored objects, and judging the dangerousness of the monitored object relative to the object to be warned about based on the position of the object to be warned about.

[0012] The terminal device of the present invention inputs an image of the work area into a machine learning model which has an image of the work area as input data and image data of the object to be monitored contained in the image as output data, thereby extracting objects to be monitored from the image of the work area, determining an object to be warned about from among the objects to be monitored based on the position and size of the extracted objects to be monitored, and issuing a warning to the object to be warned about based on the judgment result of an information processing device which judges the dangerousness of the object to be monitored to the object to be warned about based on the position of the object to be warned about.

[0013] The program of the terminal device of the present invention causes a computer to execute a process of extracting monitored objects from the image of the work area by inputting an image of the work area into a machine learning model in which an image of the work area is used as input data and image data of monitored objects contained in the image is used as output data, determining an object to be warned about from among the monitored objects based on the position and size of the extracted monitored objects, and issuing a warning to the object to be warned about based on the judgment result of an information processing device which judges the dangerousness of the monitored objects to the object to be warned about based on the position of the object to be warned about.

[0014] The terminal device according to the present invention issues an attention-call to the attention-call target object in response to an instruction from the information processing device according to the present invention.

[0015] The work detection method of the present invention includes a photographing step of photographing an image of a work area, and a step of extracting monitored objects from the image photographed in the photographing step by inputting the image photographed in the photographing step into a machine learning model in which the image of the work area is input data and image data of monitored objects contained in the image is output data, determining an attention-request object from among the monitored objects based on the position and size of the extracted monitored object, and determining the dangerousness of the monitored object relative to the attention-request object based on the position of the attention-request object. Effect of the Invention

[0016] According to the work detection system, information processing device, program for the information processing device, terminal device, program for the terminal device, and work detection method of the present invention, it is possible to alert the worker simply by information processing using image data of the work area, thereby improving work safety without placing restrictions on the shooting conditions of the work area and without requiring the installation of additional equipment or clothing worn by the worker. [Brief description of the drawings]

[0017] [Figure 1]FIG. 1 is a block diagram showing the configuration of a crane operation detection system according to one embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram for explaining features of the crane operation performed at the work site shown in FIG. [Diagram 3] FIG. 3 is a flowchart showing the flow of the crane operation detection process according to one embodiment of the present invention. [Figure 4] FIG. 4 is a flowchart showing the flow of the attention-attention target object detection process shown in FIG. [Diagram 5] FIG. 5 is a diagram for explaining the attention-attracting object detection process shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, the configuration and operation of a crane operation detection system according to an embodiment of the present invention will be described with reference to the drawings. Note that, although this embodiment is an application of the present invention to crane operation, the scope of application of the present invention is not limited to crane operation, and the present invention can be applied to all work around mechanically movable equipment.

[0019] 〔composition〕 First, with reference to FIG. 1, a configuration of a crane operation detection system according to an embodiment of the present invention will be described.

[0020] Fig. 1 is a block diagram showing the configuration of a crane operation detection system according to one embodiment of the present invention. As shown in Fig. 1, the crane operation detection system 1 according to one embodiment of the present invention is a system for preventing a worker from being pinched or colliding with a suspended load during crane operation, and includes an information processing device 2 and a photographing device 3 as main components. However, the information processing device 2 and the photographing device 3 may be configured as a single device.

[0021] The information processing device 2 and the photographing device 3 are configured to be able to communicate information with each other via an electric communication line. Examples of the electric communication line include the Internet, a local area network, or a combination of the Internet and a local area network. The communication method can be wired communication, wireless communication, or a combination of wired communication and wireless communication. The information processing device 2 and the photographing device 3 may be configured to be able to communicate information with each other directly without using an electric communication line. In this case, the communication method can also be wired communication, wireless communication, or a combination of wired communication and wireless communication. In this case, a signal line connecting the communication units or USB (Universal Serial Bus) can be used for wired communication, and Bluetooth (registered trademark) can be used for wireless communication.

[0022] The information processing device 2 is configured by a known information processing device such as a computer, and is installed in a management room where a manager who manages the implementation of crane work resides. The information processing device 2 functions as an object detection unit 21, an attention target discrimination unit 22, and a danger determination unit 23 by executing a computer program with an internal arithmetic processing device such as a CPU (Central Processing Unit). The functions of these units will be described later. Each unit may be distributed and arranged in other information processing devices, and information may be exchanged between each unit via an electric communication line. The information processing device 2 includes an object information database (object information DB) 24. The object information DB 24 stores object information used by the attention target discrimination unit 22 when determining an attention target object. The object information may be, for example, information on the specified dimensions of the object (such as the average height if the object is a worker).

[0023] The photographing device 3 is composed of a known photographing device such as a CCD (Charge Coupled Device) camera, and is installed at an arbitrary position on the site IM where the crane work is performed. The photographing device 3 photographs still images and videos of the site IM in accordance with a control signal from the information processing device 2, and outputs the photographed image data in digital form to the information processing device 2 as monitoring image data. The resolution of the monitoring image data is set within a range in which the crane hook suspending the load, the worker, etc. can be determined from the monitoring image data.

[0024] [Characteristics of crane work] Next, the characteristics of the crane work carried out at the above-mentioned on-site site IM will be described with reference to Figs.

[0025] FIG. 2(a) is a diagram showing an example of an image of the work site IM photographed by the photographing device 3, and FIG. 2(b) is a diagram for explaining the crane work performed at the work site IM shown in FIG. 2(a). In the example shown in FIG. 2(b), the crane work is a slinging work in which the eye of the wire W to which the load is tied is hung on the crane hook F to hang the load, or the eye of the wire W hung on the crane hook F is attached to the load to hang the load. In this slinging work, the crane conductor (a person performing work specific to slinging work) P1 who gives instructions (poses) uniformly determined for each construction unit, the load guide (guide who operates the load) P2 who holds the jig (claw bar, guide rope, etc.) necessary for guiding the load, and the worker P3 who touches the load directly below the crane are working in the crane work area R1. There is also a worker P4 who is not directly involved in the slinging work but is working in an area R2 adjacent to the crane work area R1.

[0026] In such slinging work, the crane hook F to which the wire W is attached, the crane commander P1, the load guide P2, and the workers P3 and P4 become objects that require attention in order to prevent pinching or collision with the load. In the crane work detection system 1 of this embodiment, the information processing device 2 executes the crane work detection process described below to detect objects that require attention from the monitoring image data output from the image capture device 3, and issues a warning based on position information and the like of the detected object that requires attention. Hereinafter, the operation of the information processing device 2 when executing the crane work detection process (crane work detection method) will be described with reference to Figs. 3 to 5.

[0027] [Crane operation detection processing] Fig. 3 is a flowchart showing the flow of the crane operation detection process according to one embodiment of the present invention. Fig. 4 is a flowchart showing the flow of the attention-attracting object detection process shown in Fig. 3. The flowchart shown in Fig. 3 starts when the crane operation detection system 1 is started, and the crane operation detection process proceeds to step S1. The crane operation detection process is repeatedly executed at predetermined control intervals while the crane operation detection system 1 is started.

[0028] In the process of step S1, the object detection unit 21 acquires surveillance image data from the imaging device 3, and inputs the acquired surveillance image data into an object detection program to detect the type and image data of a surveillance object contained in the surveillance image data. Then, the object detection unit 21 outputs the type and image data of the detected surveillance object to the attention target discrimination unit 22. The object detection program is a machine learning model in which the surveillance image data is input data, and the type and image data of a surveillance object contained in the surveillance image data are output data. This machine learning model can be generated by executing machine learning using a set of the performance data of the surveillance image data and the type and image data of a surveillance object contained in the surveillance image data as teacher data.

[0029] The machine learning model may be any machine learning model as long as it can provide sufficient detection accuracy for practical use. For example, commonly used neural networks (including deep learning and convolutional neural networks), decision tree learning, random forests, support vector regression, etc. may be used. An ensemble model combining multiple models may be used. Examples of the types of monitored objects include workers, jigs, crane hooks F, wires W, and suspended loads. By machine learning image data for each type of monitored object photographed under various conditions (the orientation and posture of the monitored object, the photographing time, etc.), the detection accuracy of the monitored object can be improved. This completes the process of step S1, and the crane operation detection process proceeds to the process of step S2.

[0030] In the process of step S2, the attention target discrimination unit 22 calculates the coordinate position of each monitored object using the type and image data of the monitored object detected by the object detection unit 21. For example, as shown in Figs. 5(a) and (b), when the monitored objects are workers P5 and P6, the attention target discrimination unit 22 extrapolates a rectangular frame to the image data of the workers P5 and P6. Then, the attention target discrimination unit 22 calculates the coordinates of the intersection between the frame line and the heads of the workers P5 and P6 as the coordinate positions (Xatop, Yatop), (Xdtop, Ydtop) of the workers P5 and P6. For example, as shown in Fig. 5(c), when the monitored object is a crane hook F, the attention target discrimination unit 22 extrapolates a rectangular frame including the image data of the crane hook F to the image data of the crane hook F. Then, the attention target determination unit 22 calculates the coordinates of the intersections between the frame line and the upper end and lower end of the crane hook F as the coordinate positions (Xctop, Yctop), (Xcbot, Ycbot) of the crane hook F. This completes the processing of step S2, and the crane work detection processing proceeds to the processing of step S3.

[0031] In the process of step S3, the attention target discrimination unit 22 detects an attention target object from among the monitored objects detected by the object detection unit 21. Specifically, as shown in FIG. 4, first, the attention target discrimination unit 22 determines an attention target object based on the state of the monitored object, such as the posture, belongings, and accessories of the monitored object (step S31). For example, when the monitored object is in the form of a worker making a predetermined pose, the attention target discrimination unit 22 determines that the monitored object is a crane director P1. For example, when the monitored object is in the form of a worker holding a jig, the attention target discrimination unit 22 determines that the monitored object is a load guide P2. This determination process may be performed using a machine learning model trained using image data for each classification of workers.

[0032] Next, the attention target discrimination unit 22 determines the object to be alerted based on the position information of the object to be monitored (step S32). For example, as shown in FIG. 5(c), the attention target discrimination unit 22 sets the proximity ranges wl (=Hc×α1), wr (=Hc×α2), ht (=Hc×α3), and hb (=Hc×α4) of the rectangular crane hook F based on the height Hc of the crane hook F. α1 to α4 are arbitrary coefficients. Note that the attention target discrimination unit 22 may correct the number of pixels corresponding to each monitored object (the range of the monitored object in the image) based on the number of pixels corresponding to each monitored object and the specified dimensions of the object stored in the object information DB 24 so that the size of each monitored object corresponds to the specified dimensions of the object. Then, the attention target discrimination unit 22 determines the worker whose coordinate position calculated in the process of step S2 is within the proximity range as the object to be alerted. For example, when there are multiple workers whose coordinate positions are within the proximity range, the attention target discrimination unit 22 determines that the multiple workers are worker P4 who is in area R2 that is close to the crane work area R1. For example, when there is one worker whose coordinate position is within the proximity range, the attention target discrimination unit 22 determines that the worker is worker P3 who is touching the suspended load directly below the crane hook F. Note that the attention target discrimination unit 22 may perform the above process when the height Hc of the crane hook F is within a predetermined upper and lower limit range. This process can prevent the worker from being erroneously determined to be directly below the crane hook F due to the shooting angle (bird's-eye view angle) when he or she is actually behind or in front of the crane hook F. This completes the process of step S3, and the crane work detection process proceeds to the process of step S4.

[0033] Returning to FIG. 3, in the process of step S4, the danger determination unit 23 determines whether or not the danger of the monitored object, such as a suspended load, is high based on the position information of the attention-attracting object detected in the process of step S3. Specifically, when the distance between the attention-attracting object and the monitored object becomes less than a predetermined value, the danger determination unit 23 determines that the monitored object is highly dangerous to the attention-attracting object. Then, when it is determined that the monitored object is highly dangerous to the attention-attracting object, the danger determination unit 23 executes various processes to draw attention to the attention-attracting object. Specifically, the danger determination unit 23 issues an alarm, turns on a PATLITE (registered trademark), notifies a worker terminal device (such as a mobile terminal device or a smartphone), and the like, for the site. In addition, the danger determination unit 23 notifies a manager terminal device (such as a mobile terminal device or a smartphone), records video when a danger is detected, and the like, for the manager. When the worker terminal device or the manager terminal device receives a notification from the danger determination unit 23 via the electric communication line, it calls attention to the attention-requiring object by voice, screen display, SMS, app notification, etc. In this case, the functions of the worker terminal device or the manager terminal device are realized by a computer in the terminal device executing a computer program. This completes the process of step S4, and the series of crane operation detection processes ends.

[0034] As is clear from the above description, in the crane work detection process according to one embodiment of the present invention, the information processing device 2 inputs an image of the crane work area into a machine learning model in which the image of the crane work area is input data and the image data of the monitored object contained in the image of the crane work area is output data, thereby extracting the monitored object from the image of the crane work area, determining an attention-drawing object from among the monitored objects based on the position and size of the extracted monitored object, and judging the danger of the monitored object relative to the attention-drawing object according to the position of the attention-drawing object. According to this configuration, the worker can be alerted only by information processing using image data of the crane work area, and therefore the safety of the crane work can be improved without imposing restrictions on the shooting conditions of the crane work area and without requiring additional equipment or equipment worn by the worker.

[0035] Although the embodiments of the present invention have been described above, the present invention is not limited by the descriptions and drawings that form part of the disclosure of the present invention according to the present embodiments. In other words, other embodiments, examples, and operation techniques, etc., made by those skilled in the art based on the present embodiments are all included in the scope of the present invention. [Explanation of symbols]

[0036] 1. Crane operation detection system 2. Information processing equipment 3. Imaging Equipment 21 Object detection unit 22 Warning target discrimination unit 23 Risk Assessment Department 24 Object Information Database (Object Information DB) F Crane hook IM On-site Site P1 Crane Conductor P2 Hanging load guide P3,P4 worker W Wire

Claims

1. an imaging device for capturing an image of the work area; an information processing device which extracts monitored objects from the images captured by the photographing device by inputting images captured by the photographing device into a machine learning model in which an image of a work area is used as input data and image data of monitored objects contained in the image is used as output data, and which determines an attention-drawing target object from among the monitored objects based on the position and size of the extracted monitored objects, and judges the danger of the monitored objects relative to the attention-drawing target object based on the position of the attention-drawing target object; A work detection system comprising:

2. The activity detection system according to claim 1 , wherein the information processing device determines an object for which attention should be drawn based on a relative positional relationship of the monitored objects.

3. The operation detection system according to claim 1 , wherein the information processing device determines an object to be alerted to based on a posture, belongings, and accessories of the monitored object.

4. The operation detection system according to claim 1 , wherein the operation area is a crane operation area in which a load is transported by a crane hook.

5. An information processing device that inputs an image of a work area into a machine learning model that has an image of the work area as input data and image data of monitored objects contained within the image as output data, extracts monitored objects from the image of the work area, determines from among the monitored objects a target object for attention based on the position and size of the extracted monitored object, and judges the dangerousness of the monitored object relative to the target object for attention based on the position of the monitored object.

6. A program for an information processing device that causes a computer to execute a process of inputting an image of the work area into a machine learning model in which an image of the work area is used as input data and image data of monitored objects contained within the image is used as output data, extracting monitored objects from the image of the work area, determining objects to be warned about from among the monitored objects based on the positions and sizes of the extracted monitored objects, and judging the danger of the monitored objects relative to the monitored objects to be warned about based on the position of the objects to be warned about.

7. A terminal device which extracts monitored objects from the image of the work area by inputting an image of the work area into a machine learning model which has an image of the work area as input data and image data of monitored objects contained within the image as output data, determines an object to be warned about from among the monitored objects based on the position and size of the extracted monitored objects, and issues a warning to the object to be warned about based on the judgment result of an information processing device which judges the dangerousness of the monitored objects relative to the object to be warned about based on the position of the object to be warned about.

8. A program for a terminal device that causes a computer to execute a process of extracting monitored objects from the image of the work area by inputting an image of the work area into a machine learning model in which an image of the work area is used as input data and image data of monitored objects contained within the image as output data, determining an object to be warned about from among the monitored objects based on the position and size of the extracted monitored objects, and issuing a warning to the object to be warned about in accordance with the judgment result of an information processing device that judges the dangerousness of the monitored objects relative to the object to be warned about in accordance with the position of the object to be warned about.

9. An image capturing step of capturing an image of the work area; a step of inputting the image captured in the photographing step into a machine learning model in which an image of the work area is used as input data and image data of the monitored object contained in the image is used as output data, thereby extracting the monitored object from the image captured in the photographing step, determining an object to be warned about from among the monitored objects based on the position and size of the extracted monitored object, and judging the danger of the monitored object relative to the object to be warned about based on the position of the object to be warned about; A work detection method comprising:

Citation Information

Patent Citations

  • Crane safety risk control method and system

    CN115571792A

  • Information processing device

    JP2021197083A

  • Crane monitoring device and crane monitoring method as well as overhead crane

    JP2022061682A

  • Utility vehicle, and device, method and computer program corresponding to utility vehicle

    JP2022151735A

  • Utility Vehicle and Corresponding Apparatus, Method and Computer Program for a Utility Vehicle

    US20220307231A1