Safety support system and how to create a trained model
The safety support system uses imaging and three-dimensional sensing with trained models to enhance crane safety by issuing real-time warnings, addressing the limitations of conventional systems that require design-level automation.
Patent Information
- Application Number
- JP2024067032
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2024-04-17
- Publication Date
- 2024-11-28
- Estimated Expiration
- 2044-04-17
AI Technical Summary
Conventional safety support systems for cranes require large-scale automation from the design stage and cannot be retrofitted to existing old crane equipment, limiting their application to improving safety in such setups.
A safety support system that includes an acquisition unit for imaging and three-dimensional sensing, coupled with a trained model to determine the necessity of alarms based on input images, allowing for real-time safety assessments and warnings without requiring design changes to existing crane equipment.
Enhances safety in crane operations by providing real-time warnings for potential accidents, improving safety without modifying the existing crane infrastructure.
Smart Images

Figure 0007591226000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a safety support system and a method for creating a trained model. [Background technology]
[0002] In workplaces where small and medium-sized heavy objects are handled, full automation is difficult, and it is essential that humans and machines coexist. In such workplaces, many industrial accidents occur every year due to the collapse of the objects to be lifted (suspended loads). In order to increase the safety of work at workplaces, cranes incorporating safety support systems have been proposed (see, for example, Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Special Publication No. 2015-518804 Summary of the Invention [Problem to be solved by the invention]
[0004] In most construction sites, it is not uncommon for old crane equipment to have been used for many years. However, conventional safety support systems (see, for example, Patent Document 1) required the construction of a large-scale system that was premised on automation, including safety support functions, from the crane design stage. Therefore, these safety support systems could not be applied to existing old crane equipment, and there were cases where it was not possible to improve the safety of crane work with existing equipment.
[0005] The present invention has been made in consideration of the above circumstances, and aims to provide a safety support system that can improve the safety of crane operations even in existing crane equipment, and a method for creating a trained model. [Means for solving the problem]
[0006] In order to solve the above problems, the safety support system of aspect 1 of the present invention comprises an acquisition unit that acquires an image including a ground worker, a hoisting device held by a crane, and a load being lifted by the hoisting device, a judgment unit that inputs the image into a trained model and judges whether or not an alarm by an alarm unit is necessary in the situation shown in the image, and a control unit that causes the alarm unit to issue the alarm based on the result of the judgment by the judgment unit, and the trained model is trained to judge whether or not the alarm is necessary when the image is input.
[0007] Further, a second aspect of the present invention relates to a safety support system of the first aspect, wherein the trained models include a first trained model and a second trained model, and the first trained model has been trained to output at least one of the position and status of the ground worker, the position of the hoisting equipment, and the position of the suspended load when the image is input, and the second trained model has been trained to output the work content of the crane when at least one of the position and status of the ground worker, the position of the hoisting equipment, and the position of the suspended load are input, and the judgment unit includes a first judgment unit that outputs at least one of the position and status of the ground worker, the position of the hoisting equipment, and the position of the suspended load by inputting the image into the first trained model, and a second judgment unit that outputs the work content of the crane by inputting the output by the first judgment unit into the second trained model.
[0008] Further, aspect 3 of the present invention relates to the safety support system of aspect 2, wherein the trained model includes a third trained model and a fourth trained model, the third trained model has been trained to output a danger area in the image when the work content of the crane and the image are input, and the fourth trained model has been trained to determine whether or not the alarm is needed when at least one of the position and status of the ground worker and the danger area are input, and the judgment unit includes a third judgment unit that outputs the danger area by inputting the image and the output by the second judgment unit into the third trained model, and a fourth judgment unit that judges whether or not the alarm is needed by inputting the output by the first judgment unit and the output by the third judgment unit into the fourth trained model.
[0009] Furthermore, aspect 4 of the present invention is a safety support system of any one of aspects 1 to 3, wherein the trained model includes a fifth trained model and a sixth trained model, the fifth trained model has been trained to output the position of the sling when the image is input, and the sixth trained model has been trained to determine whether or not the alarm is necessary when the position of the sling and the image are input, and the judgment unit includes a fifth judgment unit that outputs the position of the sling by inputting the image into the fifth trained model, and a sixth judgment unit that determines whether or not the alarm is necessary by inputting the image and the output by the fifth judgment unit into the sixth trained model.
[0010] Furthermore, in aspect 5 of the present invention, in a safety support system according to any one of aspects 1 to 4, the acquisition unit repeatedly acquires the image, the judgment unit repeatedly judges whether or not the alarm is necessary, and the control unit causes the alarm unit to issue the alarm based on the results of the repeated judgments of the judgment unit.
[0011] A sixth aspect of the present invention is the safety support system according to any one of the first to fifth aspects, wherein the control unit causes the alarm unit to issue a visually different alarm depending on the type of the alarm.
[0012] Furthermore, in aspect 7 of the present invention, in the safety support system of any one of aspects 1 to 6, the acquisition unit acquires height distribution information indicating a height distribution of the suspended load, and the trained model is trained to determine whether or not the alarm is necessary when the image and information related to the height distribution information are input.
[0013] Furthermore, in aspect 8 of the present invention, in the safety support system of aspect 7, a calculation unit is further provided which calculates deflection information indicating the deflection of the suspended load based on the height distribution information, and the trained model is trained to determine whether or not the alarm is necessary when the image and the deflection information are input.
[0014] Furthermore, a ninth aspect of the present invention relates to a safety support system according to any one of the first to eighth aspects, and further includes a learning processing unit that re-learns the trained model using a pair of the image and the need for an alarm corrected by an experienced worker as training data.
[0015] In order to solve the above problem, a method for creating a trained model according to aspect 10 of the present invention acquires a plurality of images including a ground worker, a lifting device held by a crane, and a load being lifted by the lifting device, acquires alarm necessity information that is linked one-to-one with the acquired plurality of images and indicates whether an alarm by an alarm unit is necessary in the situation shown in the linked images, and creates a trained model that learns the relationship between the content of the images and the necessity of the alarm using a pair of the images and the alarm necessity information linked to the images as training data. Effect of the Invention
[0016] According to the above aspects of the present invention, it is possible to provide a safety support system and a method for creating a trained model that can improve the safety of crane operations even in existing crane equipment. [Brief description of the drawings]
[0017] [Figure 1] 1 is a diagram showing an example of a crane to which a safety support system according to an embodiment is applied; [Diagram 2] 1 is a block diagram showing an example of a configuration of a safety support system according to an embodiment. [Diagram 3] FIG. 4 is a diagram illustrating an example of an alarm issued by an alarm unit according to the embodiment. [Figure 4] 2 is a diagram showing an example of a method for attaching an imaging unit, a distance sensor, and an alarm unit according to the embodiment to the crane shown in FIG. 1. [Diagram 5] 1A to 1C are diagrams showing examples of the shape of deflection of a suspended load. [Figure 6] 11A to 11C are diagrams showing other examples of the shape of the deflection of a suspended load. [Figure 7] FIG. 4 is a block diagram showing an example of a determination unit according to the embodiment. [Figure 8] FIG. 1 is a block diagram illustrating an example of a trained model according to an embodiment. [Figure 9] 5A to 5C are diagrams illustrating an example of an image acquired by an image acquisition unit and an output by a determination unit according to the embodiment. [Figure 10] FIG. 11 is a diagram showing an example of a change in a danger area depending on a working state of a crane. [Figure 11] FIG. 1 is a diagram showing an example of a position where a sling is attached to a suspended load. [Figure 12] 4 is a flowchart illustrating an example of a process performed by the safety support system according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0018] A safety support system S according to an embodiment of the present invention will be described below with reference to the drawings.
[0019] <Crane configuration and safety support system overview> The safety support system S according to this embodiment is a system that is applied to a crane (crane facility) and ensures the safety of crane work. A crane (crane facility) 1 shown in Fig. 1 is an example of an object to which the safety support system S is applied. The object to which the safety support system S is applied is not limited to the crane 1 in Fig. 1 and can be changed as appropriate.
[0020] Hereinafter, the direction along the gravity (i.e., the vertical direction) is referred to as the gravity direction Z. The upward direction in the gravity direction Z is represented by the +Z direction and simply referred to as "upward". The downward direction in the gravity direction Z is represented by the -Z direction and simply referred to as "downward". In addition, one direction that crosses (for example, perpendicular to) the gravity direction Z is referred to as the first direction X. One direction in the first direction X is represented by the +X direction and referred to as "right". The direction opposite to the +X direction is represented by the -X direction and referred to as "left". A direction that crosses (for example, perpendicular to) both the gravity direction Z and the first direction X is referred to as the second direction Y. One direction in the second direction Y is represented by the +Y direction and referred to as "forward". The direction opposite to the +Y direction is represented by the -Y direction and referred to as "backward". The first direction X and the second direction Y may be directions perpendicular to the gravity direction Z (ie, horizontal directions).
[0021] The crane 1 shown in Fig. 1 is a device that hoists a load L using a hoisting tool 22 (described later). The load L is an object to be hoisted by the crane 1, and may be, for example, a plate-shaped member. In the following, an example will be described in which the load L is a plate-shaped steel material having a rectangular shape in a plan view. However, the type and shape of the load L can be changed as appropriate.
[0022] The crane 1 includes a pair of rails 2, a pair of girders 3, a pair of moving parts 4, a hoisting device 5, a rope 6, a sheave 7, a hook 8, a trolley 10, and a hooking mechanism 20. The hooking mechanism 20 includes a rope 21 and a pair of hoists 22 (hackers). The crane 1 is operated by a crane operator U' operating the operation part 9.
[0023] Each of the pair of rails 2 extends in the second direction Y. The pair of rails 2 are arranged at an interval in the first direction X. The rails 2 guide the movement of the moving part 4 in the second direction Y (described later).
[0024] Each of the pair of girders 3 extends in a first direction X. The pair of girders 3 are spaced apart in a second direction Y. The girders 3 guide the movement (described later) of the trolley 10 in the first direction X.
[0025] The pair of moving parts 4 are provided at both ends of the pair of girders 3 in the first direction X. More specifically, one of the pair of moving parts 4 is provided so as to connect the right side (+X side) ends of the pair of girders 3, and the other of the pair of moving parts 4 is provided so as to connect the left side (-X side) ends of the pair of girders 3.
[0026] The moving part 4 (and the girder 3 connected thereto) moves along the rails 2 in the second direction Y in response to an operation performed on the operation part 9 by, for example, a crane operator U'. Specifically, a drive mechanism (not shown) having a motor or the like for moving the moving part 4 may be provided on the crane 1, and the drive mechanism may be driven in response to an operation performed on the operation part 9 by the crane operator U'.
[0027] The trolley 10 is provided on the pair of girders 3 so as to straddle the pair of girders 3. The trolley 10 may have a driver's cab in which a crane operator U' sits, or may not have a driver's cab.
[0028] The trolley 10 moves in the first direction X along the girder 3 in response to an operation performed on the operation unit 9 by, for example, a crane operator U'. Specifically, a drive mechanism (not shown) having a motor or the like for moving the trolley 10 may be provided on the crane 1, and the drive mechanism may be driven in response to an operation performed on the operation unit 9 by the crane operator U'.
[0029] The hoisting device 5 is provided on a trolley 10. A rope 6 having a sheave 7 connected to its end is attached to the hoisting device 5. The hoisting device 5 has a motor and the like (not shown) for winding up and lowering the rope 6. Note that "hoisting" refers to the operation of driving the motor so that the sheave 7 moves upward. "Lowering" refers to the operation of driving the motor so that the sheave 7 moves downward.
[0030] The hoisting device 5 winds up and down the rope 6 in response to an operation performed by, for example, a crane operator U' on the operation unit 9. Specifically, a motor or the like provided in the hoisting device 5 may be driven in response to an operation performed by the crane operator U' on the operation unit 9.
[0031] The hanging mechanism 20 is a mechanism for hanging (hooking) the suspended load L. The hanging mechanism 20 is suspended from a hook 8 provided on the sheave 7. Specifically, the center portion of a rope 21 held by the hanging mechanism 20 is suspended from the hook 8. A pair of suspenders 22 are connected to both ends of the rope 21.
[0032] Each of the hoisting tools 22 has a base 22b and a pair of claws 22a. Each of the claws 22a is a portion that contacts the lower surface of the load L to support the load L when the load L is hung on the hanging mechanism 20.
[0033] In this embodiment, the claw portion 22a of one of the pair of suspenders 22 is hooked to one long side of the load L, and the claw portion 22a of the other of the pair of suspenders 22 is hooked to the other long side of the load L. That is, when the load L is hung on the hanging mechanism 20, the pair of suspenders 22 face each other in the short direction of the load L. In addition, in each suspender 22, the pair of claw portions 22a are arranged at intervals in the horizontal direction (longitudinal direction of the load L). Therefore, when the load L is hung on the hanging mechanism 20, each of the four claw portions 22a of the pair of suspenders 22 contacts the lower surface of the load L to support the load L. However, as long as the load L can be hung, the configuration of the hanging mechanism 20 (suspender 22) can be appropriately changed.
[0034] In this embodiment, removal of the hoisting tool 22 from the load L is performed by a ground worker (ground slinger) U. The ground worker U is a worker who performs work on the ground. The ground worker U is, for example, a different person from the crane operator U'.
[0035] The operation unit 9 accepts operations by the crane operator U'. The operation unit 9 may be a physical mechanism (a lever, a button, etc.) for operating the crane 1, or may be an information processing terminal for operating the crane 1, etc.
[0036] After the ground worker U hooks the load L to the hooking mechanism 20 (hoisting tool 22), the crane operator U' operates the operation unit 9 to wind up the rope 6, so that the load L is lifted by the hooking mechanism 20 (hoisting tool 22) and rises. In this state, the crane operator U' operates the operation unit 9 to move the trolley 10 and / or the moving part 4 (girder 3). Then, the load L suspended from the hooking mechanism 20 moves in the first direction X and / or the second direction Y. In other words, the load L moves in the first direction X as the trolley 10 moves in the first direction X, and moves in the second direction Y as the moving part 4 (girder 3) moves in the second direction Y.
[0037] Hereinafter, the movement of the load L (crane 1) in the first direction X accompanying the movement of the trolley 10 may be referred to as "traverse." Also, the movement of the load L (crane 1) in the second direction Y accompanying the movement of the moving part 4 (girder 3) may be referred to as "travel." That is, in this embodiment, the first direction X coincides with the traverse direction of the load L (crane 1), and the second direction Y coincides with the traveling direction of the load L (crane 1).
[0038] In the crane 1 as described above, there is a possibility that a load collapse may occur due to some factor. Load collapse is a phenomenon in which the suspended load L falls from the hooking mechanism 20 (the hoisting tool 22). When a load collapse occurs, there is a possibility that an accident may occur, such as the dropped load L colliding with a ground worker U. Such an accident may occur, for example, when the ground worker U is close to the suspended load L. Hereinafter, a case in which an accident is predicted in which the dropped load L will collide with the ground worker U when a load collapse occurs, such as when the ground worker U is close to the suspended load L, may be referred to as a "first alarm case." The safety support system S is configured to detect the first alarm case, and to issue an alarm when such a detection is made.
[0039] Moreover, the collapse of the load occurs when the hooking mechanism 20 (the hoisting device 22) comes off the suspended load L. Hereinafter, a case in which there is a sign that the hooking mechanism 20 (the hoisting device 22) will come off the suspended load L may be referred to as a "second alarm case." The safety support system S is configured to detect the second alarm case and issue an alarm when such a detection is made.
[0040] The various components of the safety support system S that issues these warnings and the processes performed by the safety support system S will be described in detail below.
[0041] <Safety support system configuration> FIG. 2 is a block diagram showing an example of the configuration of the safety support system S according to this embodiment. The safety support system S according to this embodiment includes a first terminal device 100, a second terminal device 200, an imaging unit 301, a three-dimensional sensor 302, and an alarm unit 303. As described in a modified example below, the safety support system S does not need to include the three-dimensional sensor 302. The first terminal device 100 and the second terminal device 200 are communicatively connected via a network 400. The network 400 may be a network using wireless communication or a network using wired communication. The network 400 may be configured using a local area network (LAN) such as Wi-Fi (registered trademark). The network 400 may be configured by combining a plurality of networks.
[0042] The imaging unit 301 captures an image IM (see, for example, FIG. 9) including the ground worker U, the hoist 22, and the suspended load L. The imaging unit 301 may include, for example, at least one camera for capturing the image IM. The imaging unit 301 outputs the captured image IM to an image acquisition unit 121 (described later) included in the first terminal device 100.
[0043] The three-dimensional sensor 302 measures three-dimensional information of the load L. The three-dimensional information of the load L includes information on the three-dimensional position of the load L and information on the three-dimensional shape of the load L. Therefore, the three-dimensional information of the load L includes height distribution information indicating the height distribution of the load L (distribution of positions in the gravity direction Z). The three-dimensional sensor 302 may have, for example, a LiDAR that measures the distance to an object using reflected light obtained by irradiating light onto the object. The three-dimensional sensor 302 outputs the measured three-dimensional information to a three-dimensional information acquisition unit 122 (described later) of the first terminal device 100.
[0044] The alarm unit 303 issues an alarm in response to control by a control unit 140 (described later) possessed by the first terminal device 100. The alarm unit 303 issues an alarm in a form that can be recognized by the ground worker U and / or the crane operator U'. By issuing an alarm to the ground worker U using the alarm unit 303, it is possible to notify the ground worker U of a danger and urge the ground worker U to evacuate or to correct the attachment position of the hoisting device 22, etc. By issuing an alarm to the crane operator U' using the alarm unit 303, it is possible to notify the crane operator U' of a danger and urge him / her to stop work using the crane 1, etc. This makes it possible to prevent accidents caused by cargo collapse.
[0045] The alarm unit 303 may include a light that issues an alarm by light (e.g., an LED light), a speaker that issues an alarm by sound, a wearable terminal worn by the ground worker U or the crane operator U', etc. The alarm unit 303 may issue an alarm by combining a plurality of types of these alarm means.
[0046] When a light is used as the alarm unit 303, the alarm unit 303 may project an image P as an alarm by irradiating a predetermined location (for example, the vicinity of the suspended load L) with light (for example, see FIG. 3). It is desirable that the position where the image P is projected is a position visible from both the ground worker U and the crane operator U' (in the driver's cab).
[0047] Fig. 3 is a diagram showing an example of an alarm issued by the alarm unit 303. In the example of Fig. 3, the alarm unit 303 is a plurality of lights, which projects an image P including a plurality of circles arranged in a circle from above onto the periphery of the suspended load L as an alarm. A part of the image P is projected onto the upper surface of the suspended load L. Even when the hoisting tool 22 is lifted by the crane 1, it is common for the hoisting tool 22 to be located below the eyes of the ground worker U. Therefore, by projecting the image P as an alarm onto the periphery of the suspended load L (the upper surface of the suspended load L) in this way, it becomes easier for the ground worker U to visually recognize the image P as an alarm.
[0048] If a goggle-type wearable terminal worn by a ground worker U or a crane operator U' is used as the alarm unit 303, the alarm unit 303 may issue an alarm by displaying visual information (text, figures, etc.) on the wearable terminal.
[0049] The imaging unit 301, the three-dimensional sensor 302, and the alarm unit 303 are provided (attached) to the crane 1. The imaging unit 301, the three-dimensional sensor 302, and the alarm unit 303 may be provided, for example, to the trolley 10. As shown in FIG. 1, the ground worker U, the hoist 22, and the load L are located immediately below the trolley 10. Therefore, by providing the imaging unit 301 to the trolley 10, an image IM including these objects can be easily captured. Similarly, by providing the three-dimensional sensor 302 to the trolley 10, the three-dimensional information of the load L can be easily measured. In addition, when the alarm unit 303 is a light, by providing the alarm unit 303 to the trolley 10, the image P can be easily projected onto the periphery of the load L (the upper surface of the load L).
[0050] 4 is a diagram showing an example of a method for attaching the imaging unit 301, the three-dimensional sensor 302, and the alarm unit 303 to the trolley 10. However, the method for attaching the imaging unit 301, the three-dimensional sensor 302, and the alarm unit 303 to the crane 1 (trolley 10) can be changed as appropriate.
[0051] As shown in Fig. 4, in the safety support system S according to this embodiment, a frame-shaped mounting jig J is attached to the trolley 10. When the mounting jig J is attached to the trolley 10, the mounting jig J extends downward from the trolley 10, and the lower end of the mounting jig J is located below the lower end of the girder 3. An imaging unit 301, a three-dimensional sensor 302, and an alarm unit 303 are attached to a portion of the mounting jig J located below the girder 3 so as to face downward. Note that the imaging unit 301 in the illustrated example includes two cameras.
[0052] Positioning the imaging unit 301 below the girder 3 can prevent the girder 3 from entering the imaging range of the imaging unit 301. Similarly, positioning the three-dimensional sensor 302 below the girder 3 can prevent the girder 3 from entering the detection range of the three-dimensional sensor 302. In addition, when a light that irradiates light downward to project the image P is used as the alarm unit 303, positioning the alarm unit 303 below the girder 3 can prevent the light from the alarm unit 303 from being blocked by the girder 3 and causing the image P to be missing.
[0053] The first terminal device 100 (see FIG. 2) is configured using an information processing terminal such as a personal computer. The first terminal device 100 is provided, for example, on a trolley 10. The first terminal device 100 is also called an edge PC. As shown in FIG. 2, the first terminal device 100 includes a communication unit 110, an acquisition unit 120, a storage unit 130, and a control unit 140. The acquisition unit 120 includes, for example, an image acquisition unit 121 and a three-dimensional information acquisition unit 122.
[0054] The communication unit 110 has hardware for connecting to the network 400 using a cellular network, a Wi-Fi network, or the like. For example, the communication unit 110 has an antenna, a transmitting / receiving device, and the like. The communication unit 110 communicates with the second terminal device 200 (communication unit 210) via the network 400. However, the communication unit 110 may be connected to the second terminal device 200 (communication unit 210) by wire and communicate with it.
[0055] The image acquisition unit 121 acquires an image IM including the ground worker U, the hoist 22, and the suspended load L. The image acquisition unit 121 acquires, for example, the image IM output from the imaging unit 301. The image acquisition unit 121 also outputs the acquired image IM to the second terminal device 200 (communication unit 210) via the communication unit 110 and the network 400.
[0056] The three-dimensional information acquisition unit 122 acquires three-dimensional information of the suspended load L. The three-dimensional information acquisition unit 122 acquires, for example, three-dimensional information of the suspended load L output from the three-dimensional sensor 302. In addition, the three-dimensional information acquisition unit 122 outputs the acquired three-dimensional information of the suspended load L to the second terminal device 200 (communication unit 210) via the communication unit 110 and the network 400.
[0057] The storage unit 130 is realized by, for example, a random access memory (RAM), a read only memory (ROM), a flash memory, an SD card, a register, a hard disk drive, etc. The storage unit 130 stores data used by the control unit 140. The storage unit 130 stores data required when the control unit 140 performs processing.
[0058] Some or all of the functions of the control unit 140 are realized by, for example, a processor such as a CPU (Central Processing Unit) executing a program (software) stored in its own storage unit. Some or all of the functions of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by cooperation between software and hardware. The program may be stored in a storage device such as an HDD (Hard Disk Drive) or a flash memory in advance, or may be stored in a removable storage medium such as a DVD or a CD-ROM, and installed by mounting the storage medium in a drive device.
[0059] The control unit 140 controls the alarm unit 303 to issue an alarm. The control unit 140 controls the alarm unit 303 based on a control signal output by the second terminal device 200 (the generation unit 253) and transmitted via the communication unit 110 and the network 400, for example.
[0060] The control unit 140 may cause the alarm unit 303 to issue visually different alarms depending on the type of alarm. Specifically, different control signals are output from the generation unit 253 depending on the determination result by the determination unit 252 (details will be described later), and the control unit 140 may cause the alarm unit 303 to issue visually different alarms depending on the different control signals. For example, the alarm issued by the alarm unit 303 may be visually different when the first alarm case is detected and when the second alarm case is detected. For example, when the alarm unit 303 is a light that projects an image P, the shape, size, color, blinking speed, etc. of the image P may be different between the first alarm case and the second alarm case. For example, when the alarm unit 303 is a wearable terminal, the information displayed on the wearable terminal may be different between the first alarm case and the second alarm case. This makes it possible to effectively convey the contents of the alarm to the alarm target (ground worker U or crane operator U') even in a site with a lot of noise.
[0061] The second terminal device 200 is configured using an information processing terminal such as a personal computer. The second terminal device 200 is provided, for example, outside the crane 1. The second terminal device 200 includes a communication unit 210, an input unit 220, an output unit 230, a storage unit 240, and a processing unit 250.
[0062] The communication unit 210 has hardware for connecting to the network 400 using a cellular network, a Wi-Fi network, or the like. For example, the communication unit 210 has an antenna, a transmitting / receiving device, and the like. The communication unit 210 communicates with the first terminal device 100 (communication unit 110) via the network 400. However, the communication unit 210 may be connected to the first terminal device 100 (communication unit 110) by wire and communicate with it.
[0063] The image IM acquired by the communication unit 210 from the first terminal device 100 is output to the processing unit 250 and stored in the memory unit 240 as a part of image data 241. The three-dimensional information of the suspended load L acquired by the communication unit 210 from the first terminal device 100 is output to the processing unit 250 and stored in the memory unit 240 as a part of three-dimensional data 242.
[0064] The input unit 220 is configured using existing input devices such as a keyboard, a pointing device (mouse, tablet, etc.), a button, a touch panel, etc. The input unit 220 is operated by a user when inputting the user's instructions to the second terminal device 200. The input unit 220 may be an interface for connecting the input device to the second terminal device 200. The input unit 220 may be configured using a microphone and a voice recognition device. The input unit 220 may be configured in any way as long as it is configured to input the user's instructions to the second terminal device 200.
[0065] The output unit 230 outputs information in a form that can be recognized by the user. The output unit 230 may be, for example, an image display device such as a liquid crystal display or an organic EL (Electro Luminescence) display. The output unit 230 may be an interface for connecting the image display device to the second terminal device 200. The output unit 230 may be a device that outputs sound such as a speaker. The output unit 230 may be an interface for connecting a sound output device such as a speaker or a headphone to the first terminal device 100. The output unit 230 may be configured as a touch panel integrated with the input unit 220.
[0066] The storage unit 240 is realized by, for example, a random access memory (RAM), a read only memory (ROM), a flash memory, an SD card, a register, a hard disk drive, etc. The storage unit 240 stores data used by the processing unit 250. The storage unit 240 stores data required when the processing unit 250 performs processing.
[0067] The storage unit 240 stores, for example, image data 241, three-dimensional data 242, processing information 243, and a trained model 244. The image data 241 includes an image IM acquired by the image acquisition unit 121. The three-dimensional data 242 includes three-dimensional information of the suspended load L acquired by the three-dimensional information acquisition unit 122, and deflection information (described later) calculated by the calculation unit 251. The processing information 243 includes information output by the processing performed by the processing unit 250.
[0068] The trained model 244 is data indicating parameters, connection structure, function characteristics, etc. of the trained model used in processing by the processing unit 250 (determination unit 252). The trained model 244 is trained to determine (output) at least whether or not an alarm is required by the alarm unit 303 when an image IM and three-dimensional information of the suspended load L are input. More specifically, the trained model 244 is trained to determine (output) whether or not a situation shown in the image IM and the three-dimensional information corresponds to a first alarm case, a second alarm case, both alarm cases, or none of the alarm cases when an image IM and three-dimensional information of the suspended load L are input. Details of the trained model 244 will be described later.
[0069] The processing unit 250 includes, for example, a calculation unit 251, a determination unit 252, a generation unit 253, and a learning processing unit 254. Some or all of the functions of the processing unit 250 are realized by, for example, a processor such as a CPU (Central Processing Unit) executing a program (software) stored in its own storage unit. In addition, some or all of the functions of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by cooperation between software and hardware. The program may be stored in advance in a storage device such as an HDD (Hard Disk Drive) or a flash memory, or may be stored in a removable storage medium such as a DVD or a CD-ROM, and installed by mounting the storage medium in a drive device.
[0070] The calculation unit 251 calculates deflection information of the load L based on three-dimensional information (height distribution information) of the load L acquired from the first terminal device 100 via the communication unit 210. The calculation unit 251 outputs the calculated deflection information of the load L to the determination unit 252. The calculation unit 251 associates the calculated deflection information of the load L with the three-dimensional information (height distribution information) of the load L as part of the three-dimensional data 242 and stores it in the storage unit 240. Here, the "deflection information of the load L" is information indicating the deflection of the load L. For example, the deflection information of the load L may include information indicating the amount of deflection of the load L and the shape of the deflection.
[0071] FIG. 5 is a diagram showing an example of the shape of the deflection of the load L. In the diagram, both ends of the load L in the longitudinal direction are indicated by the symbols L1 and L2. As shown in FIG. 5, the load L lifted by the hoist 22 may deflect due to its own weight. Here, the calculation unit 251 may calculate the height difference Δ1 shown in the diagram as the amount of deflection (deflection information). Here, the height difference Δ1 is the difference in height between the highest point on the surface of the load L and the lowest point on the surface of the load L. The calculation unit 251 may also calculate the height difference Δ2 as information related to the deflection. The height difference Δ2 is the difference in height between the first end L1 of the load L and the second end L2 of the load L.
[0072] FIG. 6 is a diagram showing another example of the shape of the deflection of the load L. In the examples described so far, the crane 1 has a pair of the hoisting tools 22, but the crane 1 may have two or more pairs of the hoisting tools 22. FIGS. 6(a) to (c) show examples of the shape of the deflection (deflection mode) that may occur in the load L when the crane 1 has two pairs of the hoisting tools 22. FIGS. 6(d) to (e) show examples of the shape of the deflection (deflection mode) that may occur in the load L when the crane 1 has three pairs of the hoisting tools 22. The calculation unit 251 may be configured to calculate (determine) the deflection mode of the load L from the acquired three-dimensional information of the load L, and output it as deflection information of the load L. In addition, a predetermined parameter (a parameter corresponding to the above-mentioned amounts Δ1 and Δ2, for example, the amount of convexity of each convex portion of the load L) may be set for each deflection mode, and the calculation unit 251 may be configured to calculate the predetermined parameter and output it as deflection information.
[0073] The determination unit 252 (see FIG. 2) inputs the image IM and the three-dimensional information to the trained model 244, and thereby determines whether or not an alarm by the alarm unit 303 is necessary in at least the situation shown in the image IM and the three-dimensional information. That is, the determination unit 252 determines whether the situation shown in the image IM and the three-dimensional information corresponds to a first alarm case, a second alarm case, both alarm cases, or none of the alarm cases. The determination unit 252 outputs the determination result (i.e., alarm necessity information, which will be described later) to the generation unit 253. Details of the determination unit 252 will be described later.
[0074] The generating unit 253 generates a control signal for the control unit 140 to control the alarm unit 303 based on the determination result by the determining unit 252 (i.e., alarm necessity information described later). The content of the generated control signal may differ depending on the determination result (alarm necessity information). The generating unit 253 outputs the control signal to the first terminal device 100 via the communication unit 210 and the network 400. As a result, based on the determination result by the determining unit 252, the control unit 140 causes the alarm unit 303 to issue an alarm according to the determination result.
[0075] The learning processing unit 254 performs re-learning of the trained model 244. The re-learning by the learning processing unit 254 will be described in detail later.
[0076] <Judgement section and trained model> Fig. 7 is a block diagram showing an example of the determination unit 252 according to this embodiment. Fig. 8 is a block diagram showing an example of the trained model 244 according to this embodiment. Fig. 9 is a diagram showing an example of an image IM acquired by the image acquisition unit 121 according to this embodiment and an output of the determination unit 252 based on the image IM.
[0077] As shown in Fig. 7, the judgment unit 252 according to this embodiment includes a first judgment unit 252a, a second judgment unit 252b, a third judgment unit 252c, a fourth judgment unit 252d, a fifth judgment unit 252e, and a sixth judgment unit 252f. As shown in Fig. 8, the trained model 244 according to this embodiment includes a first trained model 244a, a second trained model 244b, a third trained model 244c, a fourth trained model 244d, a fifth trained model 244e, and a sixth trained model 244f. The first judgment unit 252a to the fourth judgment unit 252d and the first trained model 244a to the fourth trained model 244d correspond to a configuration for detecting a first alarm case. The fifth judgment unit 252e to the sixth judgment unit 252f and the fifth trained model 244e to the sixth trained model 244f correspond to a configuration for detecting a second alarm case.
[0078] First, the configuration for detecting the first alarm case will be described.
[0079] The first trained model 244a (see FIG. 8) is trained to output at least one of the position and state of the ground worker U, the position of the hoist 22, and the position of the suspended load L when an image IM is input. That is, the first trained model 244a is a model for image recognition of at least one of the position and state of the ground worker U, the position of the hoist 22, and the position of the suspended load L. Note that the "state of the ground worker U" may include information such as the orientation of the ground worker U and the status of the work of the ground worker U (for example, whether or not the ground worker U is performing work to attach the hoist 22 to the suspended load L). Hereinafter, the "ground worker U, the hoist 22, and the suspended load L" may be collectively referred to as the "recognition target," and the "at least one of the position and state of the ground worker U, the position of the hoist 22, and the position of the suspended load L" may be collectively referred to as the "recognition target information."
[0080] The first determination unit 252a (see FIG. 7) outputs the recognition target information by inputting the image IM acquired by the image acquisition unit 121 (see also FIG. 2) to the first trained model 244a. The first determination unit 252a outputs the output recognition target information to the second determination unit 252b and the fourth determination unit 252d. The first determination unit 252a associates the output recognition target information with the image data 241 (and the three-dimensional data 242) as part of the processing information 243 and stores it in the storage unit 240. In the example shown in FIG. 9, the first determination unit 252a outputs the position B1 of the ground worker U, the position B2 of the hoist 22, and the position B3 of the load L in the form of bounding boxes as the recognition target information, and also outputs the orientation D of the ground worker U.
[0081] The second trained model 244b (see FIG. 8) is trained to output the work content of the crane 1 when the recognition target information is input. Here, examples of the work content of the crane 1 include "hoisting", "lowering", "traverse", "travel", and "traverse and travel". That is, the second trained model 244b is trained to output (determine) whether the work content of the crane 1 is "hoisting", "lowering", "traverse", "travel", or "travel and traverse" when the recognition target information is input. Note that "travel and traverse" refers to a state in which the crane 1 (load L) moves in a direction inclined with respect to both the first direction X and the second direction Y as a result of the crane 1 (load L) traveling and the crane 1 (load L) traveling at the same time.
[0082] The second determination unit 252b (see FIG. 7) outputs the work content of the crane 1 by inputting the output of the first determination unit 252a (i.e., the recognition target information) to the second trained model 244b. The second determination unit 252b outputs the output work content to the third determination unit 252c. The second determination unit 252b associates the output work content with the image data 241 (and the three-dimensional data 242) as part of the processing information 243 and stores it in the memory unit 240.
[0083] The third trained model 244c (see FIG. 8) is trained to output a danger area A in the image IM when the work content of the crane 1 and the image IM are input. Here, the danger area A is an area where an accident involving a drop of the suspended load L may occur if a ground worker U enters. In other words, if the ground worker U is present in the danger area A, it may correspond to the first alarm case described above. In the example shown in FIG. 9, the danger area A is output as an area including the area B3 of the suspended load L and its surrounding area.
[0084] As a result of careful consideration by the inventors of the present application, it was determined that this danger area A should change depending on the type of work being performed by the crane 1. Therefore, the third trained model 244c according to this embodiment is trained to output a different danger area A depending on the type of work being performed by the crane 1. For example, the danger area A that is output differs depending on whether the work state of the crane 1 is "hoisting", "lowering", "traverse", "travel", or "travel and traverse".
[0085] Fig. 10 is a diagram showing an example of the change in the danger area A according to the working state of the crane 1. Fig. 10(a) shows the case where the working state of the crane 1 is "hoisting" or "lowering". Fig. 10(b), Fig. 10(c), and Fig. 10(d) show the cases where the working state of the crane 1 is "traverse", "travel", and "travel and traverse", respectively. The white arrows shown in Figs. 10(b) to 10(d) indicate the direction of travel of the crane 1 (suspended load L).
[0086] 10(a), when the working state of the crane 1 is "hoisting" or "lowering", an area A0 including the suspended load L (area B3) and its surrounding area is output as the danger area A. Hereinafter, this area A0 may be referred to as the "basic area A0".
[0087] On the other hand, as a result of intensive study by the inventors of the present application, it was found that when the working state of the crane 1 is any of "traverse", "travel", and "travel and traverse", not only the basic area A0 but also the area A1 located forward of the basic area A0 in the moving direction of the crane 1 (hanging load L) should be set as the danger area A (see Fig. 10(b) to Fig. 10(d)). In other words, when the crane 1 (hanging load L) is moving in the horizontal direction, it was found that not only the basic area A0 but also the area A1 should be included in the danger area A. Hereinafter, this area A1 to be added to the basic area A0 may be referred to as the "additional area A1". The third trained model 244c according to this embodiment is trained to output the danger area A combining the basic area A0 and the additional area A1 when the crane 1 is any of "traverse", "travel", and "travel and traverse" (i.e., when the crane 1 (hanging load L) is moving in the horizontal direction).
[0088] It is considered that the size of the basic area A0 and the additional area A1 should change depending on the size, shape, material, and weight of the load L, and the surface condition (slipperiness, etc.) of the load L. Therefore, the third trained model 244c according to this embodiment is configured to output the danger area A based not only on the working state of the crane 1 but also on the image IM. This makes it possible to set the danger area A appropriately for various loads L.
[0089] The third judgment unit 252c (see FIG. 7) inputs the image IM acquired by the image acquisition unit 121 and the output of the second judgment unit 252b (i.e., the work content of the crane 1) to the third trained model 244c, thereby outputting a danger area A in the image IM. The third judgment unit 252c outputs the output danger area A to the fourth judgment unit 252d. The third judgment unit 252c links the output danger area A to the image data 241 (and the three-dimensional data 242) and stores it in the memory unit 240 as part of the processing information 243.
[0090] The fourth trained model 244d (see FIG. 8) is trained to determine whether or not an alarm is required by the alarm unit 303 when at least one of the position and state of the ground worker U and the danger area A are input. In other words, the fourth trained model 244d is trained to determine whether or not the first alarm case occurs when at least one of the position and state of the ground worker U and the danger area A are input.
[0091] For example, if a ground worker U (area B1) is present in the danger area A (see, for example, FIG. 9), there is a high possibility that an accident will occur in which the load L comes into contact with the ground worker U when the load L falls. Therefore, if the ground worker U (area B1) is present in the danger area A, there is a high possibility that an alarm by the alarm unit 303 is necessary. On the other hand, even if the ground worker U (area B1) is located in the danger area A, there may be cases in which an alarm by the alarm unit 303 is unnecessary depending on the state of the ground worker U. One example of such a case is when the ground worker U is considered to be paying attention to the load L, such as when the ground worker U is facing the direction of the load L. Therefore, by including the state of the ground worker U (orientation D, etc.) in the input of the fourth trained model 244d, it is possible to more accurately determine whether an alarm is necessary.
[0092] The fourth determination unit 252d inputs the output of the first determination unit 252a (i.e., at least one of the position and state of the ground worker U) and the output of the third determination unit 252c (i.e., the danger area A) to the fourth trained model 244d, thereby determining whether or not an alarm is required by the alarm unit 303. That is, the fourth determination unit 252d determines whether or not an alarm is required based on the first alarm case.
[0093] Next, a configuration for detecting the second alarm case will be described.
[0094] The fifth trained model 244e (see FIG. 8) is trained to output the position of the suspending device 22 when an image IM is input. In other words, the fifth trained model 244e is a model for image recognition of the position of the suspending device 22.
[0095] The fifth determination unit 252e (see FIG. 7) outputs the position of the sling 22 by inputting the image IM acquired by the image acquisition unit 121 to the fifth trained model 244e. The fifth determination unit 252e outputs the output position of the sling 22 to the sixth determination unit 252f. The sixth determination unit 252f may store the output position of the sling 22 in the storage unit 240 as part of the processing information 243, in association with the image data 241 (and the three-dimensional data 242). Note that the fifth determination unit 252e may output the position of the sling 22 by inputting the image IM to the first trained model 244a described above. In this case, the trained model 244 does not need to include the fifth trained model 244e.
[0096] The sixth trained model 244f (see FIG. 8) is trained to determine whether or not an alarm is required by the alarm unit 303 when the position of the hoisting device 22, the image IM, and information related to the three-dimensional information of the load L are input. In other words, the sixth trained model 244f is trained to determine whether or not the second alarm case occurs when the position of the hoisting device 22, the image IM, and information related to the three-dimensional information of the load L are input.
[0097] As a result of thorough investigations, the inventors of the present application have found that there are two cases in which the suspending tool 22 is likely to come off the load L (in other words, there are signs that the suspending tool 22 will come off the load L). The first case is when the position at which the suspending tool 22 is attached to the load L is inappropriate. The second case is when the suspending tool 22 slips due to bending of the load L. Based on this consideration, the sixth trained model 244f according to this embodiment employs, as inputs, the position of the suspending tool 22, the image IM, and information related to the three-dimensional information of the load L. These will be described in detail below.
[0098] First, the first case will be described. FIG. 11 is a diagram showing an example of a position where the sling 22 is attached to the load L. Here, an optimal position (optimum position ID) exists for attaching the sling 22 to the load L. The optimal position ID is an optimal position for stably supporting the load L by the sling 22. It is considered that the optimal position ID is determined by the position of the center of gravity of the sling 22. As shown in FIG. 11, when the position where the sling 22 is attached to the load L (hereinafter referred to as the "attachment position of the sling 22") is deviated from the optimal position ID, there is a high possibility that the load L will fall from the sling 22 (the sling 22 will come off from the load L). Therefore, the sixth trained model 244f according to this embodiment is configured to determine whether or not an alarm is required (i.e., whether or not the above-mentioned first case applies) based on the position of the sling 22.
[0099] The processing unit 250 may include an estimation unit (not shown) that estimates the center of gravity of the load L from the image IM. The estimation unit may estimate the position of the center of gravity of the load L by inputting the image IM to a trained model that outputs (estimates) the position of the center of gravity of the load L when the image IM including the load L is input. Alternatively, the estimation unit may estimate the position of the center of gravity of the load L on a rule basis. The input of the sixth trained model 244f may further include the position of the center of gravity of the load L estimated by the estimation unit. That is, the sixth trained model 244f may be trained to determine whether or not an alarm is required by the alarm unit 303 when the position of the hoisting tool 22, the image IM, information related to the three-dimensional information of the load L, and the position of the center of gravity of the load L are input. The estimation unit may store the estimated center of gravity of the suspended load L in the storage unit 240 as part of the processing information 243 in association with the image data 241 (and the three-dimensional data 242).
[0100] Next, the second case will be described. The likelihood of slipping of the hoist 22 is considered to vary depending on the magnitude and shape of the deflection of the load L. Here, the three-dimensional information (height distribution information) of the load L acquired by the three-dimensional information acquisition unit 122 is considered to include information on the magnitude and shape of the deflection of the load L. Therefore, the sixth trained model 244f according to this embodiment is configured to determine whether or not an alarm is required (i.e., whether or not the above-mentioned second case applies) based on information related to the three-dimensional information (height distribution information) of the load L.
[0101] Here, the "information related to the three-dimensional information (height distribution information) of the suspended load L" may be, for example, the three-dimensional information (height distribution information) of the suspended load L itself, or may be information calculated based on the three-dimensional information (height distribution information) of the suspended load L. For example, the "information related to the three-dimensional information (height distribution information) of the suspended load L" may be deflection information calculated by the calculation unit 251. By using the deflection information instead of the three-dimensional information of the suspended load L itself as input to the sixth trained model 244f, it is expected that the judgment accuracy by the sixth trained model 244f will be improved.
[0102] As a method for detecting the stable state of the load L, a method for detecting the inclination of the load L is known (for example, the method disclosed in JP 2021-054584 A). However, since the load L, such as a steel material, bends, it has been difficult to determine the stable state of the load L based on the inclination alone. In this embodiment, three-dimensional information (height distribution information) of the load L including information related to the deflection and the deflection information calculated by the calculation unit 251 are used as inputs to the trained model. Therefore, compared to conventional methods that use only the inclination, the detection accuracy of the stable state of the load L can be improved.
[0103] In addition, in the first case, even if the attachment position of the suspending tool 22 is the same, the possibility of the load L falling from the suspending tool 22 is considered to change depending on the size, shape, material, weight, and surface condition (slipperiness, etc.) of the load L. Similarly, in the second case, even if the deflection of the load L is the same, the possibility of the load L falling from the suspending tool 22 is considered to change depending on the size, shape, material, weight, and surface condition (slipperiness, etc.) of the load L. Therefore, the sixth trained model 244f according to this embodiment is configured to determine whether or not an alarm is required based not only on the position of the suspending tool 22 and information related to the three-dimensional information of the load L, but also on the image IM. This makes it possible to appropriately determine whether or not an alarm is required for various loads L.
[0104] The sixth determination unit 252f (see FIG. 7) inputs the image IM, the output of the fifth determination unit 252e (i.e., the position of the hoist 22), and information related to the three-dimensional information of the load L to the sixth trained model 244f, thereby determining whether or not an alarm is required by the alarm unit 303. When the "information related to the three-dimensional information of the load L", which is one of the inputs of the sixth trained model 244f, is the three-dimensional information of the load L, the sixth determination unit 252f may use the three-dimensional information of the load L acquired by the three-dimensional information acquisition unit 122 as the input of the sixth trained model 244f. When the "information related to the three-dimensional information of the load L", which is one of the inputs of the sixth trained model 244f, is the deflection information of the load L, the sixth determination unit 252f may use the deflection information of the load L calculated by the calculation unit 251 as the input of the sixth trained model 244f.
[0105] The determination unit 252 outputs the determination result by the fourth determination unit 252d and the determination result by the sixth determination unit 252f to the generation unit 253. Specifically, the determination unit 252 outputs warning necessity information to the generation unit 253 based on the determination results by the fourth determination unit 252d and the sixth determination unit 252f. For example, the determination unit 252 may integrate the determination results by the fourth determination unit 252d and the sixth determination unit 252f to generate warning necessity information and output it to the generation unit 253. The determination unit 252 stores the generated warning necessity information in the storage unit 240 as part of the processing information 243, in association with the image data 241 and the three-dimensional data 242.
[0106] Here, the "alarm necessity information" is information related to an alarm to be issued by the alarm unit 303. Specifically, the alarm necessity information may be information indicating whether an alarm by the alarm unit 303 is necessary. More specifically, the alarm necessity information may be information indicating any of "an alarm related to a first alarm case is necessary", "an alarm related to a second alarm case is necessary", "an alarm related to both the first alarm case and the second alarm case is necessary", and "an alarm related to neither the first alarm case nor the second alarm case is necessary". In other words, the alarm necessity information may include not only information indicating whether an alarm is necessary, but also information indicating the type of alarm (what type of alarm should be issued).
[0107] The trained models 244 (first trained model 244a to sixth trained model 244f) as described above may be created by learning (machine learning) the relationship between the input information and the output information using a set of multiple pieces of input information and output information linked one-to-one to the input information as teacher data. The input information includes, for example, an image IM including the recognition target and three-dimensional information of the suspended load L. The output information includes recognition target information, the work content of the crane 1, the danger area A, and information on whether an alarm is required.
[0108] That is, the trained model 244 (first trained model 244a to sixth trained model 244f) may be created by performing the steps of acquiring multiple pieces of input information as described above, acquiring output information linked one-to-one to the acquired multiple pieces of input information, and creating the trained model 244 (first trained model 244a to sixth trained model 244f) using the acquired pairs of input information and output information as teacher data.
[0109] It is preferable that the multiple pieces of input information used for learning include input information corresponding to different suspended loads L. In this case, a trained model 244 (first trained model 244a to sixth trained model 244f) capable of providing highly accurate output for various suspended loads L can be created.
[0110] Furthermore, the trained models 244 (first trained model 244a to sixth trained model 244f) may be created by the safety support system S (for example, the learning processing unit 254). In this case, the input information that is part of the teacher data may be acquired by the acquisition unit 120. The acquired input information may be presented to a user (for example, a skilled worker) by the output unit 230 or the like. The user may then confirm the acquired input information via the output unit 230 or the like, and determine output information that is considered appropriate based on the confirmed input information. The user may then have the safety support system S acquire the output information that is determined to be appropriate by operating the input unit 220 or the like.
[0111] Alternatively, the trained model 244 may be created by a learning device (hereinafter, referred to as a "separate learning device") provided separately from the safety support system S, and then the created trained model 244 may be stored in the storage unit 240. In this case, the input information that is part of the teacher data may be acquired by an acquisition means provided in the separate learning device. Also, the output information that is part of the teacher data may be input by a user (e.g., an experienced worker) to the separate learning device using a method similar to that described above.
[0112] In addition, the specific method of learning (machine learning) in the safety support system S or a separate learning device is not particularly limited, but for example, a neural network (deep learning) such as CNN or RNN may be used. In addition, learning (machine learning) may be performed using the method disclosed in JP 2022-102930 A.
[0113] <Relearning by the learning processing unit> The judgment of whether or not an alarm is required by the judgment unit 252 may not be perfect. That is, the judgment unit 252 may judge that an alarm is unnecessary in a scene where an alarm is actually necessary (non-alarm), or judge that an alarm is necessary in a scene where an alarm is not actually necessary (excessive alarm). Here, as described above, the storage unit 240 according to this embodiment stores (accumulates) the image data 241 (image IM), the three-dimensional data 242, and the processing information 243 output by the processing of the judgment unit 252. The processing information 243 includes alarm necessity information linked to the image data 241 and the three-dimensional data 242. Therefore, a skilled worker can detect a non-alarm or excessive alarm by checking the image data 241, the three-dimensional data 242, and the processing information 243 accumulated in the storage unit 240. When a non-alarm or excessive alarm is confirmed by the skilled worker, the learning processing unit 254 has the skilled worker correct it, and re-learns the trained models 244 (the first trained model 244a to the sixth trained model 244f) using the corrected information as training data. This makes it possible to improve the accuracy of the alarm based on the viewpoint and tacit knowledge of the skilled worker.
[0114] The learning processing unit 254 will be described in detail below. In the relearning of the learning processing unit 254, a set of input information used for the relearning and output information used for the relearning is used as teacher data. In the following description, the input information used for the relearning of the learning processing unit 254 may be referred to as "input information for relearning", and the output information used for the relearning of the learning processing unit 254 may be referred to as "output information for relearning".
[0115] The re-learning input information includes, for example, image data 241 (image IM) and three-dimensional data 242 stored in the storage unit 240. Then, the re-learning input information and processing information 243 linked to the re-learning input information are presented to the skilled worker. For example, the processing unit 250 (learning processing unit 254) may control the output unit 230 to cause the processing unit 250 to output the re-learning input information and the processing information 243 in a form recognizable by the skilled worker.
[0116] Here, the processing information 243 includes the recognition target information, the work content of the crane 1, the danger area A, and the warning necessity information output by the determination unit 252. That is, the processing information 243 includes final information (warning necessity information) that is finally used in the generation of a control signal by the generation unit 253, and intermediate information (recognition target information, the work content of the crane 1, and the danger area A) that is used in the processing by the determination unit 252 up to that point. Therefore, in this embodiment, the skilled worker is presented with re-learning input information, intermediate information, and final information. Each of the intermediate information and the final information is linked to the re-learning input information (image data 241 and three-dimensional data 242).
[0117] The skilled worker checks the presented input information for re-learning and the processing information 243, and checks whether there is a problem with the processing information 243. If there is a problem with the processing information 243, the skilled worker inputs the corrected processing information 243 to the safety support system S, for example, by operating the input unit 220 or the like. For example, if a non-alarm or excessive alarm is confirmed in the presented processing information 243 (i.e., if a problem is confirmed in the alarm necessity information, which is the final information), the skilled worker makes corrections to at least the alarm necessity information of the processing information 243. Also, if a problem is confirmed in intermediate information (recognition target information, work content of the crane 1, and danger area A) of the presented processing information 243, the skilled worker makes corrections to the intermediate information in which a problem is confirmed. The processing information 243 corrected by the skilled worker in this way is acquired by the safety support system S (determination unit 252) as output information for re-learning.
[0118] The learning processing unit 254 performs re-learning of the trained model 244 using a set of the re-learning output information thus acquired and the re-learning input information based on the information accumulated in the storage unit 240 as teacher data. The specific method of re-learning in the learning processing unit 254 is not particularly limited, and for example, a neural network (deep learning) such as CNN or RNN may be used. Re-learning may also be performed using the method disclosed in JP 2022-102930 A.
[0119] <Processing performed by the safety support system> Next, an example of processing performed in the safety support system S according to this embodiment will be described. FIG. 12 is a flowchart showing an example of processing performed in the safety support system S according to this embodiment. Note that the flowchart shown in FIG. 12 relates to an alarm by the alarm unit 303, and the processing related to the above-mentioned re-learning is omitted. The processing shown in FIG. 12 is started, for example, by a user issuing a command to start the processing. For example, a command to start the processing may be issued by a crane operator U' operating the operation unit 9 or by a user operating the input unit 220.
[0120] (Step S101) First, the process of step S101 is performed. In the process of step S101, the image acquisition unit 121 acquires an image IM captured by the imaging unit 301. The image IM includes a ground worker U, a hoist 22, and a suspended load L. The image acquisition unit 121 outputs the acquired image IM to the second terminal device 200 (communication unit 210) via the communication unit 110 and the network 400. The communication unit 210 outputs the image IM to the processing unit 250. In addition, the three-dimensional information acquisition unit 122 acquires three-dimensional information of the suspended load L acquired by the three-dimensional sensor 302. The three-dimensional information acquisition unit 122 outputs the acquired three-dimensional information of the suspended load L to the second terminal device 200 (communication unit 210) via the communication unit 110 and the network 400. The communication unit 210 outputs the three-dimensional information of the suspended load L to the processing unit 250. After the process of step S101 is performed, the process of step S102 (first process) and the process of step S106 (fifth process) are performed.
[0121] (Step S102) In the process of step S102, the first judgment unit 252a outputs the recognition target information by inputting the image IM to the first trained model 244a. The first judgment unit 252a outputs the output recognition target information to the second judgment unit 252b and the fourth judgment unit 252d. After the process of step S102 is performed, the process of step S103 (second process) is performed.
[0122] (Step S103) In the process of step S103, the second judgment unit 252b inputs the output of the process of step S102 (i.e., recognition target information) to the second trained model 244b, thereby outputting the work content of the crane 1. The second judgment unit 252b outputs the output work content to the third judgment unit 252c. After the process of step S103 is performed, the process of step S104 (third process) is performed.
[0123] (Step S104) In the process of step S104, the third judgment unit 252c inputs the image IM acquired by the process of step S101 and the output of the process of step S103 (i.e., the work content of the crane 1) to the third trained model 244c, thereby outputting a danger area A in the image IM. The third judgment unit 252c outputs the output danger area A to the fourth judgment unit 252d. After the process of step S104 is performed, the process of step S105 (fourth process) is performed.
[0124] (Step S105) In the process of step S105, the fourth determination unit 252d inputs the output of the process of step S102 (i.e., at least one of the position and state of the ground worker U) and the output of the process of step S104 (i.e., the danger area A) to the fourth trained model 244d, thereby determining whether or not an alarm is required by the alarm unit 303. That is, in the process of step S105, it is determined whether or not an alarm is required based on the first alarm case.
[0125] (Step S106) In the process of step S106, fifth determination unit 252e inputs image IM acquired by the process of step S101 to fifth trained model 244e, thereby outputting the position of suspending device 22. Fifth determination unit 252e outputs the output position of suspending device 22 to sixth determination unit 252f. After the process of step S106 is performed, the process of step S107 (sixth process) is performed.
[0126] (Step S107) In the process of step S107, the sixth determination unit 252f inputs, to the sixth trained model 244f, the image IM acquired in the process of step S101, the output of the process of step S106 (i.e., the position of the hoist 22), and information related to the three-dimensional information of the load L, thereby determining whether or not an alarm is required by the alarm unit 303. That is, in the process of step S107, it is determined whether or not an alarm is required based on the second alarm case.
[0127] Here, when the "information related to the three-dimensional information of the load L" is the three-dimensional information of the load L, the sixth determination unit 252f may use the three-dimensional information of the load L acquired in the process of step S101 as an input of the sixth trained model 244f. Also, when the "information related to the three-dimensional information of the load L" is the deflection information of the load L, a deflection information calculation process may be performed prior to the process of step S106. In the deflection information calculation process, the calculation unit 251 calculates the deflection information of the load L based on the three-dimensional information (height distribution information) of the load L, and outputs the calculated deflection information to the determination unit 252 (sixth determination unit 252f). Then, in the process of step S107, the output of the deflection information calculation process may be used as an input of the sixth trained model 244f.
[0128] (Step S108) After the process of step S105 and the process of step S107 are performed, the process of step S108 is performed. In the process of step S108, the judgment unit 252 integrates the judgment result of the process of step S105 and the judgment result of the process of step S107 to generate warning necessity information. The judgment unit 252 outputs the generated warning necessity information to the generation unit 253. The generation unit 253 generates a control signal for the control unit 140 to control the warning unit 303 based on the output warning necessity information. Here, the content of the generated control signal differs depending on the judgment result (warning necessity information). The generation unit 253 outputs the generated control signal to the first terminal device 100 via the communication unit 210 and the network 400. After the process of step S108 is performed, the process of step S109 is performed.
[0129] (Step S109) In the process of step S109, the control unit 140 controls the alarm unit 303 based on the control signal output in the process of step S108, and causes the alarm unit 303 to issue an alarm. Specifically, the control unit 140 causes the alarm unit 303 to issue a visually different alarm in response to a control signal that differs depending on the determination result (alarm necessity information). The method of differentiating the alarms is as described above. After the process of step S109 is performed, the process of step S110 is performed.
[0130] (Step S110) In the process of step S110, it is determined whether or not to repeat the processes of steps S101 to S109. This determination may be made by, for example, the second terminal device 200 (the determination unit 252) or the first terminal device 100. For example, when there is a command to end the repetition from the user, it may be determined to end the repetition, and when there is no command to end the repetition from the user, it may be determined not to end the repetition. When the work of the crane 1 is completed, it may be determined to end the repetition, and when the work of the crane 1 is not completed, it may be determined not to end the repetition. For example, the determination unit 252 and the trained model 244 may be configured to detect the end of the work of the crane 1 based on the image IM, the three-dimensional information of the suspended load L, and the like. Then, when the end of the work of the crane 1 is detected based on the image IM, the three-dimensional information of the suspended load L, and the like, it may be determined to end the repetition, and when the end of the work of the crane 1 is not detected, it may be determined not to end the repetition.
[0131] When it is determined that the repetition is to be ended (step S110; YES), the processing of the flowchart is ended. When it is determined that the repetition is not to be ended (step S110; NO), the processing of steps S101 to S109 is repeated. By repeating the processing of steps S101 to S109 in this manner, it is possible to continue to determine whether or not an alarm is required throughout a series of operations performed by the crane 1, and to issue an alarm as necessary. The series of operations performed by the crane 1 may include, for example, hoisting the load L, moving the load L (i.e., traversing and / or traveling), and lowering the load L. For example, if a ground worker U approaches the load L during a series of operations performed by the crane 1, and the first alarm case occurs, this can be detected and an alarm can be issued. Also, if the hoisting tool 22 shifts during a series of operations performed by the crane 1, and the second alarm case occurs, this can be detected and an alarm can be issued.
[0132] <Summary> According to the present embodiment described above, the safety of crane work can be improved by the warning. Moreover, when applying the configuration (imaging unit 301, three-dimensional sensor 302, warning unit 303, first terminal unit 100, and second terminal unit 200) of the safety support system S to the crane 1, there is no need to make any changes to the design of the existing crane 1. For example, the imaging unit 301, three-dimensional sensor 302, warning unit 303, and second terminal unit 200 only need to be attached to the crane 1 (for example, the trolley 10), and the first terminal unit 100 can be placed outside the crane 1. Moreover, since the work content of the crane 1 is identified by image recognition from the image IM, there is no need to make any changes to the control system of the crane 1 (for example, a change to output a control signal for controlling the crane 1 to the safety support system S). Therefore, according to the safety support system S of this embodiment, the safety of crane work can be improved even in existing crane facilities.
[0133] <Modification> The technical scope of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit of the present invention.
[0134] For example, it is possible to appropriately change the contents of the warning necessity information output by the determination unit 252. For example, the warning necessity information may be information indicating only whether or not a warning by the warning unit 303 is necessary.
[0135] In addition, the trained model 244 (the sixth trained model 244f) and the judgment unit 252 (the sixth judgment unit 252f) may not have three-dimensional information (height distribution information) of the suspended load L as input. In other words, the need for an alarm related to load collapse based on the deflection of the suspended load L may not be determined. In this case, the safety support system S may not have the three-dimensional sensor 302 and the three-dimensional information acquisition unit 122. Even in this case, the need for an alarm can be determined based on the image IM acquired by the image acquisition unit 121 (and the imaging unit 301). Since the three-dimensional sensor 302 is often expensive, for example, in cases where it is not necessary to determine the need for an alarm related to load collapse based on the deflection of the suspended load L, the three-dimensional sensor 302 can be omitted to reduce costs.
[0136] In addition, in the above embodiment, the recognition target information, the work content of the crane 1, the danger area A, and the necessity of an alarm related to the first alarm case are output using a plurality of trained models (the first trained model 244a to the fourth trained model 244d) and a plurality of judgment units (the first judgment unit 252a to the fourth judgment unit 252d), but this is not limited to this. The recognition target information, the work content of the crane 1, the danger area A, and the necessity of an alarm related to the first alarm case may be output using a single trained model and a single judgment unit. Similarly, in the above embodiment, the position of the hoist 22 and the necessity of an alarm related to the second alarm case are output using a plurality of trained models (the fifth trained model 244e to the sixth trained model 244f) and a plurality of judgment units (the fifth judgment unit 252e to the sixth judgment unit 252f), but this is not limited to this. The position of the sling 22 and whether or not an alarm is required for the second alarm case may be output by a single trained model and a single determination unit.
[0137] In addition, in the above embodiment, the first terminal device 100 and the second terminal device 200 are configured as different devices, but they may be configured as an integrated device. For example, the functions of the second terminal device 200 may be implemented in the first terminal device 100. Furthermore, the functions of the first terminal device 100 and the second terminal device 200 may be implemented by three or more devices.
[0138] In addition, within the scope of the spirit of the present invention, it is possible to replace the components in the above-described embodiments with well-known components as appropriate, and the above-described embodiments and variations may be combined as appropriate. [Explanation of symbols]
[0139] S...Safety support system 1...Crane 22...Hoisting equipment 120...Acquisition unit 244...Trained model 244a...First trained model 244b...Second trained model 244c...Third trained model 244d...Fourth trained model 244e...Fifth trained model 244f...Sixth trained model 251...Calculation unit 252...Judgment unit 252a...First judgment unit 252b...Second judgment unit 252c...Third judgment unit 252d...Fourth judgment unit 252e...Fifth judgment unit 252f...Sixth judgment unit 254...Learning processing unit L...Suspended load U...Ground worker IM...Image
Claims
1. an acquisition unit that acquires an image including a ground worker, a lifting tool of the crane, and a load lifted by the lifting tool; A determination unit that determines whether or not an alarm is required by an alarm unit in a situation shown in the image by inputting the image into a trained model; a control unit that causes the alarm unit to issue the alarm based on a result of the determination by the determination unit, The trained model is trained to determine whether or not the warning is necessary when the image is input, The trained models include a first trained model, a second trained model, a third trained model, and a fourth trained model; The first trained model is trained to output, when the image is input, the position and orientation of the ground worker, the position of the hoisting tool, and the position of the suspended load; The second trained model is trained to output the work content of the crane when at least one of the position and state of the ground worker, the position of the lifting tool, and the position of the lifted load are input, The third trained model is trained to output a dangerous area in the image when the work content of the crane and the image are input, The fourth trained model is trained to determine whether or not the warning is necessary when the position and orientation of the ground worker and the danger area are input, The determination unit is a first determination unit that inputs the image into the first trained model and outputs the position and orientation of the ground worker, the position of the hoist, and the position of the suspended load; A second determination unit that outputs the work content of the crane by inputting the output of the first determination unit to the second trained model; A third determination unit that outputs the dangerous area by inputting the image and the output by the second determination unit to the third trained model; A fourth determination unit that determines whether or not the warning is necessary by inputting the output by the first determination unit and the output by the third determination unit into the fourth learned model. Safety support system.
2. The trained models include a fifth trained model and a sixth trained model, The fifth trained model is trained to output the position of the hanging tool when the image is input, The sixth trained model is trained to determine whether or not the alarm is necessary when the position of the hanging device and the image are input, The determination unit is A fifth determination unit that outputs a position of the hanging device by inputting the image to the fifth trained model; A sixth determination unit that determines whether or not the warning is necessary by inputting the image and the output by the fifth determination unit into the sixth learned model. The safety support system according to claim 1 .
3. The acquisition unit repeatedly acquires the image, The determination unit repeatedly determines whether or not the warning is necessary, The control unit causes the alarm unit to issue the alarm based on the repeated results of the determination by the determination unit. The safety support system according to claim 1 or 2.
4. The control unit causes the alarm unit to issue a visually different alarm depending on the type of the alarm. The safety support system according to claim 1 or 2.
5. The acquisition unit acquires height distribution information indicating a height distribution of the suspended load, The trained model is trained to determine whether or not the warning is required when the image and information related to the height distribution information are input. The safety support system according to claim 1 or 2.
6. A calculation unit that calculates deflection information indicating a deflection of the suspended load based on the height distribution information, The trained model is trained to determine whether or not the warning is required when the image and the deflection information are input. The safety support system according to claim 5 .
7. A learning processing unit that re-learns the learned model using a set of the image and the necessity of the warning corrected by a skilled worker as training data. The safety support system according to claim 1 or 2.
8. Acquire a plurality of images including a ground worker, a lifting tool of the crane, and a load being lifted by the lifting tool; acquiring warning necessity information that is associated with the acquired plurality of images on a one-to-one basis and indicates whether or not a warning by a warning unit is necessary in a situation shown in the associated images; A trained model is created by using a pair of the image and the warning necessity information linked to the image as training data to learn the relationship between the content of the image and the necessity of the warning, Creating the trained model includes: Creating a first trained model that is trained to output the position and orientation of the ground worker, the position of the hoisting device, and the position of the suspended load when the image is input; creating a second trained model that is trained to output the work content of the crane when at least one of the position and state of the ground worker, the position of the hoist, and the position of the suspended load are input; Creating a third trained model that is trained to output a dangerous area in the image when the work content of the crane and the image are input; Creating a fourth trained model trained to determine whether or not the alarm is required when the position and orientation of the ground worker and the danger area are input. How to create a trained model.
Citation Information
Patent Citations
JP1976104571U
Crane and related operating methods
JP2015518804A
Crane work monitoring system, crane work monitoring method, dangerous state determination device, and program
JP2020093890A
Warning system, warning method, and warning program
JP2021060837A
Information processing device for crane
WO2022075340A1
