Safety support system and trained model creation method

Abstract

To provide a safety support system that can improve the safety of crane operations even with existing crane equipment, and a trained model creation method.SOLUTION: A safety support system comprises an acquisition unit that acquires an image including a ground worker, a lifting device of a crane, and a load being lifted by the lifting device, a determination unit that inputs the image into a trained model and determines whether an alarm by an alarm unit is necessary in the situation shown in the image, and a control unit that causes the alarm unit to issue the alarm based on the result of the determination by the determination unit. The trained model is trained to determine whether an alarm is necessary when the image is input.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a safety support system and a method for creating a trained model. [Background technology]

[0002] Full automation is difficult at worksites that handle small to medium-sized heavy objects, and it is essential that humans and machines work together. At such worksites, many industrial accidents occur every year due to the collapse of the objects being lifted (suspended loads). To improve the safety of work at worksites, cranes incorporating safety support systems have been proposed (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2015-518804 Summary of the Invention [Problem to be solved by the invention]

[0004] It is not uncommon for old crane equipment to have been in use for many years at the majority of construction sites. However, conventional safety support systems (see, for example, Patent Document 1) required the construction of a large-scale system that assumed automation, including safety support functions, from the crane design stage. Therefore, these safety support systems could not be applied to existing old crane equipment, and there were cases where it was not possible to improve the safety of crane work with existing equipment.

[0005] The present invention has been made in consideration of these circumstances, and aims to provide a safety support system that can improve the safety of crane operations even with existing crane equipment, and a method for creating a trained model. [Means for solving the problem]

[0006] In order to solve the above problem, the safety support system of aspect 1 of the present invention comprises an acquisition unit that acquires an image including a ground worker, a lifting device held by a crane, and a load being lifted by the lifting device, a judgment unit that inputs the image into a trained model and determines whether or not an alarm by an alarm unit is necessary in the situation shown in the image, and a control unit that causes the alarm unit to issue the alarm based on the result of the judgment by the judgment unit, and the trained model is trained to determine whether or not the alarm is necessary when the image is input.

[0007] Furthermore, aspect 2 of the present invention is a safety support system of aspect 1, wherein the trained models include a first trained model and a second trained model, and the first trained model is trained to output at least one of the position and status of the ground worker, the position of the lifting gear, and the position of the suspended load when the image is input, and the second trained model is trained to output the work content of the crane when at least one of the position and status of the ground worker, the position of the lifting gear, and the position of the suspended load are input, and the judgment unit includes a first judgment unit that inputs the image into the first trained model and outputs at least one of the position and status of the ground worker, the position of the lifting gear, and the position of the suspended load, and a second judgment unit that inputs the output of the first judgment unit into the second trained model and outputs the work content of the crane.

[0008] Furthermore, aspect 3 of the present invention relates to the safety support system of aspect 2, wherein the trained model includes a third trained model and a fourth trained model, and the third trained model is trained to output a danger area in the image when the work content of the crane and the image are input, and the fourth trained model is trained to determine whether or not the alarm is necessary when at least one of the position and status of the ground worker and the danger area are input, and the judgment unit includes a third judgment unit that outputs the danger area by inputting the image and the output by the second judgment unit into the third trained model, and a fourth judgment unit that determines whether or not the alarm is necessary by inputting the output by the first judgment unit and the output by the third judgment unit into the fourth trained model.

[0009] Furthermore, aspect 4 of the present invention is a safety support system according to any one of aspects 1 to 3, wherein the trained models include a fifth trained model and a sixth trained model, the fifth trained model being trained to output the position of the sling when the image is input, and the sixth trained model being trained to determine whether or not an alarm is required when the position of the sling and the image are input, and the determination unit includes a fifth determination unit that outputs the position of the sling by inputting the image into the fifth trained model, and a sixth determination unit that determines whether or not an alarm is required by inputting the image and the output from the fifth determination unit into the sixth trained model.

[0010] Furthermore, aspect 5 of the present invention is a safety support system according to any one of aspects 1 to 4, wherein the acquisition unit repeatedly acquires the image, the judgment unit repeatedly judges whether or not the alarm is necessary, and the control unit causes the alarm unit to issue the alarm based on the results of the repeated judgments of the judgment unit.

[0011] A sixth aspect of the present invention is the safety support system according to any one of the first to fifth aspects, wherein the control unit causes the alarm unit to issue a visually different alarm depending on the type of the alarm.

[0012] Furthermore, aspect 7 of the present invention is a safety support system according to any one of aspects 1 to 6, wherein the acquisition unit acquires height distribution information indicating the height distribution of the suspended load, and the trained model is trained to determine whether or not the alarm is necessary when the image and information related to the height distribution information are input.

[0013] Furthermore, in aspect 8 of the present invention, in the safety support system of aspect 7, a calculation unit is further provided that calculates deflection information indicating the deflection of the suspended load based on the height distribution information, and the trained model is trained to determine whether or not the alarm is necessary when the image and the deflection information are input.

[0014] Furthermore, aspect 9 of the present invention relates to a safety support system according to any one of aspects 1 to 8, further comprising a learning processing unit that re-learns the trained model using a pair of the image and the need for an alarm corrected by an experienced worker as training data.

[0015] In order to solve the above problem, a method for creating a trained model according to aspect 10 of the present invention acquires multiple images including a ground worker, a lifting device held by a crane, and a load being lifted by the lifting device, acquires alarm necessity information that is linked one-to-one with the multiple acquired images and indicates whether an alarm by an alarm unit is necessary in the situation shown in the linked images, and creates a trained model that learns the relationship between the content of the image and whether an alarm is necessary, using a pair of the image and the alarm necessity information linked to the image as training data. [Effects of the Invention]

[0016] According to the above aspects of the present invention, it is possible to provide a safety support system and a method for creating a trained model that can improve the safety of crane operations even in existing crane equipment. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a diagram illustrating an example of a crane to which a safety support system according to an embodiment is applied. [Figure 2] 1 is a block diagram showing an example of the configuration of a safety support system according to an embodiment. [Figure 3] FIG. 10 is a diagram illustrating an example of an alarm issued by an alarm unit according to the embodiment. [Figure 4] 2 is a diagram showing an example of a method for attaching an imaging unit, a distance sensor, and an alarm unit according to the embodiment to the crane shown in FIG. 1. FIG. [Figure 5] 10A and 10B are diagrams illustrating examples of the shape of deflection of a suspended load. [Figure 6] 10A and 10B are diagrams showing other examples of the shape of the deflection of the suspended load. [Figure 7] FIG. 2 is a block diagram illustrating an example of a determination unit according to the embodiment. [Figure 8] FIG. 1 is a block diagram illustrating an example of a trained model according to an embodiment. [Figure 9] 5A to 5C are diagrams illustrating an example of an image acquired by an image acquisition unit and an example of an output by a determination unit according to the embodiment. [Figure 10] FIG. 10 is a diagram showing an example of a change in a danger area depending on the working state of a crane. [Figure 11] FIG. 10 is a diagram showing an example of a position where a sling is attached to a suspended load. [Figure 12] 4 is a flowchart illustrating an example of processing performed by the safety support system according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] A safety support system S according to an embodiment of the present invention will be described below with reference to the drawings.

[0019] <Crane configuration and safety support system overview> The safety support system S according to this embodiment is a system that is applied to a crane (crane facility) and ensures the safety of crane work. The crane (crane facility) 1 shown in FIG. 1 is an example of an object to which the safety support system S is applied. The object to which the safety support system S is applied is not limited to the crane 1 shown in FIG. 1 and can be changed as appropriate.

[0020] Hereinafter, the direction along gravity (i.e., the vertical direction) will be referred to as the gravity direction Z. The upward direction in the gravity direction Z will be represented by the +Z direction and simply referred to as "upward." The downward direction in the gravity direction Z will be represented by the -Z direction and simply referred to as "downward." Furthermore, a direction that intersects (e.g., is perpendicular to) the gravity direction Z will be referred to as the first direction X. One direction in the first direction X will be represented by the +X direction and referred to as "rightward." The direction opposite to the +X direction will be represented by the -X direction and referred to as "leftward." A direction that intersects (e.g., is perpendicular to) both the gravity direction Z and the first direction X will be referred to as the second direction Y. One direction in the second direction Y will be represented by the +Y direction and referred to as "forward." The direction opposite to the +Y direction will be represented by the -Y direction and referred to as "backward." The first direction X and the second direction Y may be directions perpendicular to the gravity direction Z (ie, horizontal directions).

[0021] The crane 1 shown in FIG. 1 is a device that hoists a load L using a hoisting tool 22 (described below). The load L is an object to be hoisted by the crane 1, and may be, for example, a plate-shaped member. In the following, an example will be described in which the load L is a plate-shaped steel material that is rectangular in plan view. However, the type and shape of the load L can be changed as appropriate.

[0022] The crane 1 includes a pair of rails 2, a pair of girders 3, a pair of moving parts 4, a hoisting device 5, a rope 6, a sheave 7, a hook 8, a trolley 10, and a hooking mechanism 20. The hooking mechanism 20 includes a rope 21 and a pair of lifting devices (hackers) 22. The crane 1 is operated by a crane operator U' operating the operating part 9.

[0023] Each of the pair of rails 2 extends in the second direction Y. The pair of rails 2 are arranged at an interval in the first direction X. The rails 2 guide the movement of the moving part 4 in the second direction Y (described later).

[0024] Each of the pair of girders 3 extends in a first direction X. The pair of girders 3 are spaced apart in a second direction Y. The girders 3 guide the movement of the trolley 10 in the first direction X (described later).

[0025] The pair of moving parts 4 are provided at both ends of the pair of girders 3 in the first direction X. More specifically, one of the pair of moving parts 4 is provided so as to connect the right side (+X side) ends of the pair of girders 3 together, and the other of the pair of moving parts 4 is provided so as to connect the left side (-X side) ends of the pair of girders 3 together.

[0026] The moving part 4 (and the girder 3 connected thereto) moves along the rails 2 in the second direction Y in response to an operation performed on the operation part 9 by, for example, a crane operator U'. Specifically, a drive mechanism (not shown) having a motor or the like for moving the moving part 4 may be provided on the crane 1, and the drive mechanism may be driven in response to an operation performed on the operation part 9 by the crane operator U'.

[0027] The trolley 10 is installed on the pair of girders 3 so as to straddle the pair of girders 3. The trolley 10 may or may not have a cab for the crane operator U'.

[0028] The trolley 10 moves in the first direction X along the girder 3 in response to an operation performed on the operation unit 9 by, for example, a crane operator U'. Specifically, a drive mechanism (not shown) having a motor or the like for moving the trolley 10 may be provided on the crane 1, and the drive mechanism may be driven in response to an operation performed on the operation unit 9 by the crane operator U'.

[0029] The hoisting device 5 is provided on a trolley 10. A rope 6 having a sheave 7 connected to the end thereof is attached to the hoisting device 5. The hoisting device 5 has a motor and the like (not shown) for winding up and winding down the rope 6. Note that "winding up" refers to the operation of driving the motor so that the sheave 7 moves upward. "Winding down" refers to the operation of driving the motor so that the sheave 7 moves downward.

[0030] The hoisting device 5 hoists and lowers the rope 6 in accordance with, for example, an operation performed by a crane operator U' on the operation unit 9. Specifically, a motor or the like provided in the hoisting device 5 may be driven in accordance with an operation performed by the crane operator U' on the operation unit 9.

[0031] The hanging mechanism 20 is a mechanism for hanging (hooking) the suspended load L. The hanging mechanism 20 is hung from a hook 8 provided on the sheave 7. Specifically, the center portion of a rope 21 of the hanging mechanism 20 is hung from the hook 8. A pair of suspenders 22 are connected to both ends of the rope 21.

[0032] Each hoisting tool 22 has a base 22b and a pair of claws 22a. Each claw 22a is a part that comes into contact with the underside of the load L to support the load L when the load L is hung on the hanging mechanism 20.

[0033] In this embodiment, the claw 22a of one of the pair of suspenders 22 is hooked to one long side of the load L, and the claw 22a of the other of the pair of suspenders 22 is hooked to the other long side of the load L. That is, when the load L is hung on the hanging mechanism 20, the pair of suspenders 22 face each other in the short direction of the load L. Furthermore, in each suspender 22, the pair of claws 22a are arranged spaced apart in the horizontal direction (the longitudinal direction of the load L). Therefore, when the load L is hung on the hanging mechanism 20, each of the four claws 22a of the pair of suspenders 22 contacts the underside of the load L to support the load L. However, the configuration of the hanging mechanism 20 (hanging mechanism 22) can be modified as appropriate as long as it is possible to hang the load L.

[0034] In this embodiment, the removal of the hoisting device 22 from the load L is performed by a ground worker (ground slinger) U. The ground worker U is a worker who performs work on the ground. The ground worker U is, for example, a different person from the crane operator U'.

[0035] The operation unit 9 accepts operations by the crane operator U'. The operation unit 9 may be a physical mechanism (levers, buttons, etc.) for operating the crane 1, or may be an information processing terminal for operating the crane 1, etc.

[0036] After the ground worker U hooks the load L to the hooking mechanism 20 (hanging tool 22), the crane operator U' operates the control unit 9 to wind up the rope 6, and the load L is lifted by the hooking mechanism 20 (hanging tool 22) and rises. In this state, the crane operator U' operates the control unit 9 to move the trolley 10 and / or the moving unit 4 (girder 3). Then, the load L suspended from the hooking mechanism 20 moves in the first direction X and / or the second direction Y. In other words, the load L moves in the first direction X as the trolley 10 moves in the first direction X, and moves in the second direction Y as the moving unit 4 (girder 3) moves in the second direction Y.

[0037] Hereinafter, the movement of the load L (crane 1) in the first direction X accompanying the movement of the trolley 10 may be referred to as "lateral movement." Furthermore, the movement of the load L (crane 1) in the second direction Y accompanying the movement of the moving part 4 (girder 3) may be referred to as "traveling." That is, in this embodiment, the first direction X coincides with the lateral movement direction of the load L (crane 1), and the second direction Y coincides with the traveling direction of the load L (crane 1).

[0038] In the crane 1 described above, there is a possibility that a load may collapse due to some factor. Load collapse is a phenomenon in which the suspended load L falls from the latching mechanism 20 (the hoisting device 22). If a load collapse occurs, there is a possibility of an accident, such as the dropped load L colliding with a ground worker U. Such an accident can occur, for example, when the ground worker U is too close to the suspended load L. Hereinafter, a case in which a load collapse occurs and an accident in which the dropped load L colliding with the ground worker U is predicted, such as when the ground worker U is too close to the suspended load L, may be referred to as a "first alarm case." The safety support system S is configured to detect the first alarm case and issue an alarm when such a detection is made.

[0039] Furthermore, the load collapse occurs when the latching mechanism 20 (the sling 22) comes off the suspended load L. Hereinafter, a case where there is a sign that the latching mechanism 20 (the sling 22) will come off the suspended load L may be referred to as a "second alarm case." The safety support system S is configured to detect the second alarm case and issue an alarm when such a detection is made.

[0040] The various components of the safety support system S that issues these warnings and the processing performed by the safety support system S will be described in detail below.

[0041] <Safety support system configuration> FIG. 2 is a block diagram showing an example of the configuration of a safety support system S according to this embodiment. The safety support system S according to this embodiment includes a first terminal device 100, a second terminal device 200, an imaging unit 301, a three-dimensional sensor 302, and an alarm unit 303. As will be described in a modified example below, the safety support system S does not necessarily include the three-dimensional sensor 302. The first terminal device 100 and the second terminal device 200 are communicably connected via a network 400. The network 400 may be a network using wireless communication or a network using wired communication. The network 400 may be configured using a local area network (LAN) such as Wi-Fi (registered trademark), for example. The network 400 may also be configured by combining a plurality of networks.

[0042] The imaging unit 301 captures an image IM (see, for example, FIG. 9 ) including the ground worker U, the sling 22, and the suspended load L. The imaging unit 301 may include, for example, at least one camera for capturing the image IM. The imaging unit 301 outputs the captured image IM to an image acquisition unit 121 (described later) included in the first terminal device 100.

[0043] The three-dimensional sensor 302 measures three-dimensional information of the suspended load L. The three-dimensional information of the suspended load L includes information on the three-dimensional position of the suspended load L and information on the three-dimensional shape of the suspended load L. Therefore, the three-dimensional information of the suspended load L includes height distribution information indicating the height distribution of the suspended load L (the distribution of positions in the gravity direction Z). The three-dimensional sensor 302 may include, for example, a LiDAR that measures the distance to an object using reflected light obtained by irradiating the object with light. The three-dimensional sensor 302 outputs the measured three-dimensional information to a three-dimensional information acquisition unit 122 (described below) of the first terminal device 100.

[0044] The alarm unit 303 issues an alarm in accordance with control by a control unit 140 (described later) possessed by the first terminal device 100. The alarm unit 303 issues an alarm in a form that can be recognized by the ground worker U and / or the crane operator U'. By issuing an alarm to the ground worker U using the alarm unit 303, the ground worker U can be notified of the danger and can be urged to take refuge or correct the attachment position of the hoisting device 22. By issuing an alarm to the crane operator U' using the alarm unit 303, the crane operator U' can be notified of the danger and can be urged to stop work using the crane 1, etc. This makes it possible to prevent accidents caused by cargo collapse.

[0045] The alarm unit 303 may include a light that issues an alarm by light (for example, an LED light), a speaker that issues an alarm by sound, a wearable terminal worn by the ground worker U or the crane operator U', etc. The alarm unit 303 may issue an alarm by combining a plurality of types of these alarm means.

[0046] When a light is used as the alarm unit 303, the alarm unit 303 may project an image P as a warning by irradiating a predetermined location (for example, the vicinity of the suspended load L) with light (see, for example, FIG. 3). The position where the image P is projected is preferably a position that can be seen by both the ground worker U and the crane operator U' (in the driver's cab).

[0047] FIG. 3 is a diagram showing an example of an alarm issued by the alarm unit 303. In the example of FIG. 3, the alarm unit 303 is a plurality of lights, and projects an image P including a plurality of circles arranged in a circle from above onto the periphery of the suspended load L as an alarm. A portion of the image P is projected onto the upper surface of the suspended load L. Even when the hoisting device 22 is lifted by the crane 1, it is common for the hoisting device 22 to be positioned below the eyes of the ground worker U. Therefore, by projecting the image P as an alarm onto the periphery of the suspended load L (the upper surface of the suspended load L) in this way, the image P as an alarm can be more easily seen by the ground worker U.

[0048] If the alarm unit 303 is a goggle-type wearable terminal worn by a ground worker U or a crane operator U', the alarm unit 303 may issue an alarm by displaying visual information (text, figures, etc.) on the wearable terminal.

[0049] The imaging unit 301, three-dimensional sensor 302, and alarm unit 303 are provided (attached) to the crane 1. The imaging unit 301, three-dimensional sensor 302, and alarm unit 303 may be provided, for example, on the trolley 10. As shown in FIG. 1, the ground worker U, the hoisting device 22, and the load L are located near directly below the trolley 10. Therefore, by providing the imaging unit 301 on the trolley 10, it is possible to easily capture an image IM including these objects. Similarly, by providing the three-dimensional sensor 302 on the trolley 10, it is possible to easily measure three-dimensional information about the load L. Furthermore, if the alarm unit 303 is a light, providing the alarm unit 303 on the trolley 10 makes it possible to easily project an image P onto the periphery of the load L (the top surface of the load L).

[0050] 4 is a diagram showing an example of a method for attaching the imaging unit 301, the three-dimensional sensor 302, and the alarm unit 303 to the trolley 10. However, the method for attaching the imaging unit 301, the three-dimensional sensor 302, and the alarm unit 303 to the crane 1 (trolley 10) can be changed as appropriate.

[0051] As shown in Fig. 4, in the safety support system S according to this embodiment, a frame-shaped mounting jig J is attached to the trolley 10. When the mounting jig J is attached to the trolley 10, the mounting jig J extends downward from the trolley 10, and the lower end of the mounting jig J is located below the lower end of the girder 3. An imaging unit 301, a three-dimensional sensor 302, and an alarm unit 303 are attached to the portion of the mounting jig J located below the girder 3 so as to face downward. Note that the imaging unit 301 in the illustrated example includes two cameras.

[0052] Positioning the imaging unit 301 below the girder 3 can prevent the girder 3 from entering the imaging range of the imaging unit 301. Similarly, positioning the three-dimensional sensor 302 below the girder 3 can prevent the girder 3 from entering the detection range of the three-dimensional sensor 302. Furthermore, when a light that emits light downward to project an image P is used as the alarm unit 303, positioning the alarm unit 303 below the girder 3 can prevent the light from the alarm unit 303 from being blocked by the girder 3 and causing the image P to be missing.

[0053] The first terminal device 100 (see FIG. 2) is configured using an information processing terminal such as a personal computer. The first terminal device 100 is provided, for example, on a trolley 10. The first terminal device 100 is also referred to as an edge PC. As shown in FIG. 2, the first terminal device 100 includes a communication unit 110, an acquisition unit 120, a storage unit 130, and a control unit 140. The acquisition unit 120 includes, for example, an image acquisition unit 121 and a three-dimensional information acquisition unit 122.

[0054] The communication unit 110 has hardware for connecting to the network 400 using a cellular network, a Wi-Fi network, or the like. For example, the communication unit 110 has an antenna, a transmitting / receiving device, and the like. The communication unit 110 communicates with the second terminal device 200 (communication unit 210) via the network 400. However, the communication unit 110 may also be connected to and communicate with the second terminal device 200 (communication unit 210) via a wired connection.

[0055] The image acquisition unit 121 acquires an image IM including the ground worker U, the hoisting device 22, and the suspended load L. The image acquisition unit 121 acquires, for example, the image IM output from the imaging unit 301. The image acquisition unit 121 also outputs the acquired image IM to the second terminal device 200 (communication unit 210) via the communication unit 110 and the network 400.

[0056] The three-dimensional information acquisition unit 122 acquires three-dimensional information of the suspended load L. The three-dimensional information acquisition unit 122 acquires, for example, three-dimensional information of the suspended load L output from the three-dimensional sensor 302. The three-dimensional information acquisition unit 122 also outputs the acquired three-dimensional information of the suspended load L to the second terminal device 200 (communication unit 210) via the communication unit 110 and the network 400.

[0057] The storage unit 130 is realized by, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an SD card, a register, a hard disk drive, etc. The storage unit 130 stores data used by the control unit 140. The storage unit 130 stores data required when the control unit 140 performs processing.

[0058] Some or all of the functions of the control unit 140 are realized by, for example, a processor such as a CPU (Central Processing Unit) executing a program (software) stored in its own storage unit. Furthermore, some or all of the functions of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device such as an HDD (Hard Disk Drive) or flash memory, or may be stored on a removable storage medium such as a DVD or CD-ROM, and installed by inserting the storage medium into a drive device.

[0059] The control unit 140 controls the alarm unit 303 to issue an alarm. The control unit 140 controls the alarm unit 303 based on a control signal output by the second terminal device 200 (generation unit 253) and transmitted via the communication unit 110 and the network 400, for example.

[0060] The control unit 140 may cause the alarm unit 303 to issue visually different alarms depending on the type of alarm. Specifically, different control signals are output from the generation unit 253 depending on the determination result by the determination unit 252 (details will be described later), and the control unit 140 may cause the alarm unit 303 to issue visually different alarms in response to these different control signals. For example, the alarms issued by the alarm unit 303 when a first alarm case is detected may be visually different from those when a second alarm case is detected. For example, if the alarm unit 303 is a light that projects an image P, the shape, size, color, blinking speed, etc. of the image P may be different between the first alarm case and the second alarm case. For example, if the alarm unit 303 is a wearable terminal, the information displayed on the wearable terminal may be different between the first alarm case and the second alarm case. This allows the content of the alarm to be effectively communicated to the person to be alerted (a ground worker U or a crane operator U') even in a noisy work site.

[0061] The second terminal device 200 is configured using an information processing terminal such as a personal computer. The second terminal device 200 is provided, for example, outside the crane 1. The second terminal device 200 includes a communication unit 210, an input unit 220, an output unit 230, a storage unit 240, and a processing unit 250.

[0062] The communication unit 210 has hardware for connecting to the network 400 using a cellular network, a Wi-Fi network, or the like. For example, the communication unit 210 has an antenna, a transmitting / receiving device, and the like. The communication unit 210 communicates with the first terminal device 100 (communication unit 110) via the network 400. However, the communication unit 210 may also be connected to and communicate with the first terminal device 100 (communication unit 110) via a wired connection.

[0063] The image IM acquired by the communication unit 210 from the first terminal device 100 is output to the processing unit 250 and stored in the memory unit 240 as part of image data 241. The three-dimensional information of the suspended load L acquired by the communication unit 210 from the first terminal device 100 is output to the processing unit 250 and stored in the memory unit 240 as part of three-dimensional data 242.

[0064] The input unit 220 is configured using existing input devices such as a keyboard, a pointing device (mouse, tablet, etc.), buttons, a touch panel, etc. The input unit 220 is operated by a user when inputting the user's instructions to the second terminal device 200. The input unit 220 may be an interface for connecting an input device to the second terminal device 200. The input unit 220 may be configured using a microphone and a voice recognition device. The input unit 220 may be configured in any way as long as it is capable of inputting the user's instructions to the second terminal device 200.

[0065] The output unit 230 outputs information in a form that can be recognized by the user. The output unit 230 may be, for example, an image display device such as a liquid crystal display or an organic EL (Electro Luminescence) display. The output unit 230 may be an interface for connecting the image display device to the second terminal device 200. The output unit 230 may be a device that outputs sound, such as a speaker. The output unit 230 may be an interface for connecting an audio output device, such as a speaker or headphones, to the first terminal device 100. The output unit 230 may be configured as a touch panel integrated with the input unit 220.

[0066] The storage unit 240 is realized by, for example, a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an SD card, a register, a hard disk drive, etc. The storage unit 240 stores data used by the processing unit 250. The storage unit 240 stores data required when the processing unit 250 performs processing.

[0067] The storage unit 240 stores, for example, image data 241, three-dimensional data 242, processing information 243, and a trained model 244. The image data 241 includes an image IM acquired by the image acquisition unit 121. The three-dimensional data 242 includes three-dimensional information of the suspended load L acquired by the three-dimensional information acquisition unit 122 and deflection information (described later) calculated by the calculation unit 251. The processing information 243 includes information output by the processing performed by the processing unit 250.

[0068] The trained model 244 is data indicating the parameters, connection structure, function characteristics, etc. of the trained model used in processing by the processing unit 250 (determination unit 252). The trained model 244 has been trained to determine (output) at least whether or not an alarm is required by the alarm unit 303 when an image IM and three-dimensional information about the suspended load L are input. More specifically, the trained model 244 has been trained to determine (output) whether or not the situation shown in the image IM and the three-dimensional information when the image IM and three-dimensional information about the suspended load L are input, whether or not the situation corresponds to the first alarm case, the second alarm case, both alarm cases, or none of the alarm cases. Details of the trained model 244 will be described later.

[0069] The processing unit 250 includes, for example, a calculation unit 251, a determination unit 252, a generation unit 253, and a learning processing unit 254. Some or all of the functions of the processing unit 250 are realized by, for example, a processor such as a CPU (Central Processing Unit) executing a program (software) stored in its own storage unit. Furthermore, some or all of the functions of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device such as an HDD (Hard Disk Drive) or flash memory, or may be stored in a removable storage medium such as a DVD or CD-ROM and installed by inserting the storage medium into a drive device.

[0070] The calculation unit 251 calculates deflection information of the suspended load L based on three-dimensional information (height distribution information) of the suspended load L acquired from the first terminal device 100 via the communication unit 210. The calculation unit 251 outputs the calculated deflection information of the suspended load L to the determination unit 252. The calculation unit 251 associates the calculated deflection information of the suspended load L with the three-dimensional information (height distribution information) of the suspended load L as part of the three-dimensional data 242 and stores it in the storage unit 240. Here, the "deflection information of the suspended load L" is information indicating the deflection of the suspended load L. For example, the deflection information of the suspended load L may include information indicating the amount of deflection of the suspended load L and the shape of the deflection.

[0071] FIG. 5 is a diagram showing an example of the shape of deflection of a suspended load L. In the diagram, both longitudinal ends of the suspended load L are indicated by the symbols L1 and L2. As shown in FIG. 5, the suspended load L lifted by the hoisting tool 22 may deflect due to its own weight. Here, the calculation unit 251 may calculate the height difference Δ1 shown in the diagram as the amount of deflection (deflection information). Here, the height difference Δ1 is the difference in height between the highest point on the surface of the suspended load L and the lowest point on the surface of the suspended load L. The calculation unit 251 may also calculate the height difference Δ2 as information related to the deflection. The height difference Δ2 is the difference in height between the first end L1 of the suspended load L and the second end L2 of the suspended load L.

[0072] FIG. 6 is a diagram showing another example of the shape of deflection of the load L. In the examples described so far, the crane 1 has a pair of hoisting gears 22, but the crane 1 may have two or more pairs of hoisting gears 22. FIGS. 6(a) to 6(c) show examples of the shape of deflection (deflection mode) that may occur in the load L when the crane 1 has two pairs of hoisting gears 22. FIGS. 6(d) to 6(e) show examples of the shape of deflection (deflection mode) that may occur in the load L when the crane 1 has three pairs of hoisting gears 22. The calculation unit 251 may be configured to calculate (determine) the deflection mode of the load L from the acquired three-dimensional information of the load L and output it as deflection information of the load L. In addition, a predetermined parameter (a parameter corresponding to the above-mentioned amounts Δ1 and Δ2, for example, the amount of convexity of each convexity of the load L) may be set for each deflection mode, and the calculation unit 251 may be configured to calculate the predetermined parameter and output it as deflection information.

[0073] The determination unit 252 (see FIG. 2) inputs the image IM and the three-dimensional information into the trained model 244, and thereby determines whether or not an alarm by the alarm unit 303 is necessary in at least the situation shown in the image IM and the three-dimensional information. That is, the determination unit 252 determines whether the situation shown in the image IM and the three-dimensional information corresponds to the first alarm case, the second alarm case, both alarm cases, or none of the alarm cases. The determination unit 252 outputs the determination result (i.e., alarm necessity information, which will be described later) to the generation unit 253. Details of the determination unit 252 will be described later.

[0074] The generation unit 253 generates a control signal for the control unit 140 to control the alarm unit 303 based on the determination result by the determination unit 252 (i.e., alarm necessity information, which will be described later). The content of the generated control signal may differ depending on the determination result (alarm necessity information). The generation unit 253 outputs the control signal to the first terminal device 100 via the communication unit 210 and the network 400. As a result, the control unit 140 causes the alarm unit 303 to issue an alarm according to the determination result based on the determination result by the determination unit 252.

[0075] The learning processing unit 254 re-learns the trained model 244. Details of the re-learning by the learning processing unit 254 will be described later.

[0076] <Judgement unit and trained model> Fig. 7 is a block diagram showing an example of the determination unit 252 according to this embodiment. Fig. 8 is a block diagram showing an example of the trained model 244 according to this embodiment. Fig. 9 is a diagram showing an example of an image IM acquired by the image acquisition unit 121 according to this embodiment and an example of an output of the determination unit 252 based on the image IM.

[0077] As shown in Fig. 7, the judgment unit 252 according to this embodiment includes a first judgment unit 252a, a second judgment unit 252b, a third judgment unit 252c, a fourth judgment unit 252d, a fifth judgment unit 252e, and a sixth judgment unit 252f. As shown in Fig. 8, the trained model 244 according to this embodiment includes a first trained model 244a, a second trained model 244b, a third trained model 244c, a fourth trained model 244d, a fifth trained model 244e, and a sixth trained model 244f. The first judgment unit 252a to the fourth judgment unit 252d and the first trained model 244a to the fourth trained model 244d correspond to a configuration for detecting a first alarm case. The fifth determination unit 252e to the sixth determination unit 252f and the fifth trained model 244e to the sixth trained model 244f correspond to the configuration for detecting the second alarm case.

[0078] First, the configuration for detecting the first alarm case will be described.

[0079] The first trained model 244a (see FIG. 8) has been trained to output, when an image IM is input, at least one of the position and status of the ground worker U, the position of the hoisting gear 22, and the position of the suspended load L. In other words, the first trained model 244a is a model for image recognition of at least one of the position and status of the ground worker U, the position of the hoisting gear 22, and the position of the suspended load L. Note that the "status of the ground worker U" may include information such as the orientation of the ground worker U and the status of the work of the ground worker U (e.g., whether or not the ground worker U is attaching the hoisting gear 22 to the suspended load L). Hereinafter, the "ground worker U, the hoisting gear 22, and the suspended load L" may be collectively referred to as the "recognition target," and the "at least one of the position and status of the ground worker U, the position of the hoisting gear 22, and the position of the suspended load L" may be collectively referred to as the "recognition target information."

[0080] The first determination unit 252a (see FIG. 7) outputs recognition target information by inputting the image IM acquired by the image acquisition unit 121 (see also FIG. 2) into the first trained model 244a. The first determination unit 252a outputs the output recognition target information to the second determination unit 252b and the fourth determination unit 252d. The first determination unit 252a associates the output recognition target information with the image data 241 (and the three-dimensional data 242) as part of the processing information 243 and stores it in the storage unit 240. In the example shown in FIG. 9, the first determination unit 252a outputs the position B1 of the ground worker U, the position B2 of the hoist 22, and the position B3 of the load L in the form of a bounding box as the recognition target information, as well as the orientation D of the ground worker U.

[0081] The second trained model 244b (see FIG. 8) is trained to output the operation details of the crane 1 when recognition target information is input. Here, examples of the operation details of the crane 1 include "hoisting," "lowering," "traverse," "travel," and "traverse and travel." That is, the second trained model 244b is trained to output (determine) whether the operation details of the crane 1 are "hoisting," "lowering," "traverse," "travel," or "travel and traverse" when recognition target information is input. Note that "travel and traverse" refers to a state in which the crane 1 (load L) is moving in a direction tilted with respect to both the first direction X and the second direction Y as a result of the crane 1 (load L) traveling and traversing at the same time.

[0082] The second determination unit 252b (see FIG. 7) inputs the output of the first determination unit 252a (i.e., the recognition target information) into the second trained model 244b, thereby outputting the work content of the crane 1. The second determination unit 252b outputs the output work content to the third determination unit 252c. The second determination unit 252b associates the output work content with the image data 241 (and the three-dimensional data 242) and stores it in the memory unit 240 as part of the processing information 243.

[0083] The third trained model 244c (see FIG. 8) has been trained to output a danger area A in the image IM when the work content of the crane 1 and the image IM are input. Here, the danger area A is an area where, if a ground worker U enters, an accident due to the fall of the suspended load L may occur. In other words, if the ground worker U is present in the danger area A, it may correspond to the first alarm case described above. In the example shown in FIG. 9, the danger area A is output as an area including the area B3 of the suspended load L and its surrounding area.

[0084] As a result of careful consideration by the inventors of the present application, it was determined that this danger area A should change depending on the type of work being performed by the crane 1. Therefore, the third trained model 244c according to this embodiment is trained to output a different danger area A depending on the type of work being performed by the crane 1. For example, the danger area A that is output differs depending on whether the work state of the crane 1 is "hoisting," "lowering," "traversing," "traveling," or "traveling and traversing."

[0085] FIG. 10 is a diagram showing an example of changes in the danger zone A depending on the working state of the crane 1. FIG. 10(a) shows the case where the working state of the crane 1 is "hoisting" or "lowering." FIG. 10(b), FIG. 10(c), and FIG. 10(d) show the cases where the working state of the crane 1 is "traverse," "travel," and "travel and traverse," respectively. The white arrows shown in FIG. 10(b) to FIG. 10(d) indicate the direction of travel of the crane 1 (load L).

[0086] 10(a), when the working state of the crane 1 is "hoisting up" or "lowering down," an area A0 including the suspended load L (area B3) and its surrounding area is output as the danger area A. Hereinafter, this area A0 may be referred to as the "basic area A0."

[0087] Meanwhile, as a result of careful consideration by the inventors of the present application, it was found that when the working state of the crane 1 is either "traverse," "travel," or "travel and traverse," not only the basic region A0 but also the region A1 located forward of the basic region A0 in the direction of movement of the crane 1 (the load L) should be set as the danger region A (see Figures 10(b) to 10(d)). In other words, when the crane 1 (the load L) is moving horizontally, it was found that the danger region A should include not only the basic region A0 but also the region A1. Hereinafter, this region A1 to be added to the basic region A0 may be referred to as the "additional region A1." The third trained model 244c according to this embodiment is trained to output a danger region A that combines the basic region A0 and the additional region A1 when the crane 1 is either "traverse," "travel," or "travel and traverse" (i.e., when the crane 1 (the load L) is moving horizontally).

[0088] It is considered that the size of the basic area A0 and the additional area A1 should vary depending on the size, shape, material, weight, and surface condition (slipperiness, etc.) of the load L. Therefore, the third trained model 244c according to this embodiment is configured to output the danger area A based not only on the working state of the crane 1 but also on the image IM. This makes it possible to set the danger area A appropriately for various loads L.

[0089] The third determination unit 252c (see FIG. 7) inputs the image IM acquired by the image acquisition unit 121 and the output of the second determination unit 252b (i.e., the work content of the crane 1) into the third trained model 244c, and outputs a danger area A in the image IM. The third determination unit 252c outputs the output danger area A to the fourth determination unit 252d. The third determination unit 252c associates the output danger area A with the image data 241 (and the three-dimensional data 242) and stores it in the memory unit 240 as part of the processing information 243.

[0090] The fourth trained model 244d (see Figure 8) is trained to determine whether or not an alarm is required by the alarm unit 303 when at least one of the position and status of ground worker U and the danger area A are input. In other words, the fourth trained model 244d is trained to determine whether or not the first alarm case is required when at least one of the position and status of ground worker U and the danger area A are input.

[0091] For example, if a ground worker U (area B1) is present within the danger area A (see, for example, FIG. 9), there is a high possibility that an accident will occur in which the load L comes into contact with the ground worker U when the load L falls. Therefore, if the ground worker U (area B1) is present within the danger area A, there is a high possibility that an alarm from the alarm unit 303 is necessary. On the other hand, even if the ground worker U (area B1) is located within the danger area A, there may be cases in which an alarm from the alarm unit 303 is unnecessary depending on the state of the ground worker U. One example of such a case is when the ground worker U is considered to be paying attention to the load L, for example, when the ground worker U is facing the direction of the load L. Therefore, by including the state of the ground worker U (such as orientation D) in the input of the fourth trained model 244d, it is possible to more accurately determine whether an alarm is necessary.

[0092] The fourth determination unit 252d inputs the output of the first determination unit 252a (i.e., at least one of the position and state of the ground worker U) and the output of the third determination unit 252c (i.e., the danger area A) to the fourth trained model 244d, thereby determining whether or not an alarm is required by the alarm unit 303. That is, the fourth determination unit 252d determines whether or not an alarm is required based on the first alarm case.

[0093] Next, a configuration for detecting the second alarm case will be described.

[0094] The fifth trained model 244e (see FIG. 8) is trained to output the position of the sling 22 when an image IM is input. In other words, the fifth trained model 244e is a model for image recognition of the position of the sling 22.

[0095] The fifth determination unit 252e (see FIG. 7) outputs the position of the sling 22 by inputting the image IM acquired by the image acquisition unit 121 into the fifth trained model 244e. The fifth determination unit 252e outputs the output position of the sling 22 to the sixth determination unit 252f. The sixth determination unit 252f may associate the output position of the sling 22 with the image data 241 (and the three-dimensional data 242) as part of the processing information 243 and store it in the storage unit 240. Note that the fifth determination unit 252e may output the position of the sling 22 by inputting the image IM into the above-described first trained model 244a. In this case, the trained model 244 does not need to include the fifth trained model 244e.

[0096] The sixth trained model 244f (see FIG. 8) is trained to determine whether or not an alarm is required by the alarm unit 303 when the position of the sling 22, the image IM, and information related to the three-dimensional information of the load L are input. In other words, the sixth trained model 244f is trained to determine whether or not the second alarm case occurs when the position of the sling 22, the image IM, and information related to the three-dimensional information of the load L are input.

[0097] As a result of careful consideration, the inventors of the present application have concluded that there are two cases in which the sling 22 is likely to come off the load L (in other words, when there are signs that the sling 22 will come off the load L). The first case is when the position at which the sling 22 is attached to the load L is inappropriate. The second case is when the sling 22 slips due to deflection of the load L. Based on this consideration, the sixth trained model 244f according to this embodiment employs as input the position of the sling 22, the image IM, and information related to the three-dimensional information of the load L. These will be described in detail below.

[0098] First, the first case will be described. FIG. 11 is a diagram showing an example of the position at which the sling 22 is attached to the load L. Here, an optimal position (optimal position ID) exists for attaching the sling 22 to the load L. The optimal position ID is the optimal position for stably supporting the load L with the sling 22. The optimal position ID is considered to be determined by the position of the center of gravity of the sling 22. As shown in FIG. 11, if the position at which the sling 22 is attached to the load L (hereinafter referred to as the "attachment position of the sling 22") deviates from the optimal position ID, there is a high possibility that the load L will fall from the sling 22 (the sling 22 will become detached from the load L). Therefore, the sixth trained model 244f according to this embodiment is configured to determine whether an alarm is necessary (i.e., whether the first case described above applies) based on the position of the sling 22.

[0099] The processing unit 250 may also include an estimation unit (not shown) that estimates the center of gravity of the load L from the image IM. The estimation unit may estimate the position of the center of gravity of the load L, for example, by inputting the image IM to a trained model that outputs (estimates) the position of the center of gravity of the load L when the image IM including the load L is input. Alternatively, the estimation unit may estimate the position of the center of gravity of the load L on a rule-based basis. The input of the sixth trained model 244f may further include the position of the center of gravity of the load L estimated by the estimation unit. In other words, the sixth trained model 244f may be trained to determine whether or not an alarm needs to be issued by the alarm unit 303 when the position of the hoisting device 22, the image IM, information related to the three-dimensional information of the load L, and the position of the center of gravity of the load L are input. The estimation unit may store the estimated center of gravity of the suspended load L as part of the processing information 243 in the storage unit 240 in association with the image data 241 (and the three-dimensional data 242).

[0100] Next, the second case will be described. The likelihood of slippage of the hoisting tool 22 is thought to vary depending on the magnitude and shape of the deflection of the load L. Here, the three-dimensional information (height distribution information) of the load L acquired by the three-dimensional information acquisition unit 122 is thought to include information on the magnitude and shape of the deflection of the load L. Therefore, the sixth trained model 244f according to this embodiment is configured to determine whether an alarm is necessary (i.e., whether the above-described second case applies) based on information related to the three-dimensional information (height distribution information) of the load L.

[0101] Here, the "information related to the three-dimensional information (height distribution information) of the suspended load L" may be, for example, the three-dimensional information (height distribution information) of the suspended load L itself, or information calculated based on the three-dimensional information (height distribution information) of the suspended load L. For example, the "information related to the three-dimensional information (height distribution information) of the suspended load L" may be deflection information calculated by the calculation unit 251. By using deflection information instead of the three-dimensional information of the suspended load L itself as input to the sixth trained model 244f, it is expected that the judgment accuracy by the sixth trained model 244f will be improved.

[0102] Note that a method of detecting the stable state of a suspended load L is known, which involves detecting the inclination of the suspended load L (for example, the method disclosed in JP 2021-054584 A). However, because a suspended load L such as a steel material is subject to deflection, it has been difficult to determine the stable state of the suspended load L based on the inclination alone. In this embodiment, three-dimensional information (height distribution information) of the suspended load L, which includes information related to deflection, and deflection information calculated by the calculation unit 251 are used as inputs to the trained model. Therefore, compared to conventional methods that use only the inclination, the accuracy of detecting the stable state of the suspended load L can be improved.

[0103] Furthermore, in the first case, even if the attachment position of the sling 22 is the same, the possibility of the load L falling from the sling 22 is thought to change depending on the size, shape, material, weight, and surface condition (slipperiness, etc.) of the load L. Similarly, in the second case, even if the deflection of the load L is the same, the possibility of the load L falling from the sling 22 is thought to change depending on the size, shape, material, weight, and surface condition (slipperiness, etc.) of the load L. Therefore, the sixth trained model 244f according to this embodiment is configured to determine whether an alarm is necessary based not only on information related to the position of the sling 22 and the three-dimensional information of the load L, but also on the image IM. This makes it possible to appropriately determine whether an alarm is necessary for various loads L.

[0104] The sixth determination unit 252f (see FIG. 7) inputs the image IM, the output of the fifth determination unit 252e (i.e., the position of the hoisting tool 22), and information related to the three-dimensional information of the load L into the sixth trained model 244f, thereby determining whether or not an alarm needs to be issued by the alarm unit 303. Note that if "information related to the three-dimensional information of the load L," which is one of the inputs of the sixth trained model 244f, is three-dimensional information of the load L, the sixth determination unit 252f may use the three-dimensional information of the load L acquired by the three-dimensional information acquisition unit 122 as input to the sixth trained model 244f. Note that if "information related to the three-dimensional information of the load L," which is one of the inputs of the sixth trained model 244f, is deflection information of the load L, the sixth determination unit 252f may use the deflection information of the load L calculated by the calculation unit 251 as input to the sixth trained model 244f.

[0105] The determination unit 252 outputs the determination result by the fourth determination unit 252d and the determination result by the sixth determination unit 252f to the generation unit 253. Specifically, the determination unit 252 outputs alarm necessity information to the generation unit 253 based on the determination results by the fourth determination unit 252d and the sixth determination unit 252f. For example, the determination unit 252 may combine the determination results by the fourth determination unit 252d and the sixth determination unit 252f to generate alarm necessity information and output it to the generation unit 253. The determination unit 252 stores the generated alarm necessity information in the storage unit 240 as part of the processing information 243, linked to the image data 241 and the three-dimensional data 242.

[0106] Here, the "alarm necessity information" is information related to an alarm to be issued by the alarm unit 303. Specifically, the alarm necessity information may be information indicating whether an alarm by the alarm unit 303 is necessary. More specifically, the alarm necessity information may be information indicating any of "an alarm related to a first alarm case is necessary," "an alarm related to a second alarm case is necessary," "an alarm related to both the first alarm case and the second alarm case is necessary," and "an alarm related to neither the first alarm case nor the second alarm case is necessary." In other words, the alarm necessity information may include not only information indicating whether an alarm is necessary, but also information indicating the type of alarm (what type of alarm should be issued).

[0107] The trained models 244 (first trained model 244a to sixth trained model 244f) described above may be created by learning (machine learning) the relationship between the input information and the output information using pairs of multiple pieces of input information and output information linked one-to-one to the input information as training data. The input information includes, for example, an image IM including the recognition target and three-dimensional information of the suspended load L. The output information includes recognition target information, the work content of the crane 1, the danger area A, and information on whether an alarm is required.

[0108] That is, the trained model 244 (first trained model 244a to sixth trained model 244f) may be created by performing the steps of acquiring multiple pieces of input information as described above, acquiring output information linked one-to-one to the acquired multiple pieces of input information, and creating the trained model 244 (first trained model 244a to sixth trained model 244f) using the acquired pairs of input information and output information as training data.

[0109] It is preferable that the multiple pieces of input information used for learning include input information corresponding to different suspended loads L. In this case, it is possible to create trained models 244 (first trained model 244a to sixth trained model 244f) that can provide highly accurate outputs for various suspended loads L.

[0110] Furthermore, the trained models 244 (first trained model 244a to sixth trained model 244f) may be created by the safety support system S (for example, the learning processing unit 254). In this case, input information that is part of the training data may be acquired by the acquisition unit 120. The acquired input information by the acquisition unit 120 may be presented to a user (for example, a skilled worker) by the output unit 230 or the like. The user may then confirm the acquired input information via the output unit 230 or the like, and determine output information that is considered appropriate based on the confirmed input information. The user may then have the safety support system S acquire the output information that is determined to be appropriate by operating the input unit 220 or the like.

[0111] Alternatively, the trained model 244 may be created by a learning device (hereinafter referred to as a "separate learning device") provided separately from the safety support system S, and then the created trained model 244 may be stored in the storage unit 240. In this case, the input information that is part of the training data may be acquired by an acquisition means provided in the separate learning device. Furthermore, the output information that is part of the training data may be input by a user (e.g., an experienced worker) to the separate learning device using a method similar to that described above.

[0112] Furthermore, the specific method of learning (machine learning) in the safety support system S or a separate learning device is not particularly limited, but for example, neural networks (deep learning) such as CNN and RNN may be used. Furthermore, learning (machine learning) may be performed using the method disclosed in Japanese Patent Application Laid-Open No. 2022-102930.

[0113] <Relearning by the learning processing unit> The determination of whether an alarm is necessary by the determination unit 252 may be imperfect. That is, the determination unit 252 may determine that an alarm is unnecessary when an alarm is actually necessary (non-alarm), or may determine that an alarm is necessary when an alarm is not actually necessary (excessive alarm). As described above, the storage unit 240 according to this embodiment stores (accumulates) image data 241 (image IM), three-dimensional data 242, and processing information 243 output by processing by the determination unit 252. The processing information 243 includes alarm necessity information linked to the image data 241 and the three-dimensional data 242. Therefore, a skilled worker can detect a non-alarm or excessive alarm by checking the image data 241, three-dimensional data 242, and processing information 243 accumulated in the storage unit 240. When a false alarm or an excessive alarm is confirmed by the skilled worker, the learning processing unit 254 has the skilled worker correct it and re-learns the trained models 244 (the first trained model 244a to the sixth trained model 244f) using the corrected information as training data. This makes it possible to improve the accuracy of warnings based on the viewpoints and tacit knowledge of the skilled worker.

[0114] The learning processing unit 254 will be described in detail below. For the relearning by the learning processing unit 254, a set of input information used for the relearning and output information used for the relearning is used as training data. In the following description, the input information used for the relearning by the learning processing unit 254 may be referred to as "relearning input information," and the output information used for the relearning by the learning processing unit 254 may be referred to as "relearning output information."

[0115] The relearning input information includes, for example, image data 241 (image IM) and three-dimensional data 242 stored in the storage unit 240. Then, the relearning input information and processing information 243 linked to the relearning input information are presented to the skilled worker. For example, the processing unit 250 (learning processing unit 254) may control the output unit 230 to cause the processing unit 250 to output the relearning input information and the processing information 243 in a form recognizable by the skilled worker.

[0116] Here, the processing information 243 includes the recognition target information, the work content of the crane 1, the danger area A, and the alarm necessity information output by the determination unit 252. That is, the processing information 243 includes final information (alarm necessity information) that is ultimately used to generate a control signal by the generation unit 253, and intermediate information (recognition target information, the work content of the crane 1, and the danger area A) that is used in the processing by the determination unit 252 up to that point. Therefore, in this embodiment, the skilled worker is presented with re-learning input information, intermediate information, and final information. Each of the intermediate information and final information is linked to the re-learning input information (image data 241 and three-dimensional data 242).

[0117] The skilled worker checks the presented input information for relearning and the processing information 243, and confirms whether there is a problem with the processing information 243. If there is a problem with the processing information 243, the skilled worker operates the input unit 220, for example, to input the corrected processing information 243 to the safety support system S. For example, if a false alarm or an excessive alarm is confirmed in the presented processing information 243 (i.e., if a problem is confirmed in the alarm necessity information, which is the final information), the skilled worker corrects at least the alarm necessity information in the processing information 243. Furthermore, if a problem is confirmed in the intermediate information (recognition target information, work content of the crane 1, and danger area A) in the presented processing information 243, the skilled worker corrects the intermediate information in which the problem is confirmed. The processing information 243 corrected by the skilled worker in this way is acquired by the safety support system S (determination unit 252) as output information for relearning.

[0118] The learning processing unit 254 uses, as training data, a set of the thus acquired re-learning output information and the re-learning input information based on the information stored in the storage unit 240, and performs re-learning of the trained model 244. The specific method of re-learning in the learning processing unit 254 is not particularly limited, and may be, for example, a neural network (deep learning) such as CNN or RNN. Alternatively, re-learning may be performed using the method disclosed in JP 2022-102930 A.

[0119] <Processing performed by the safety support system> Next, an example of processing performed in the safety support system S according to this embodiment will be described. FIG. 12 is a flowchart showing an example of processing performed in the safety support system S according to this embodiment. Note that the flowchart shown in FIG. 12 relates to an alarm issued by the alarm unit 303, and the processing related to the above-mentioned relearning is omitted. The processing shown in FIG. 12 is started, for example, when a user issues a command to start the processing. For example, the command to start the processing may be issued when the crane operator U' operates the operation unit 9 or when the user operates the input unit 220.

[0120] (Step S101) First, the processing of step S101 is performed. In the processing of step S101, the image acquisition unit 121 acquires an image IM captured by the imaging unit 301. The image IM includes the ground worker U, the hoisting device 22, and the suspended load L. The image acquisition unit 121 outputs the acquired image IM to the second terminal device 200 (communication unit 210) via the communication unit 110 and the network 400. The communication unit 210 outputs the image IM to the processing unit 250. Furthermore, the three-dimensional information acquisition unit 122 acquires three-dimensional information of the suspended load L acquired by the three-dimensional sensor 302. The three-dimensional information acquisition unit 122 outputs the acquired three-dimensional information of the suspended load L to the second terminal device 200 (communication unit 210) via the communication unit 110 and the network 400. The communication unit 210 outputs the three-dimensional information of the suspended load L to the processing unit 250. After the process of step S101 is performed, the process of step S102 (first process) and the process of step S106 (fifth process) are performed.

[0121] (Step S102) In the processing of step S102, the first determination unit 252a inputs an image IM to the first trained model 244a, thereby outputting recognition target information. The first determination unit 252a outputs the output recognition target information to the second determination unit 252b and the fourth determination unit 252d. After the processing of step S102 is performed, the processing of step S103 (second processing) is performed.

[0122] (Step S103) In the processing of step S103, the second determination unit 252b inputs the output of the processing of step S102 (i.e., recognition target information) into the second trained model 244b, thereby outputting the work content of the crane 1. The second determination unit 252b outputs the output work content to the third determination unit 252c. After the processing of step S103 is performed, the processing of step S104 (third processing) is performed.

[0123] (Step S104) In the processing of step S104, the third determination unit 252c inputs the image IM acquired by the processing of step S101 and the output of the processing of step S103 (i.e., the work content of the crane 1) into the third trained model 244c, thereby outputting a danger area A in the image IM. The third determination unit 252c outputs the output danger area A to the fourth determination unit 252d. After the processing of step S104 is performed, the processing of step S105 (fourth processing) is performed.

[0124] (Step S105) In the processing of step S105, the fourth determination unit 252d inputs the output of the processing of step S102 (i.e., at least one of the position and state of the ground worker U) and the output of the processing of step S104 (i.e., the danger area A) into the fourth trained model 244d, thereby determining whether or not an alarm is required by the alarm unit 303. That is, in the processing of step S105, it is determined whether or not an alarm is required based on the first alarm case.

[0125] (Step S106) In the processing of step S106, fifth determination unit 252e inputs image IM acquired by the processing of step S101 into fifth trained model 244e, thereby outputting the position of sling device 22. Fifth determination unit 252e outputs the output position of sling device 22 to sixth determination unit 252f. After the processing of step S106 is performed, the processing of step S107 (sixth processing) is performed.

[0126] (Step S107) In the processing of step S107, the sixth determination unit 252f inputs to the sixth trained model 244f the image IM acquired by the processing of step S101, the output of the processing of step S106 (i.e., the position of the hoisting tool 22), and information related to the three-dimensional information of the load L, thereby determining whether or not an alarm is required by the alarm unit 303. That is, in the processing of step S107, it is determined whether or not an alarm is required based on the second alarm case.

[0127] Here, if the "information related to the three-dimensional information of the load L" is three-dimensional information of the load L, the sixth determination unit 252f may use the three-dimensional information of the load L acquired in the processing of step S101 as input to the sixth trained model 244f. Also, if the "information related to the three-dimensional information of the load L" is deflection information of the load L, a deflection information calculation process may be performed prior to the processing of step S106. In the deflection information calculation process, the calculation unit 251 calculates the deflection information of the load L based on the three-dimensional information (height distribution information) of the load L and outputs the calculated deflection information to the determination unit 252 (sixth determination unit 252f). Then, in the processing of step S107, the output of the deflection information calculation process may be used as input to the sixth trained model 244f.

[0128] (Step S108) After the processing of step S105 and the processing of step S107 are performed, the processing of step S108 is performed. In the processing of step S108, the determination unit 252 integrates the determination result of the processing of step S105 and the determination result of the processing of step S107 to generate alarm necessity information. The determination unit 252 outputs the generated alarm necessity information to the generation unit 253. The generation unit 253 generates a control signal for the control unit 140 to control the alarm unit 303 based on the output alarm necessity information. Here, the content of the generated control signal differs depending on the determination result (alarm necessity information). The generation unit 253 outputs the generated control signal to the first terminal device 100 via the communication unit 210 and the network 400. After the processing of step S108 is performed, the processing of step S109 is performed.

[0129] (Step S109) In the processing of step S109, the control unit 140 controls the alarm unit 303 based on the control signal output in the processing of step S108, and causes the alarm unit 303 to issue an alarm. Specifically, the control unit 140 causes the alarm unit 303 to issue a visually different alarm in response to a control signal that differs depending on the determination result (alarm necessity information). The method of differentiating the alarms is as described above. After the processing of step S109 is performed, the processing of step S110 is performed.

[0130] (Step S110) In the processing of step S110, it is determined whether or not to repeat the processing of steps S101 to S109. This determination may be made, for example, by the second terminal device 200 (determination unit 252) or the first terminal device 100. For example, it may be determined to end the repetition when a command to end the repetition is received from the user, and it may be determined not to end the repetition when a command to end the repetition is not received from the user. It may be determined to end the repetition when the operation of the crane 1 has finished, and it may be determined not to end the repetition when the operation of the crane 1 has not finished. For example, the determination unit 252 and the trained model 244 may be configured to detect the end of the operation of the crane 1 based on the image IM, three-dimensional information of the suspended load L, etc. Then, it may be determined to end the repetition when the end of the operation of the crane 1 is detected based on the image IM, three-dimensional information of the suspended load L, etc., and it may be determined not to end the repetition when the end of the operation of the crane 1 is not detected.

[0131] If it is determined that the repetition should be ended (step S110; YES), the processing of the flowchart ends. If it is determined that the repetition should not be ended (step S110; NO), the processing of steps S101 to S109 is repeated. By repeating the processing of steps S101 to S109 in this manner, it is possible to continuously determine whether an alarm is necessary throughout a series of operations performed by the crane 1, and to issue an alarm as necessary. The series of operations performed by the crane 1 may include, for example, hoisting the load L, moving the load L (i.e., traversing and / or traveling), and lowering the load L. For example, if a ground worker U approaches the load L during a series of operations performed by the crane 1, and a first alarm case occurs, this can be detected and an alarm can be issued. Furthermore, if the hoisting gear 22 shifts during a series of operations performed by the crane 1, and a second alarm case occurs, this can be detected and an alarm can be issued.

[0132] <Summary> According to the present embodiment described above, the safety of crane operations can be improved by issuing an alarm. Furthermore, applying the components of the safety support system S (the image capture unit 301, the three-dimensional sensor 302, the alarm unit 303, the first terminal unit 100, and the second terminal unit 200) to the crane 1 does not require any changes to the design of the existing crane 1. For example, the image capture unit 301, the three-dimensional sensor 302, the alarm unit 303, and the second terminal unit 200 can simply be attached to the crane 1 (e.g., the trolley 10), and the first terminal unit 100 can be placed outside the crane 1. Furthermore, because the work content of the crane 1 is identified by image recognition from the image IM, there is no need to make any changes to the control system of the crane 1 (e.g., a change to have the safety support system S output a control signal for controlling the crane 1). Therefore, the safety support system S according to this embodiment can improve the safety of crane operations even with existing crane facilities.

[0133] <Modification> The technical scope of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit of the present invention.

[0134] For example, it is possible to appropriately change the content of the warning necessity information output by the determination unit 252. For example, the warning necessity information may be information indicating only whether or not the warning unit 303 needs to issue a warning.

[0135] Furthermore, the trained model 244 (sixth trained model 244f) and the judgment unit 252 (sixth judgment unit 252f) do not need to input three-dimensional information (height distribution information) of the suspended load L. In other words, it is not necessary to determine whether or not an alarm related to cargo collapse is required based on the deflection of the suspended load L. In this case, the safety support system S does not need to include the three-dimensional sensor 302 and the three-dimensional information acquisition unit 122. Even in this case, it is possible to determine whether or not an alarm is required based on the image IM acquired by the image acquisition unit 121 (and the imaging unit 301). Because the three-dimensional sensor 302 is often expensive, costs can be reduced by omitting the three-dimensional sensor 302, for example, in cases where it is not necessary to determine whether or not an alarm related to cargo collapse is required based on the deflection of the suspended load L.

[0136] Furthermore, in the above embodiment, multiple trained models (first trained model 244a to fourth trained model 244d) and multiple judgment units (first judgment unit 252a to fourth judgment unit 252d) were used to output the recognition target information, the work content of crane 1, the danger area A, and whether or not an alarm is required for the first alarm case, but this is not limited to this. A single trained model and a single judgment unit may output the recognition target information, the work content of crane 1, the danger area A, and whether or not an alarm is required for the first alarm case. Similarly, in the above embodiment, multiple trained models (fifth trained model 244e to sixth trained model 244f) and multiple judgment units (fifth judgment unit 252e to sixth judgment unit 252f) were used to output the position of hoisting device 22 and whether or not an alarm is required for the second alarm case, but this is not limited to this. The position of the sling 22 and whether or not an alarm is required in the second alarm case may be output by a single trained model and a single determination unit.

[0137] Furthermore, although the first terminal device 100 and the second terminal device 200 are configured as separate devices in the above embodiment, they may be configured as an integrated device. For example, the functions of the second terminal device 200 may be implemented in the first terminal device 100. Furthermore, the functions of the first terminal device 100 and the second terminal device 200 may be implemented by three or more devices.

[0138] In addition, it is possible to replace the components in the above-described embodiments with well-known components as appropriate, and the above-described embodiments and variations may be combined as appropriate, without departing from the spirit of the present invention. [Explanation of symbols]

[0139] S...Safety support system 1...Crane 22...Lifting equipment 120...Acquisition unit 244...Trained model 244a...First trained model 244b...Second trained model 244c...Third trained model 244d...Fourth trained model 244e...Fifth trained model 244f...Sixth trained model 251...Calculation unit 252...Determination unit 252a...First determination unit 252b...Second determination unit 252c...Third determination unit 252d...Fourth determination unit 252e...Fifth determination unit 252f...Sixth determination unit 254...Learning processing unit L...Lifted load U...Ground worker IM...Image

Claims

1. an acquisition unit that acquires an image including a ground worker, a lifting tool provided on the crane, and a load lifted by the lifting tool; a determination unit that determines whether an alert by an alert unit is necessary in a situation shown in the image by inputting the image into a trained model; a control unit that causes the alarm unit to issue the alarm based on a result of the determination by the determination unit, The trained model is trained to determine whether or not the warning is necessary when the image is input. Safety support system.

2. The trained models include a first trained model and a second trained model; the first trained model is trained to output, when the image is input, at least one of the position and state of the ground worker, the position of the hoisting device, and the position of the suspended load; the second trained model is trained to output the work content of the crane when at least one of the position and state of the ground worker, the position of the lifting tool, and the position of the lifted load are input; The determination unit a first determination unit that inputs the image into the first trained model and outputs at least one of the position and state of the ground worker, the position of the hoisting device, and the position of the suspended load; A second determination unit that outputs work content of the crane by inputting the output by the first determination unit into the second trained model, The safety support system according to claim 1 .

3. The trained models include a third trained model and a fourth trained model; The third trained model is trained to output a dangerous area in the image when the work content of the crane and the image are input, The fourth trained model is trained to determine whether or not the alarm is necessary when at least one of the position and state of the ground worker and the danger area are input, The determination unit a third determination unit that outputs the dangerous area by inputting the image and the output by the second determination unit into the third trained model; A fourth determination unit that determines whether or not the warning is necessary by inputting the output by the first determination unit and the output by the third determination unit into the fourth trained model. The safety support system according to claim 2 .

4. The trained models include a fifth trained model and a sixth trained model, The fifth trained model is trained to output the position of the hanging device when the image is input, The sixth trained model is trained to determine whether or not the alarm is necessary when the position of the sling and the image are input, The determination unit a fifth determination unit that outputs the position of the hanging device by inputting the image into the fifth trained model; A sixth determination unit that determines whether or not the warning is necessary by inputting the image and the output by the fifth determination unit into the sixth trained model. The safety support system according to any one of claims 1 to 3.

5. The acquisition unit repeatedly acquires the image, The determination unit repeatedly determines whether or not the warning is necessary, The control unit causes the alarm unit to issue the alarm based on the repeated determination results of the determination unit. The safety support system according to any one of claims 1 to 3.

6. The control unit causes the alarm unit to issue a visually different alarm depending on the type of alarm. The safety support system according to any one of claims 1 to 3.

7. The acquisition unit acquires height distribution information indicating a height distribution of the suspended load, The trained model is trained to determine whether the alarm is necessary when the image and information related to the height distribution information are input. The safety support system according to any one of claims 1 to 3.

8. A calculation unit is further provided that calculates deflection information indicating the deflection of the suspended load based on the height distribution information, The trained model is trained to determine whether the alarm is necessary when the image and the deflection information are input. The safety support system according to claim 7.

9. The system further includes a learning processing unit that re-learns the trained model using a set of the image and the need for the warning corrected by a skilled worker as training data. The safety support system according to any one of claims 1 to 3.

10. Acquire a plurality of images including a ground worker, a lifting tool of the crane, and a load being lifted by the lifting tool; acquiring alarm necessity information that is associated with the acquired plurality of images on a one-to-one basis and indicates whether an alarm by an alarm unit is necessary in the situation shown in the associated images; A trained model is created that learns the relationship between the content of the image and the necessity of the alarm using a pair of the image and the alarm necessity information linked to the image as training data. How to create a trained model.

Images