Display device, excavator, and information processing device

Abstract

The present invention relates to a display device, an excavator, and an information processing device, and provides a technique for enabling an operator of the excavator, etc. to easily grasp the position of the detected object even when an object to be monitored is detected outside the range of a surrounding image displayed on the display device. An excavator (100) according to one embodiment of the present invention comprises: a camera (40) for photographing the surroundings of the excavator (100); a detection unit (304) for detecting a predetermined object around the excavator (100); and a display device (50) for displaying a surrounding image representing the situation around the excavator (100), such as a through image or a viewpoint conversion image, based on the image photographed by the camera (40), and when the predetermined object is detected outside the range of the displayed surrounding image by the detection unit (304), the display device (50) displays the situation that the predetermined object is detected outside the range of the displayed surrounding image by the detection unit (304).

Description

Technical Field

[0001] The present invention relates to a display device and the like. Background Art

[0002] There is known an excavator having a detection function for detecting an object (e.g., a person) as a monitoring target around the excavator and a display function for displaying a surrounding image representing the state around the excavator on a display device in a cab based on an image captured by a camera device (see Patent Document 1).

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2014-181508 Summary of the invention

[0006] Technical issues to be solved by the invention

[0007] However, if an object to be monitored is detected outside the range of the surrounding image displayed on the display device, the object to be monitored may not be displayed on the display device even though the object to be monitored is detected. Therefore, in this case, the operator in the cab may not be able to understand at which position around the excavator the object to be monitored is detected.

[0008] Therefore, in view of the above problems, an object of the present invention is to provide a technology that allows an operator of a shovel or the like to easily understand the position of the detected object even when the object to be monitored is detected outside the range of a surrounding image displayed on a display device.

[0009] Means for solving technical problems

[0010] In order to achieve the above-mentioned purpose, in one embodiment of the present invention, a display device is provided, which displays a surrounding image representing the conditions around the excavator based on an image captured by a camera device that captures the surroundings of the excavator, and when a specified object is detected outside the range of the displayed surrounding image, the display device displays the situation that the specified object is detected outside the range of the displayed surrounding image.

[0011] Furthermore, in another embodiment of the present invention, there is provided a shovel including the above-mentioned display device inside a cab.

[0012] Furthermore, in another embodiment of the present invention, there is provided an information processing device comprising:

[0013] the display device; and

[0014] An operating device is used for remotely operating the excavator.

[0015] Effects of the Invention

[0016] According to the above embodiment, it is possible to provide a technique that allows an operator of a shovel or the like to easily understand the position where the detected object exists even when an object to be monitored is detected outside the range of a surrounding image displayed on a display device. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a diagram showing an example of a shovel management system.

[0018] Figure 2 This is a top view of an excavator.

[0019] Figure 3 This is a block diagram showing an example of the structure of a surrounding monitoring device.

[0020] Figure 4 This is a diagram showing an example of a circuit configuration for sounding a horn.

[0021] Figure 5 This is a diagram showing an example of a circuit configuration for sounding a walking alarm.

[0022] Figure 6 This is a diagram showing an example of a monitoring image (through image) displayed on a display device.

[0023] Figure 7 This is a diagram showing another example (bird's-eye view image) of the monitoring image displayed on the display device.

[0024] Figure 8 This is a diagram showing an example of a displayable area and a detectable area based on an image captured by an imaging device.

[0025] Fig. 9 It is a diagram showing a first example of the detectable area of ​​the detection unit.

[0026] Fig.10 This is a diagram showing a specific example of a monitoring image displayed on a display device when a monitoring target is detected by a detection unit.

[0027] Fig.11 It is a diagram showing a second example of the detectable area of ​​the detection unit.

[0028] Fig.12 This is a diagram showing a first example of workers at a work site.

[0029] Fig.13 This is a diagram showing a second example of workers at a work site.

[0030] Fig.14 This is a diagram showing a third example of workers at a work site.

[0031] Fig.15 This is a diagram showing a fourth example of workers at a work site.

[0032] Fig.16 This diagram shows the operating range when optional equipment is installed.

[0033] Fig.17 It is a diagram for explaining a method of determining whether or not there is contact with a monitoring object around the shovel.

[0034] Fig.18 This is a diagram for explaining the motion control method of the motion restriction function.

[0035] Fig.19 This is a diagram for explaining the motion control method of the motion restriction function.

[0036] Fig. 20 This is a diagram showing a specific example of a monitoring image when a monitoring target is detected by a detection unit.

[0037] Fig.21 This is a diagram showing a specific example of a monitoring image when a monitoring target is detected by a detection unit.

[0038] Fig. 22 This is a diagram showing a specific example of a monitoring image when a monitoring target is detected by a detection unit.

[0039] Fig.23 This is a diagram showing an example of the configuration of a work site safety management system. DETAILED DESCRIPTION

[0040] Hereinafter, embodiments will be described with reference to the drawings.

[0041] [Overview of Excavator Management System]

[0042] First, refer to Figure 1 , Figure 2 The outline of the shovel management system 1000 according to the present embodiment will be described.

[0043] Figure 1 FIG. 1 is a diagram showing an example of a shovel management system 1000 according to the present embodiment. Figure 1 , a side view of the excavator 100 is shown. And, Figure 2 It is a top view of the shovel 100 .

[0044] The shovel management system 1000 includes a shovel 100 , a management device 200 , and a terminal device 300 .

[0045] The shovel management system 1000 monitors (manages) the operating status and operation status of the shovel 100 using, for example, the management device 200 and the terminal device 300 .

[0046] The shovel management system 1000 may include one or more shovels 100. Furthermore, the shovel management system 1000 may include one or more management devices 200. Furthermore, the shovel management system 1000 may include one or more terminal devices 300.

[0047] <Overview of Excavator>

[0048] The shovel 100 includes a lower traveling body 1 , an upper revolving body 3 rotatably mounted on the lower traveling body 1 via a revolving mechanism 2 , a boom 4 as an attachment, an arm 5 , and a bucket 6 , and a cab 10 .

[0049] The shovel 100 operates driven elements such as the lower traveling body 1 (a pair of left and right crawler tracks 1C), the upper swing body 3 , the boom 4 , the arm 5 , and the bucket 6 in accordance with operations of an operator sitting in the cab 10 .

[0050] Furthermore, the shovel 100 may be configured to be remotely operated (remotely operated) from the outside of the shovel 100 instead of being configured to be operated by an operator sitting in the cab 10, or in addition to being configured to be operated by an operator sitting in the cab 10, it may be configured to be remotely operated (remotely operated) from the outside of the shovel 100. When the shovel 100 is remotely operated, the interior of the cab 10 may be unmanned. The following description is based on the premise that the operator's operation includes at least one of the operation of the operating device 26 by the operator in the cab 10 and the remote operation by the external operator.

[0051] Remote operation includes, for example, a method of operating the shovel 100 by inputting operation related to the actuator of the shovel 100 through a predetermined external device. The predetermined external device may be, for example, the management device 200 or the terminal device 300. At this time, the shovel 100 may send image information (captured image) output by, for example, a front camera that captures the front of the upper swing body 3 or a camera device 40 to be described later, to the external device through, for example, the communication device 70 to be described later. Then, the external device may display the received image information (captured image) on a display device (hereinafter referred to as a "remote operation display device") provided in the device. In addition, various information images (information screens) displayed on the display device 50 inside the cab 10 of the shovel 100 may also be displayed on the remote operation display device of the external device in the same manner. Thus, the operator of the external device may remotely operate the shovel 100 while checking the display contents of, for example, the captured image or information screen that shows the state of the shovel 100 surrounding the shovel 100 displayed on the remote operation display device. Then, the excavator 100 can operate the actuator according to the remote operation signal indicating the content of the remote operation received from the external device through the communication device 70, thereby driving the driven elements such as the lower walking body 1 (left and right crawlers 1C), the upper rotating body 3, the boom 4, the dipper 5 and the bucket 6.

[0052] Furthermore, the remote operation may include, for example, a method of operating the shovel 100 based on voice input, gesture input, etc. from outside the shovel 100 by people around the shovel 100 (e.g., workers). Specifically, the shovel 100 recognizes voices made by workers around the shovel 100, gestures made by workers, etc., through a voice input device (e.g., microphone), a gesture input device (e.g., camera), etc. mounted on the shovel 100 (machine). Then, the shovel 100 can operate the actuator according to the content of the recognized voice, gesture, etc., thereby driving the driven elements such as the lower traveling body 1 (left and right crawler tracks 1C), the upper swing body 3, the boom 4, the arm 5, and the bucket 6.

[0053] Furthermore, the excavator 100 can automatically operate the actuator regardless of the content of the operator's operation. Thus, the excavator 100 realizes a function of automatically operating at least a part of the driven elements such as the lower traveling body 1 (left and right crawler tracks 1C), the upper swing body 3, the boom 4, the arm 5 and the bucket 6 (so-called "automatic operation function" or "machine control function").

[0054] The automatic operation function may include a function of automatically operating a driven element (actuator) other than the driven element (actuator) as the operation object in accordance with the operator's operation on the operating device 26 or remote operation (so-called "semi-automatic operation function"). In addition, the automatic operation function may include a function of automatically operating at least a part of the plurality of driven elements (actuators) under the premise that there is no operation or remote operation on the operating device 26 by the operator (so-called "full automatic operation function"). In the shovel 100, when the full automatic operation function is effective, the interior of the cab 10 may be unmanned. In addition, the semi-automatic operation function or the full automatic operation function may include a method of automatically determining according to a rule that predetermines the operation content of the driven element (actuator) as the object of the automatic operation. In addition, the semi-automatic operation function or the full automatic operation function may also include a method (so-called "autonomous operation function") in which the shovel 100 autonomously performs various judgments and autonomously determines the operation content of the driven element (actuator) as the object of the automatic operation according to the judgment results.

[0055] The lower traveling body 1 includes, for example, a pair of left and right crawler tracks 1C (a left crawler track 1CL and a right crawler track 1CR), and the crawler tracks 1CL and 1CR are hydraulically driven by corresponding traveling hydraulic motors 1M to automatically travel.

[0056] The upper revolving body 3 is hydraulically driven by the revolving hydraulic motor 2A through the revolving mechanism 2 to be revolved relative to the lower traveling body 1 .

[0057] The upper rotating body 3 has an imaging device 40 and a surrounding information acquisition device 45 mounted on its upper surface.

[0058] The upper rotating body 3 is equipped with a power source of the excavator 100. The power source of the excavator 100 includes, for example, an engine 11 (e.g., a diesel engine, etc.) operated by a predetermined fuel (e.g., diesel). In addition, the power source of the excavator 100 may include, instead of the engine 11, an electric motor operated by electric power supplied from an electric storage device (e.g., a capacitor or a lithium-ion battery, etc.) or an external power source connected via a cable, or in addition to the engine 11, an electric motor operated by electric power supplied from an electric storage device (e.g., a capacitor or a lithium-ion battery, etc.) or an external power source connected via a cable.

[0059] Furthermore, the upper slewing body 3 is equipped with various hydraulic devices such as a main pump 14 , a pilot pump 15 , and a control valve 17 .

[0060] The main pump 14 is driven by a power source such as the engine 11 or an electric motor, and supplies hydraulic oil to various hydraulic actuators under the control of the controller 30. The hydraulic actuators include the boom cylinder 7, the arm cylinder 8, and the bucket cylinder 9 described below in addition to the travel hydraulic motor 1M and the swing hydraulic motor 2A.

[0061] The pilot pump 15 is driven by a power source such as the engine 11 or an electric motor, and supplies hydraulic oil to various hydraulic machines of a hydraulic pilot type (for example, the operating device 26 or the control valve 17 ).

[0062] The control valve 17 selectively supplies the working oil discharged from the main pump 14 to each hydraulic actuator according to the operating state of the driven element (i.e., the corresponding hydraulic actuator), thereby adjusting the flow rate and flow direction of the working oil supplied to the hydraulic actuator. For example, the control valve 17 can be composed of a plurality of control valves (directional switching valves) that control the direction and flow rate of the working oil supplied to each hydraulic actuator. The control valve 17 is, for example, a hydraulically driven type (hydraulic pilot type), and a pilot pressure corresponding to the operation content of each hydraulic actuator or an operation instruction corresponding to the automatic operation function is input. Thus, the control valve (directional switching valve) corresponding to each hydraulic actuator is driven according to the input pilot pressure. In addition, the control valve 17 can also be an electrically driven type such as an electromagnetic solenoid type, and an electrical signal corresponding to the operation content of the operating device 26 or an operation instruction corresponding to the automatic operation function is input. Thus, the control valve (directional switching valve) corresponding to each hydraulic actuator is driven according to the input electrical signal.

[0063] The boom 4 is installed (pivoted) in a pitching manner at the front center of the upper rotating body 3, and the front end of the boom 4 is installed (pivoted) with a dipper arm 5 so as to be rotatable up and down, and the front end of the dipper arm 5 is installed (pivoted) with a bucket 6 so as to be rotatable up and down.

[0064] The bucket 6 is an example of an end attachment, and is mounted on the front end of the arm 5 in a manner that can be appropriately replaced according to the work content of the excavator 100. That is, a bucket different from the bucket 6 can be mounted on the front end of the arm 5 instead of the bucket 6, for example, a relatively large bucket, a bucket for slopes, a bucket for dredging, etc. In addition, an end attachment other than a bucket can be mounted on the front end of the arm 5, for example, a mixer, a crusher, a pulverizer, etc. In addition, a spare attachment such as a quick coupling or a tilt rotator can be interposed between the arm 5 and the end attachment.

[0065] The boom 4, the arm 5 and the bucket 6 are hydraulically driven by a boom cylinder 7, an arm cylinder 8 and a bucket cylinder 9, respectively.

[0066] The cab 10 is a control room in which an operator sits to operate the shovel 100 , and is mounted on the front left side of the upper swing body 3 , for example.

[0067] Inside the cab 10 , for example, an operating device 26 , a controller 30 , a display device 50 , an input device 52 , a sound output device 54 , etc. are provided. Also, on the upper surface of the cab 10 , for example, a communication device 70 is provided.

[0068] The operating device 26 is used to operate the driven elements driven by actuators (specifically, hydraulic actuators) such as the lower traveling body 1, the upper rotating body 3, and the attachments (the boom 4, the arm 5, and the bucket 6). In other words, the operating device 26 is used to operate the hydraulic actuators (the traveling hydraulic motor 1M, the rotating hydraulic motor 2A, the boom cylinder 7, the arm cylinder 8, and the bucket cylinder 9, etc., which correspond to the crawler 1CL and 1CR, respectively). The operating device 26 includes, for example, the driven elements, that is, the rod device or the pedal device corresponding to the hydraulic actuator.

[0069] The operating device 26 is, for example, a hydraulic pilot type. At this time, the operating device 26 uses the hydraulic oil supplied from the pilot pump 15 to output a pilot pressure corresponding to the operation content (for example, the operation direction and the operation amount, etc.) related to each driven element (that is, the corresponding hydraulic actuator) to the control valve 17. As a result, the control valve 17 can realize the operation of each driven element (that is, the corresponding hydraulic actuator) corresponding to the operation content of the operating device 26.

[0070] Furthermore, the operating device 26 may be, for example, an electric type. At this time, the operating device 26 outputs an electrical signal (hereinafter referred to as an "operation signal") corresponding to the operation content of each driven element (i.e., the corresponding hydraulic actuator) to the controller 30. Then, the controller 30 outputs a control instruction corresponding to the operation signal to an operating hydraulic control valve (e.g., a hydraulic control valve 56 described later) provided in the oil circuit (pilot line) between the pilot pump 15 and the control valve 17. Thus, the operating hydraulic control valve can use the working oil supplied from the pilot pump 15 to make the pilot pressure corresponding to the operation signal (i.e., the pilot pressure corresponding to the operation content related to each driven element (i.e., each hydraulic actuator) in the operating device 26) act on the control valve 17. Therefore, the control valve 17 can realize the action of each driven element (i.e., each corresponding hydraulic actuator) corresponding to the operation content of the operating device 26.

[0071] Furthermore, as described above, the driven element of the excavator 100 (i.e., the corresponding actuator (hydraulic actuator) can be remotely operated. For example, a signal (remote operation signal) indicating the content of the remote operation is sent from a predetermined external device to the excavator 100, and the controller 30 receives the remote operation signal through the communication device 70. Then, the controller 30 outputs a control instruction corresponding to the content of the remote operation specified by the remote operation signal (e.g., the driven element or hydraulic actuator as the operation object, the operation direction and the operation amount, etc.) to the operating hydraulic control valve. As a result, the operating hydraulic control valve can use the working oil supplied from the pilot pump 15 to make the pilot pressure corresponding to the content of the remote operation act on the hydraulically driven control valve 17. Therefore, the control valve 17 can realize the action of each driven element (i.e., the corresponding hydraulic actuator) corresponding to the content of the remote operation.

[0072] In addition, the shovel 100 may replace part or all of the various hydraulic actuators with electric actuators. That is, the shovel 100 may be a hybrid shovel or an electric shovel. In this case, the controller 30 may output control instructions corresponding to the operation content of the operating device 26, the content of the remote operation specified by the remote operation signal, the content of the operation instruction corresponding to the automatic operation function, etc. to the driver or the like that drives the electric actuator or the electric actuator.

[0073] Furthermore, when the door lock bar of the cab 10 is in an upright state (hereinafter referred to as a "locked state"), the operation of the operating device 26, the remote operation, and the operation command corresponding to the automatic operation function are invalidated, and the excavator 100 does not operate. On the other hand, when the door lock bar is in a lowered state (hereinafter referred to as a "released state"), the operation of the operating device 26, the remote operation, and the operation command corresponding to the automatic operation function become valid, and the excavator 100 operates. Thus, for example, the following situation can be avoided: when the operator sits on the operator's seat or leaves the operator's seat to the outside of the cab 10 in the state where the door lock bar is upright, the operator's body touches the operating device 26, causing the excavator 100 to operate. Furthermore, by, for example, lowering the door lock bar, the operator can start operating the excavator 100.

[0074] For example, the operating state of the door lock rod is linked to the open / closed state of the door lock valve at the most upstream of the pilot line that supplies the pilot pressure from the pilot pump 15 to various hydraulic machines (for example, the hydraulic pilot operating device 26 or the operating hydraulic control valve). Specifically, when the door lock rod is in the released state, the door lock valve maintains the open valve state (connected state), so that the pilot pressure is supplied from the pilot pump 15 to the operating device 26 or the operating hydraulic control valve. Therefore, the operating device 26 or the operating hydraulic control valve can supply the pilot pressure corresponding to the operator's operation to the control valve 17, so that the hydraulic actuator works. On the other hand, when the door lock rod is in the locked state, the door lock valve maintains the closed valve state (cut-off state), so that the supply of the pilot pressure supplied from the pilot pump 15 to the operating device 26 or the operating hydraulic control valve is cut off. Therefore, the operating device 26 or the operating hydraulic control valve cannot supply the pilot pressure corresponding to the operator's operation to the control valve 17, so that the operator's operation or the operation instruction corresponding to the automatic operation function can be invalidated.

[0075] Furthermore, as described above, it is sometimes possible to replace part or all of the hydraulic actuator with an electric actuator. In this case, for example, when the door lock rod is in a locked state, the controller 30 may not output a control instruction corresponding to the operator's operation or the operation instruction of the automatic operation function to the electric actuator or the driver. Thus, the operator's operation or the operation instruction corresponding to the automatic operation function can be invalidated according to the locked state of the door lock rod.

[0076] The communication device 70 communicates with the outside of the shovel 100 (for example, the management device 200 or the terminal device 300 ) through the communication line NW.

[0077] The communication line NW includes, for example, a wide area network (WAN). The wide area network may include, for example, a mobile communication network with a base station as a terminal. Furthermore, the wide area network may also include, for example, a satellite communication network using a communication satellite. Furthermore, the wide area network may also include, for example, the Internet. Furthermore, the communication line NW includes, for example, a local area network (LAN) inside a facility where the management device 200 is installed. The local area network may be wired, wireless, or a combination of the two. Furthermore, the communication line NW may also include, for example, a wireless short-distance communication line such as WiFi or Bluetooth (registered trademark).

[0078] The shovel 100 communicates with the management device 200 using, for example, the communication device 70. Thus, the shovel 100 can transmit data related to the shovel 100 (the shovel itself) to the management device 200 or receive data related to control of the shovel 100 (the shovel itself).

[0079] Furthermore, the shovel 100 may communicate with the terminal device 300 using, for example, the communication device 70. In this case, the shovel 100 may communicate with the terminal device 300 indirectly via the management device 200, or may communicate with the terminal device 300 directly.

[0080] <Overview of management device>

[0081] The management device 200 (an example of an information processing device) is provided outside the shovel 100 , and manages, for example, the operating state or operation state of the shovel 100 . Furthermore, the management device 200 may support remote operation of the shovel 100 .

[0082] The management device 200 is, for example, a cloud server such as a management center installed outside the working site of the excavator 100. Furthermore, the management device 200 may also be, for example, an edge server such as a temporary office installed in the working site of the excavator 100, a station building near the working site, a base station, etc. Furthermore, the management device 200 may also be, for example, a fixed terminal device (fixed terminal) or a portable terminal device (portable terminal) configured in a temporary office, etc., in the working site of the excavator 100. The fixed terminal may include, for example, a desktop computer terminal. Furthermore, the portable terminal may include, for example, a mobile phone, a smart phone, a tablet terminal, a laptop computer terminal, etc.

[0083] The management device 200 includes a control device 210 , a communication device 220 , an output device 230 , and an input device 240 .

[0084] The control device 210 performs control related to the management device 200. The functions of the control device 210 can be realized by any hardware or a combination of any hardware and software. The control device 210 is composed of, for example, a computer including a CPU (Central Processing Unit), a memory device such as a RAM (Random Access Memory), an auxiliary storage device such as a ROM (Read Only Memory), and an interface device for inputting from or outputting to the outside. The control device 310 described below can also have the same structure.

[0085] The communication device 220 communicates with the outside of the management device 200 (for example, the shovel 100 or the terminal device 300 ) and the like through the communication line NW.

[0086] The output device 230 (an example of a display device) outputs information to a user such as an administrator or an operator of the management device 200 (hereinafter referred to as a "management device user"). The output device 230 may include, for example, a display device or a lighting device that outputs visual information. The display device includes, for example, a liquid crystal display or an organic EL (Electroluminescence) display that outputs image information. Furthermore, the display device may include the above-mentioned display device for remote operation. Furthermore, the output device 230 may also include, for example, a sound output device that outputs auditory information. The sound output device includes, for example, a speaker or a buzzer.

[0087] The input device 240 (an example of an operating device) accepts various inputs from the user of the management device, and a signal corresponding to the input content is input to the control device 210. The input device 240 includes, for example, an operating input device that accepts operating inputs from the user of the management device. The operating input device may include, for example, a mouse, a keyboard, a touch panel, a button, a switch key, a lever, etc. Furthermore, the input device 240 may also include, for example, a sound input device and a gesture input device that accept voice input and gesture input from the user of the management device. The sound input device includes, for example, a microphone that obtains sound data emitted by the user of the management device. The gesture input device includes, for example, a camera (camera) that captures the state of the gesture of the user of the management device. Furthermore, the input device 240 may also include, for example, an operating device for remote operation.

[0088] The control device 210 communicates with the shovel 100 and the terminal device 300 using the communication device 220. Thus, the management device 200 can, for example, receive various data sent (uploaded) from the shovel 100, thereby collecting various data related to the shovel 100. Furthermore, the control device 210 can, for example, use the communication device 220 to send data related to the control of the shovel 100 to the shovel 100, thereby controlling the shovel 100 from the outside. Furthermore, the control device 210 can, for example, use the communication device 220 to provide various data to the terminal device 300 in response to a request from the terminal device 300. Furthermore, the control device 210 can, for example, use the communication device 220 to send a signal (remote operation signal) indicating the content of the remote operation received from the input device 240 (remote operation operation device) to the shovel 100 that is the object of remote operation. Thus, the management device 200 can support the remote operation of the shovel 100.

[0089] <Outline of terminal device>

[0090] The terminal device 300 (an example of an information processing device) is, for example, a terminal device (user terminal) used by a user who receives information provision in the shovel management system 1000. Furthermore, the terminal device 300 may also support remote operation of the shovel 100.

[0091] The terminal device 300 is, for example, a general-purpose portable terminal such as a laptop computer terminal, a tablet terminal, or a smart phone held by a user. In addition, the terminal device 300 may also be a general-purpose fixed terminal such as a desktop computer. In addition, the terminal device 300 may also be a dedicated terminal device (a portable terminal or a fixed terminal) for receiving data (information) related to the shovel 100 or supporting remote operation.

[0092] The terminal device 300 includes a control device 310 , a communication device 320 , an output device 330 , and an input device 340 .

[0093] The control device 310 performs control related to the terminal device 300 .

[0094] The communication device 320 communicates with the outside of the terminal device 300 (for example, the shovel 100 or the management device 200 ) and the like through the communication line NW.

[0095] The output device 330 (an example of a display device) outputs information to a user such as an administrator or an operator of the terminal device 300 (hereinafter referred to as a "terminal device user"). The output device 330 may include, for example, a display device or a lighting device that outputs visual information. The display device may include, for example, a liquid crystal display or an organic EL display that outputs image information. In addition, the display device may include the above-mentioned display device for remote operation. In addition, the output device 330 may also include, for example, a sound output device that outputs auditory information. The sound output device may include, for example, a speaker or a buzzer.

[0096] The input device 340 (an example of an operating device) accepts various inputs from the user of the terminal device, and a signal corresponding to the input content is input to the control device 310. The input device 340 includes, for example, an operating input device that accepts operating inputs from the user of the terminal device. The operating input device may include, for example, a mouse, a keyboard, a touch panel, a button, a switch key, a lever, etc. Furthermore, the input device 340 may also include, for example, a sound input device and a gesture input device that accept voice input and gesture input from the user of the terminal device. The sound input device includes, for example, a microphone that obtains sound data emitted by the user of the terminal device. The gesture input device includes, for example, a camera (camera) that captures the state of the gestures of the user of the terminal device. Furthermore, the input device 340 may also include, for example, an operating device for remote operation.

[0097] The control device 310 communicates with the management device 200 using the communication device 320. Thus, the terminal device 300 can request the management device 200 to provide data related to the shovel 100. In addition, the terminal device 300 can receive data related to the shovel 100 sent from the management device 200, and provide information related to the shovel 100 to the user through the output device 330 (display device).

[0098] Furthermore, the control device 310 may also communicate with the shovel 100 using the communication device 320. In this case, the terminal device 300 may communicate with the shovel 100 indirectly via the management device 200, or may communicate with the shovel 100 directly. For example, the control device 310 may use the communication device 320 to send a signal (remote operation signal) indicating the content of the remote operation received from the input device 340 (remote operation operation device) to the shovel 100 that is the object of remote operation. Thus, the terminal device 300 can support the remote operation of the shovel 100.

[0099] [Structure of the perimeter monitoring device]

[0100] Next, except Figure 1 , Figure 2 In addition, refer to Figure 3 to Figure 7 The configuration of the periphery monitoring device 150 mounted on the shovel 100 according to the present embodiment will be described.

[0101] Figure 3 to Figure 5 1 is a diagram for explaining the structure of the surrounding monitoring device 150 according to the present embodiment. Specifically, Figure 3 This is a block diagram showing an example of the configuration of the surrounding area monitoring device 150 according to the present embodiment. Figure 4 This is a diagram showing an example of a circuit configuration for sounding the horn 54a. Figure 5 This is a diagram showing an example of a circuit configuration for sounding the walking alarm 54b.

[0102] The perimeter monitoring device 150 monitors a predetermined object as a monitoring target (hereinafter, simply referred to as a "monitoring target") entering a predetermined range around the shovel 100. Then, when the monitoring target is detected within the predetermined range around the shovel 100, the perimeter monitoring device 150 activates a safety function for ensuring safety around the shovel 100.

[0103] The safety function may include, for example, a notification function that outputs an alarm or the like to at least one of the inside of the cab 10, the outside of the cab 10, and the remote operator, manager, etc. of the shovel 100 to notify that the monitoring object is detected. Thus, it is possible to prompt the operator inside the cab 10, the workers around the shovel 100, the operator, manager, etc. who remotely operates the shovel 100 to pay attention to the existence of the monitoring object within a predetermined range around the shovel 100. Hereinafter, the notification function to the inside of the cab 10 (i.e., the operator, etc.) is sometimes referred to as an "internal notification function". In addition, the notification function to the outside of the cab 10 (i.e., the workers around the shovel 100, etc.) is sometimes referred to as an "external notification function". In addition, the notification function to the remote operator, manager, etc. of the cab 10 is sometimes referred to as a "remote notification function". In addition, the safety function may also include, for example, an action restriction function that restricts the operation of the operating device 26 and the action of the remotely operated shovel 100.

[0104] The motion limiting function includes at least one of a motion deceleration function and a motion stop function. The motion deceleration function makes the motion speed of the excavator 100 slower than the normal motion speed in response to the operation of the operating device 26, remote operation, or an operation instruction corresponding to the automatic operation function. The motion stop function stops the motion of the excavator 100 regardless of the operation of the operating device 26, remote operation, or an operation instruction corresponding to the automatic operation function, and maintains the stopped state.

[0105] The monitoring object may include workers working around the excavator 100 or supervisors at the work site. Furthermore, the monitoring object may include any object (i.e., obstacle) other than people, such as fixed and immovable obstacles such as materials temporarily placed at the work site, temporary offices at the work site, and moving obstacles such as vehicles including trucks. In the following, in this embodiment, the description will be continued with the case where the monitoring object is a person.

[0106] like Figure 3 As shown, the surrounding monitoring device 150 includes a controller 30, an operation information output device 29, a camera device 40, a surrounding information acquisition device 45, a display device 50, an input device 52, a sound output device 54, a hydraulic control valve 56 and a communication device 70.

[0107] The controller 30 is a control device that performs control related to the functions of the surrounding monitoring device 150. The controller 30 is mounted in the cab 10, for example.

[0108] The functions of the controller 30 can be realized by any hardware or any combination of hardware and software. The controller 30 is, for example, composed of a computer including a CPU, a memory device (main storage device) such as a RAM, an auxiliary storage device such as a ROM, and an interface device for inputting from or outputting to the outside. The controller 30 includes, for example, a display processing unit 301, a setting unit 302, a detection unit 304, and a safety function control unit 305 as functional units that are realized by executing one or more programs installed in the auxiliary storage device on the CPU. In addition, the controller 30 uses a storage unit 303. The storage unit 303 can be realized by, for example, an auxiliary storage device or an external storage device connected in a manner that can communicate with the controller 30.

[0109] In addition, part or all of the functions of the controller 30 may be implemented by other controllers. That is, the functions of the peripheral monitoring device 150 may be implemented by sharing them among a plurality of controllers. Furthermore, the controller 30 may also perform controls related to the excavator 100 other than the functions of the peripheral monitoring device 150. That is, the controller 30 may be a dedicated control device that enhances the functions of the peripheral monitoring device 150, or a general-purpose control device that performs controls related to various functions of the excavator 100 including the functions of the peripheral monitoring device 150. Furthermore, part or all of the functions of the controller 30 may be transferred to the outside of the excavator 100 (for example, the management device 200). In this case, the excavator 100 may, for example, control its actions in real time based on control instructions from an external device such as the management device 200. This is because, for example, in an environment such as 5G (5 th In the case of a mobile communication network such as the fifth generation (generation): capable of performing extremely high-speed communication via the communication line NW, the operation control of the shovel 100 without delay can also be achieved by sequentially sending control commands from the management device 200 to the shovel 100.

[0110] The operation information output device 29 outputs information (hereinafter referred to as "operation information") related to the operation content of the operating device 26, the content of remote operation or the content of operation instructions corresponding to the automatic operation function (that is, the operation content related to each driven element (that is, the corresponding hydraulic actuator)).

[0111] The operation information output device 29 may be, for example, a sensor that acquires information related to the operation content of the operating device 26 (hereinafter referred to as an "operation information acquisition sensor"). The operation information acquisition sensor may be, for example, a linear encoder that senses the operation direction or operation amount of a lever or pedal of the operating device 26. Furthermore, the operation information acquisition sensor may be, for example, a pressure sensor that senses the pilot pressure on the secondary side of the hydraulic pilot operating device 26. Furthermore, the operation information output device 29 may be, for example, an electric operating device 26. This is because the operation signal output from the electric operating device 26 is equivalent to the operation information. Furthermore, in the case of remotely operating the excavator 100, the operation information output device 29 may be, for example, a communication device 70 that receives a remote operation signal from an external device. Furthermore, in the case of the excavator 100 being operated by an automatic operation function, the operation information output device 29 may be, for example, a computing device that outputs an operation instruction.

[0112] The camera device 40 is installed on the upper part of the upper rotating body 3, and captures the surroundings of the excavator 100 from an area relatively close to the excavator 100 to an area relatively far from the excavator 100, and outputs the captured images. The camera device 40 includes cameras 40B, 40L, and 40R. Hereinafter, the cameras 40B, 40L, and 40R are sometimes collectively referred to as "cameras 40X".

[0113] The camera 40B, the camera 40L, and the camera 40R are respectively installed on the upper rear end, the upper left end, and the upper right end of the upper rotating body 3 to shoot the rear, left, and right sides of the upper rotating body 3. For example, the camera 40X is a monocular camera (i.e., a wide-angle camera) with an extremely wide viewing angle. And, for example, the camera 40X may also be a stereo camera or a depth camera, etc. The camera 40B shoots a shooting range at the rear of the upper rotating body 3, for example, a shooting range in the horizontal direction (i.e., the circumferential direction when viewed from the excavator 100) extending from the left rear to the right rear. And, the camera 40L, for example, shoots a shooting range on the left side of the upper rotating body 3, for example, a shooting range in the horizontal direction (i.e., the circumferential direction when viewed from the excavator 100) extending from the left front of the upper rotating body 3 to the left rear. The camera 40R captures, for example, a shooting range on the right side of the upper swing body 3, for example, a shooting range in the horizontal direction (circumferential direction when viewed from the shovel 100) extending from the right front to the right rear of the upper swing body 3. The camera 40X is installed on the upper part of the upper swing body 3 with its optical axis directed obliquely downward to capture a shooting range in the vertical direction from the ground near the shovel 100 to the distance from the shovel 100.

[0114] The camera 40X outputs captured images at a predetermined period (eg, 1 / 30 second) from the start (ie, key switch on) to the stop (ie, key switch off) of the shovel 100. The captured images output from the camera 40X are input to the controller 30.

[0115] Furthermore, when the shovel 100 is remotely operated, in addition to the imaging device 40 , a front camera for remote operation may be provided to obtain an image for the operator performing the remote operation to check the state in front of the shovel 100 .

[0116] The surrounding information acquisition device 45 is mounted on the upper part of the upper swing body 3 to acquire information about the situation around the shovel 100. The surrounding information acquisition device 45 includes sensors 45BL, 45BR, 45L, and 45R. Hereinafter, the sensors 45BL, 45BR, 45L, and 45R may be collectively referred to as "sensors 45X".

[0117] Sensors 45BL, 45BR, 45L, and 45R are respectively installed at the upper left rear end, upper right rear end, upper left end, and upper right end of the upper rotating body 3 to obtain information related to the conditions of the left rear, right rear, left, and right sides of the upper rotating body 3. For example, sensor 45X is LIDAR (Light Detection and Ranging). And, for example, sensor 45X can be, for example, a millimeter wave radar or an ultrasonic sensor. The following description focuses on the case where sensor 45X is LIDAR.

[0118] The sensor 45X obtains information indicating the conditions around the excavator 100 by, for example, irradiating infrared light in a certain direction and receiving reflected light (infrared light) from an object in the direction, and specifically obtains information related to the received reflected light (hereinafter referred to as "received light information"). The sensor 45X is, for example, a scanning LIDAR, which is a three-dimensional laser scanner capable of scanning the irradiation direction of infrared laser in the up-down direction and the left-right direction. In addition, the sensor 45X may also be a so-called flash LIDAR that irradiates infrared light in a large three-dimensional range from a light emitting module and captures the reflected light (infrared light) using a three-dimensional range image element.

[0119] The light receiving information includes information related to the time (TOF: Time Of Flight) from irradiating infrared light to receiving reflected light in each irradiation direction of infrared light (hereinafter referred to as "TOF information") and information related to the intensity of the received reflected light in each irradiation direction of infrared light (hereinafter referred to as "light receiving intensity information").

[0120] The sensor 45BL is configured to be able to irradiate infrared rays within an irradiation range on the left rear side of the upper rotating body 3 (for example, an irradiation range in the horizontal direction (i.e., the circumferential direction when viewed from the excavator 100) extending from the left rear side of the upper rotating body 3 to the rear). Furthermore, the sensor 45BR is configured to be able to irradiate infrared rays within an irradiation range on the right rear side of the upper rotating body 3 (for example, an irradiation range in the horizontal direction (i.e., the circumferential direction when viewed from the excavator 100) extending from the right rear side of the upper rotating body 3 to the rear). Furthermore, the sensor 45L is configured to be able to irradiate infrared rays within an irradiation range on the left side of the upper rotating body 3 (for example, an irradiation range in the horizontal direction (i.e., the circumferential direction when viewed from the excavator 100) extending from the left front side of the upper rotating body 3 to the left rear). The sensor 45R is configured to be able to irradiate infrared rays within an irradiation range on the right side of the upper revolving body 3 (for example, an irradiation range in the horizontal direction (circumferential direction when viewed from the excavator) extending from the right front to the right rear of the upper revolving body 3). The sensor 45X is installed on the upper part of the upper revolving body 3 so that the optical axis (that is, the reference axis of the irradiation direction of the infrared rays) faces obliquely downward, and has an irradiation range of infrared rays in the vertical direction centered on a portion of the ground relatively close to the excavator 100.

[0121] The sensor 45X outputs light reception information at a predetermined cycle during a period from the start to the stop of the shovel 100. The light reception information output from the sensor 45X is input to the controller 30.

[0122] The display device 50 is provided around the operator's seat in the cab 10 (specifically, a position that is easily visually recognized by the operator sitting in the operator's seat) to display various image information to be notified to the operator. The display device 50 may be, for example, a liquid crystal display or an organic EL display, and is a touch panel type that also serves as the input device 52. As described later, the display device 50 displays an image (hereinafter referred to as a "monitoring image") showing the state of the surroundings of the excavator 100 (machine) based on an image captured by the camera 40, for example, under the control of the controller 30 (display processing unit 301).

[0123] The input device 52 receives various inputs related to the functions of the surrounding monitoring device 150 from the operator and outputs them to the controller 30. The input device 52 includes, for example, any hardware operation input device such as a touch panel, a touch pad, a button, a switch key, a knob, etc. In addition, the input device 52 may also include, for example, a software operation input unit that can be operated by a hardware operation unit, such as a virtual button icon displayed on the operation screen of the display device 50. In addition, the input device 52 may also include a voice input device and a gesture input device that receive voice input and gesture input from a user such as an operator.

[0124] The sound output device 54 outputs sound toward at least one of the inside and outside of the cab 10. The sound output device 54 may include, for example, a speaker or a buzzer provided inside the cab 10, and may output sound toward the operator. Furthermore, the sound output device 54 may also include, for example, a horn 54a or a travel alarm 54b, and may output sound toward the outside of the cab 10 (specifically, around the excavator 100).

[0125] The horn 54 a is mounted on, for example, the front portion of the upper revolving body 3 .

[0126] like Figure 4 As shown, the horn 54a is sounded by closing a relay 62 provided in a power path between the horn 54a and the battery 60 in response to the on operation of a rotary switch 64 provided in the cab 10. When the operator or the like turns on the rotary switch 64 to sound the horn 54a at the start of travel, the operator can notify the surroundings of the shovel 100 that the shovel 100 (lower traveling body 1) has started traveling.

[0127] The travel alarm 54 b is mounted on, for example, the rear portion of the upper swing body 3 .

[0128] like Figure 5 As shown, the travel alarm 54b outputs a specified sound (for example, a buzzer sound that repeatedly sounds "beep, beep, beep, ...") with a specified pattern when the lower traveling body 1 is traveling (ie, when the crawler belt 1C is operated) under the control of the controller 30.

[0129] Return to Figure 3 The hydraulic control valve 56 is provided in the pilot line connecting the pilot pump 15 and the control valve 17 (specifically, the pilot port of the control valve corresponding to each hydraulic actuator). The hydraulic control valve 56 can be operated by the control command from the controller 30 and adjust the pilot pressure acting on the control valve 17. The hydraulic control valve 56 can be provided in the pilot line between the pilot pump 15 and the hydraulic pilot type operating device 26, that is, the pilot line on the primary side of the operating device 26. In addition, the hydraulic control valve 56 can also be provided in the pilot line between the operating device 26 and the control valve 17, that is, the pilot line on the secondary side of the operating device 26. In addition, the hydraulic control valve 56 can also be the above-mentioned hydraulic control valve for operation in the case of the electric operating device 26, in the case of remote operation, in the case of automatic operation function, etc. The hydraulic control valve 56 is, for example, an electromagnetic proportional valve. Specifically, the hydraulic control valve 56 can adjust the pilot pressure acting on the control valve 17 under the control of the controller 30 regardless of the operation content of the operating device 26 or the content of the remote operation. In other words, the controller 30 can realize the automatic operation function by automatically controlling the operation of the shovel 100 using the hydraulic control valve 56 regardless of the operation content of the operating device 26 or the content of the remote operation.

[0130] The display processing unit 301 causes the display device 50 to display a monitoring image (surrounding image) showing the state (condition) around the shovel 100 based on the image captured by the imaging device 40 .

[0131] For example, the display processing unit 301 displays an image captured by at least one of the cameras 40B, 40L, and 40R on the display device 50 as a monitoring image in response to a predetermined operation on the input device 52. That is, the display processing unit 301 may display all images captured by the cameras 40B, 40L, and 40R or images captured by two cameras side by side on the display device 50, or may display an image captured by one camera on the display device 50. Hereinafter, the captured image displayed on the display device 50 may be referred to as a "through image".

[0132] The display processing unit 301 can switch which of the images captured by the cameras 40B, 40L, and 40R is displayed on the display device 50 in response to a predetermined operation on the input device 52. Thus, the operator can display a through image in a direction that he or she wants to observe on the display device 50 by operating the input device 52.

[0133] Furthermore, for example, the display processing unit 301 generates a composite image by synthesizing images captured by multiple cameras (at least two cameras among cameras 40B, 40L, and 40R) based on the captured image of the camera device 40, and displays the monitoring image including the composite image on the display device 50.

[0134] Specifically, the display processing unit 301 generates a viewpoint conversion image observed from a virtual viewpoint by performing known viewpoint conversion processing and synthesis processing on the images captured by the cameras 40B, 40L, and 40R in the form of a synthesized image, and displays the viewpoint conversion image on the display device 50. Furthermore, when displaying the synthesized image on the display device 50, the display processing unit 301 displays a shovel image schematically showing the shovel 100 on the display device 50 in order to clearly indicate the relative positional relationship between the imaging range of the camera 40 and the shovel 100. That is, the display processing unit 301 generates a monitoring image including a shovel image and a viewpoint conversion image arranged around the shovel image in accordance with the relative positional relationship between the shovel 100 and the imaging range of the camera 40, and displays the image on the display device 50.

[0135] In addition, the function of the display processing unit 301 may be built into the display device 50 .

[0136] For example, Figure 6 , Figure 7 is a diagram showing a specific example of a monitoring image displayed on the display device 50. Specifically, Figure 6 1 is a diagram showing an example of a monitoring image displayed on the display device 50 (a monitoring image MP1 including a through image). Figure 7 2 is a diagram showing another example of the monitoring image displayed on the display device 50 (a monitoring image MP2 including a viewpoint conversion image).

[0137] like Figure 6 As shown, in this example, the image captured by the camera 40B (through image) is displayed as the monitoring image MP1 on the display device 50. Thus, the operator can understand the situation around the shovel 100 (in this example, behind the upper swing body 3) (for example, the presence or absence of monitoring objects such as people around the shovel 100).

[0138] And, if Figure 7 As shown, the display device 50 displays a monitoring image MP2 including a shovel image CG and a viewpoint conversion image EP arranged around the shovel image CG. This allows the operator to appropriately understand the positional relationship between the shovel 100 and surrounding objects shown in the viewpoint conversion image EP.

[0139] Furthermore, a line LN1 at a certain distance from the shovel 100 is superimposed and displayed in the viewpoint conversion image EP of the monitoring image MP2. The line LN1 may represent, for example, the outer edge of the monitoring area of ​​the monitoring object monitored by the surrounding monitoring device 150 (detection unit 304). Thus, the operator can appropriately grasp the distance relationship between the shovel 100 and the surrounding objects shown in the viewpoint conversion image EP.

[0140] In this example, the viewpoint conversion image EP is composed of a combination of a top view image BVP obtained by observing the surrounding area adjacent to the excavator 100 from directly above and a horizontal image HVP obtained by observing the surrounding area arranged around the top view image BVP from the excavator 100 along the horizontal direction. The viewpoint conversion image EP is obtained as follows: after projecting the images captured by the cameras 40B, 40L, and 40R into the space model, the projected images projected into the space model are projected into another two-dimensional plane. The space model is a projection object of the captured images in the virtual space, and is composed of one or more planes or curved surfaces including planes or curved surfaces other than the plane where the captured images are located.

[0141] Furthermore, the display processing unit 301 can transmit the monitoring image generated based on the image captured by the camera device 40 to the management device 200 or the terminal device 300 via the communication device 70. Thus, the management device 200 can receive the monitoring image from the shovel 100 and display the monitoring image on the output device 230 (for example, a display device for remote operation). Similarly, the terminal device 300 can receive the monitoring image from the shovel 100 and display the monitoring image on the output device 330 (for example, a display device for remote operation). Therefore, the operator of the shovel 100 using the management device 200 or the terminal device 300 can remotely operate the shovel 100 while checking the monitoring image displayed on the output device 230 or the output device 330. Furthermore, the manager of the shovel 100 using the management device 200 or the terminal device 300 can remotely monitor the operating conditions of each of the plurality of shovels 100 operating using the automatic operation function while checking the monitoring image displayed on the output device 230 or the output device 330.

[0142] Return to Figure 3 The setting unit 302 performs various settings related to the surrounding monitoring device 150 in response to a request from an operator or the like (i.e., an operation performed by the operator or the like through the input device 52). For example, the setting unit 302 sets a detection condition related to the detection unit 304 or an operation condition of a safety function related to the safety function control unit 305. The setting contents set by the setting unit 302 are stored (registered) in the storage unit 303.

[0143] The storage unit 303 stores (registers) various information related to the surrounding monitoring device 150 .

[0144] The detection unit 304 detects a monitoring target around the shovel 100 (the upper swing body 3) based on the outputs of the imaging device 40 and the surrounding information acquisition device 45. The detection unit 304 includes a detection unit 304A and a detection unit 304B.

[0145] The detection unit 304A detects the monitoring object in a predetermined monitoring area (hereinafter referred to as "first monitoring area") around the excavator 100 (upper swing body 3) based on the output of the camera 40 (i.e., the image captured by the camera 40). The following description of the detection unit 304 is made on the assumption that the excavator 100 is located on a horizontal plane.

[0146] The detection unit 304A detects the monitored object in the first monitoring area extending in the horizontal direction (hereinafter referred to as the "horizontal direction") when viewed from the shovel 100 (i.e., in the direction along the plane (hereinafter referred to as the "operating plane") on which the shovel 100 operates (the lower traveling body 1 is grounded). Specifically, the detection unit 304A can detect the monitored object in the first monitoring area within a predetermined distance Dth1 (for example, 5 meters) from the shovel 100 (the upper swing body 3) in the horizontal direction.

[0147] For example, the detection unit 304A recognizes the monitoring object in the captured image by arbitrarily applying various known image processing methods or machine learning-based recognizers including artificial intelligence (AI), etc. Furthermore, when the monitoring object is a worker or the like, the detection unit 304A can determine which one of a plurality of pre-registered workers (hereinafter referred to as "registered workers") the recognized person corresponds to.

[0148] Furthermore, the detection unit 304A can determine (infer) the position (e.g., the sole position) (hereinafter referred to as the "actual existence position") of the identified monitoring object (person) reflected in the captured image of the monocular camera device 40 by applying various known methods.

[0149] For example, the detection unit 304A infers the position in the horizontal direction when viewed from the shovel 100 (hereinafter referred to as "horizontal position") based on the size of the monitored object identified on the captured image (for example, the size in the height direction on the captured image). This is because there is a correlation relationship that the farther the monitored object is from the shovel 100, the smaller the size of the monitored object identified on the captured image. Specifically, since there is a range of conceivable sizes of the monitored object (for example, a range of conceivable heights of people), the correlation relationship between the horizontal position of the monitored object when viewed from the shovel 100 and the size on the captured image within the conceivable size range can be predefined. Therefore, the detection unit 304A can infer the actual existing position (horizontal position from the shovel 100) of the identified monitored object based on a map or conversion formula, etc., which indicates the correlation relationship between the size of the monitored object on the captured image and the horizontal position when viewed from the shovel 100 and is pre-stored in an internal memory such as an auxiliary storage device of the controller 30.

[0150] Furthermore, for example, the detection unit 304A can infer the actual location (for example, the sole location) of the monitored object by projecting the captured image on the plane (homography matrix) on the premise that the monitored object exists on the same plane as the shovel 100 (specifically, the lower traveling body 1). In this case, a certain part (a certain point) constituting the captured image is associated with a certain location on the same plane as the shovel 100.

[0151] The detection unit 304B detects the monitoring object in a predetermined monitoring area (hereinafter referred to as "second monitoring area") around the excavator 100 (upper swing body 3) based on the output (i.e., light receiving information) of the surrounding information acquisition device 45. Hereinafter, the first monitoring area and the second monitoring area are sometimes collectively referred to as "monitoring area".

[0152] The detection unit 304B detects the monitored object in the second monitoring area extending in the horizontal direction (i.e., along the direction of the working plane), for example. Specifically, the detection unit 304B can detect the monitored object in the second monitoring area within a prescribed distance Dth2 at a distance D in the horizontal direction from the excavator 100 (upper rotating body 3). The prescribed distances Dth1 and Dth2 may be the same or different. That is, the first monitoring area and the second monitoring area may be the same or different. For example, it may be as follows: the detection unit 304A monitors the monitored object in the first monitoring area including a range relatively far from the excavator 100, and the detection unit 304B monitors the monitored object in the second monitoring area limited to a range relatively closer to the excavator 100 than the first monitoring area.

[0153] The detection unit 304B identifies the existence and position of surrounding objects based on the TOF information in the light receiving information obtained from the surrounding information acquisition device 45. In addition, the detection unit 304B can identify the shape of the object or the size of the object based on the light receiving information (TOF information) corresponding to the reflected light received from multiple irradiation directions, thereby identifying the type of the surrounding objects, thereby determining whether the object corresponds to the monitored object. In addition, the detection unit 304B can also identify the retro-reflectivity or reflectivity of the surrounding objects based on the light intensity information in the light receiving information, thereby identifying the type of the object, thereby determining whether the object corresponds to the monitored object.

[0154] Furthermore, the detection units 304A and 304B can switch their functions between on (validation) and off (invalidation) in accordance with a predetermined operation performed by an operator or the like on the input device 52. In this case, it may be such that when the function of one of the detection units 304A and 304B is turned off (invalidated), the operation of turning off (invalidating) the function of the other is invalidated. In other words, it may be such that only the function of one of the detection units 304A and 304B can be switched to off (invalidated).

[0155] In addition, the detection unit 304 may be configured as follows: instead of outputting two detection results related to the monitoring object based on the detection units 304A and 304B, the outputs of the two devices, the camera device 40 and the surrounding information acquisition device 45, are integrated to output one detection result related to the monitoring object. Furthermore, the detection unit 304 may detect the monitoring object based on only one of the outputs of the camera device 40 and the surrounding information acquisition device 45. Furthermore, part or all of the functions of the detection unit 304A may be built into the camera device 40 (camera 40X). Furthermore, the functions of the detection unit 304B may be built into the surrounding information acquisition device 45 (sensor 45X). For example, the following method may be used: the function of detecting an object based on light receiving information (TOF information and light receiving intensity information) of the detection unit 304B is built into the sensor 45X, and the controller 30 only implements the function of determining whether the object detected by the sensor 45X corresponds to the monitoring object.

[0156] When the monitoring target is detected by the detection unit 304 , the safety function control unit 305 performs control related to the safety function to activate the safety function.

[0157] For example, when the monitoring object is detected by the detection unit 304 within a predetermined range (hereinafter referred to as "notification range") included in the monitoring area, the safety function control unit 305 activates the notification function. The notification range may be the same as the monitoring area, or its outer edge may be set to be relatively closer to the shovel 100 than the monitoring area.

[0158] The safety function control unit 305 controls the sound output device 54 to activate a notification function based on sound (i.e., an auditory method) for at least one of the interior and exterior of the cab 10. At this time, the safety function control unit 305 can change the pitch, sound pressure, timbre, sound cycle when the sound is periodically sounded, sound content, etc. of the output sound according to various conditions.

[0159] Furthermore, the safety function control unit 305 activates, for example, a visual notification function for the interior of the cab 10. Specifically, the safety function control unit 305 can control the display device 50 using the display processing unit 301 to display an image (for example, a reference image) indicating that the monitored object has been detected on the monitoring image displayed on the display device 50. Figure 20 to Figure 22). Furthermore, the safety function control unit 305 may also emphasize the monitoring object shown in the monitoring image displayed on the display device 50 or the position on the monitoring image corresponding to the position of the monitoring object detected when viewed from the shovel 100 through the display processing unit 301. More specifically, the safety function control unit 305 may overlap and display a frame surrounding the monitoring object shown in the monitoring image through the display processing unit 301, or overlap and display a mark at a position on the monitoring image corresponding to the actual existence position of the detected monitoring object (for example, refer to Fig.10 , Fig.21 , Fig. 22 ). Thus, the display device 50 can realize a visual notification function for the operator. In addition, the safety function control unit 305 can also use a warning light or lighting device inside the cab 10 to notify the operator inside the cab 10 that the monitored object has been detected.

[0160] Furthermore, the safety function control unit 305 may also activate the notification function based on a visual method to the operator, supervisor, etc. around the excavator 100 by, for example, controlling the lighting device such as the headlight provided in the storage part of the upper swing body 3 or the external display device. Furthermore, the safety function control unit 305 may also activate the notification function based on a tactile method to the operator in the cab 10 by, for example, controlling the vibration generating device that vibrates the operator's seat where the operator sits. Thus, the surrounding monitoring device 150 can enable the operator or the operator and supervisor around the excavator 100 to recognize that there is a monitoring object (for example, the operator, etc.) around the excavator 100. Therefore, the surrounding monitoring device 150 can promote the operator to confirm the safety status around the excavator 100, and can promote the operator, etc. in the monitoring area to evacuate the monitoring area.

[0161] Furthermore, the safety function control unit 305 may also activate the remote notification function by, for example, sending a command signal indicating the operation of the notification function to the management device 200 or the terminal device 300 using the communication device 70. In this case, the management device 200 (control device 210) may output an alarm based on a visual method or an auditory method through the output device 230 when receiving the command signal from the shovel 100 through the communication device 220. Thus, the manager of the management device 200, the operator who remotely controls the shovel 100 through the management device 200, and the like can grasp that the monitored object has entered the notification range around the shovel 100. Similarly, the terminal device 300 (control device 310) may output an alarm based on a visual method or an auditory method through the output device 330 when receiving the command signal from the shovel 100 through the communication device 320.

[0162] In addition, the remote notification function of the safety function control unit 305 may also be transferred to the management device 200 or the terminal device 300. In this case, the management device 200 receives information related to the detection status of the monitoring object by the detection unit 304 from the shovel 100, determines whether there is a monitoring object entering the notification range based on the received information, and activates the external notification function when there is a monitoring object in the notification range. The same can be done in the case of the terminal device 300.

[0163] Furthermore, the safety function control unit 305 may change the notification form (ie, notification method) according to the positional relationship between the monitoring object detected within the notification range and the shovel 100 .

[0164] For example, when the monitoring object detected by the detection unit 304 in the notification range exists at a position relatively far from the shovel 100, the safety function control unit 305 can output an alarm with a relatively low urgency (hereinafter referred to as an "alarm of the attention level") to the operator or the like to urge the operator to pay attention to the monitoring object. Hereinafter, the range relatively far from the shovel 100 in the notification range (i.e., the range corresponding to the alarm of the attention level) is sometimes referred to as the "attention notification range". On the other hand, when the monitoring object detected by the detection unit 304 in the notification range exists at a position relatively close to the shovel 100, the safety function control unit 305 can output an alarm with a relatively high urgency (hereinafter referred to as an "alarm of the warning level") to notify that the monitoring object is close to the shovel 100 and the risk is increased. Hereinafter, the range relatively close to the shovel 100 in the notification range (i.e., the range corresponding to the alarm of the warning level) is sometimes referred to as the "alarm notification range".

[0165] At this time, the safety function control unit 305 may change the pitch, sound pressure, timbre, ringing cycle, etc. of the sound output from the sound output device 54 between the warning level alarm and the alert level alarm. Furthermore, the safety function control unit 305 may change the color, shape, size, presence or absence of flashing, flashing cycle, etc. of the image indicating the detection of the monitoring object or the image (for example, a frame or mark, etc.) that emphasizes the monitoring object or the position of the monitoring object displayed on the monitoring image displayed on the display device 50 between the warning level alarm and the alert level alarm. Thus, the surrounding monitoring device 150 enables the operator or the like to grasp the urgency, in other words, the proximity of the monitoring object to the shovel 100, based on the difference between the notification sound (alarm sound) output from the sound output device 54 or the notification image displayed on the display device 50.

[0166] The safety function control unit 305 may stop the notification function when the monitoring object originally detected by the detection unit 304 is no longer detected in the monitoring area after the notification function is activated, or when a prescribed operation to cancel the notification function is accepted through the input device 52 .

[0167] Furthermore, for example, when the monitoring object is detected by the detection unit 304 within a predetermined range (hereinafter referred to as "motion restriction range") included in the monitoring area, the safety function control unit 305 activates the motion restriction function. The motion restriction range may be the same as the monitoring area, or may be set such that its outer edge is relatively closer to the shovel 100 than the monitoring area. Furthermore, the motion restriction range includes at least one of a motion deceleration range, in which the motion speed of the shovel 100 is made slower than the normal motion speed in response to the operation of the operating device 26, remote operation, or an operation command corresponding to the automatic operation function, and a motion stop range, in which the motion of the shovel 100 is stopped regardless of the operation of the operating device 26, remote operation, or an operation command corresponding to the automatic operation function, and the stop state is maintained. For example, when the motion restriction range includes both the motion deceleration range and the motion stop range, the motion stop range is, for example, a range within the motion restriction range close to the shovel 100, and the motion deceleration range is a range within the motion restriction range set outside the motion stop range.

[0168] The safety function control unit 305 activates the motion limiting function of limiting the motion of the shovel 100 by controlling the hydraulic control valve 56. At this time, the safety function control unit 305 can limit the motion of all driven elements (i.e., the corresponding hydraulic actuators) or the motion of a part of the driven elements (hydraulic actuators). Thus, the surrounding monitoring device 150 can slow down or stop the motion of the shovel 100 when there is a monitoring object around the shovel 100. Therefore, the surrounding monitoring device 150 can prevent the monitoring object around the shovel 100 from coming into contact with the shovel 100.

[0169] Furthermore, when the monitoring object originally detected by the detection unit 304 is no longer detected after the operation of the operation restriction function, or when a prescribed operation for releasing the operation of the operation restriction function is accepted through the input device 52, the safety function control unit 305 stops the operation restriction function. The operation for releasing the operation of the notification function to the input device 52 and the operation for releasing the operation of the operation restriction function may be the same or different.

[0170] Furthermore, the safety function control unit 305 may activate the safety function in response to the establishment of other predetermined conditions (hereinafter referred to as “other operating conditions”) that are different from the detection of the monitoring object by the detection unit 304 .

[0171] For example, when the operator in the cab 10 does not wear a seat belt, the safety function control unit 305 can activate the internal notification function or the action restriction function of prohibiting (stopping) the action of the shovel 100. This can promote the operator of the cab 10 to wear a seat belt. Therefore, the safety of the shovel 100 can be improved.

[0172] Furthermore, for example, when the openable / closable window of the cab 10 is opened, the safety function control unit 305 can activate the internal notification function or the action restriction function of prohibiting (stopping) the action of the shovel 100. This is because, if the shovel 100 overturns or the like with the window of the cab 10 opened, the operator may be thrown out of the cab 10. Thus, the operator of the cab 10 can be urged to close the window of the cab 10. Therefore, the safety of the shovel 100 can be improved.

[0173] [Displayable area around the excavator and detectable area of ​​the monitored object]

[0174] Next, refer to Figure 8 to Figure 11 The range around the excavator 100 (hereinafter referred to as the "displayable area") that can be displayed on the display device 50, the output device 230 (display device) and the output device 330 (display device) (hereinafter referred to as the "display device 50, etc.") based on the image captured by the camera device 40 and the range around the excavator 100 that can be detected as a monitored object by the detection unit 304 (hereinafter referred to as the "detectable area") are described.

[0175] In the following, in this example, the description is made on the premise that the shovel 100 is located on a horizontal plane. In addition, in this example, the description is centered on the displayable area and the detectable area in the circumferential direction (rotation direction) when viewed from the shovel 100 (specifically, the upper rotating body 3). That is, in this example, the displayable area means the range in the angular direction that can be displayed by the display device 50 or the like based on the image captured by the camera device 40 within the range of the angular direction of 360 degrees in the horizontal direction when viewed from the shovel 100. Similarly, in this example, the detectable area means the range in the angular direction in which the detection unit 304 can detect the monitored object within the range of the angular direction of 360 degrees in the horizontal direction when viewed from the shovel 100. Therefore, when comparing the displayable area, the detectable area of ​​the detection unit 304A, and the detectable area of ​​the detection unit 304B, the range in the circumferential direction (i.e., the angular range in the horizontal direction) when viewed from the shovel 100 may be used as the comparison object, and the range in the radial direction (far direction) or the up-down direction may be excluded from the comparison object.

[0176] <Comparison between the displayable area and the detectable area of ​​the image captured by the imaging device>

[0177] Figure 8 The diagram shows an example of a displayable area on the display device 50 or the like based on an image captured by the imaging device 40 and a detectable area by the detection unit 304A when the shovel 100 is viewed from above.

[0178] like Figure 8 As shown, the horizontal shooting range (angle of view) of camera 40X is represented by the angle range between two dot-dashed lines extending from camera 40X. Therefore, the displayable area is represented by a range equivalent to the union of all horizontal shooting ranges of cameras 40B, 40L, and 40R.

[0179] On the other hand, the detectable area of ​​the detection unit 304A based on the captured image of the camera device 40 (camera 40X) corresponds to the pear skin portion in the figure. Specifically, the detectable area of ​​the detection unit 304A is represented by the range except the range near the excavator 100 within the angle range between two solid lines (the portion near the excavator 100 is a dotted line) extending from the cameras 40B, 40L, and 40R, respectively. This is because the upper surface of the upper revolving body 3 where the camera 40X is installed is located at a relatively high position from the ground, so that sometimes only a part of the monitored object near the ground in the range near the upper revolving body 3 is reflected in the captured image, or sometimes the body of the upper revolving body 3 is reflected in the captured image, making the ground in the range near the upper revolving body 3 a blind spot for the body.

[0180] Furthermore, the angle range between the two solid lines extending from the cameras 40B, 40L, and 40R, respectively, corresponding to the detectable area of ​​the detection unit 304A, is narrower than the actual angle of view (angle range between the two dashed lines) corresponding to the shooting range of the cameras 40B, 40L, and 40R. This is because, at the end of the shooting range (angle of view) of the cameras 40B, 40L, and 40R in the horizontal direction when viewed from the shovel 100 (the upper swing body 3), only a part of the monitored object may be reflected, and the detection unit 304A may not be able to recognize (detect) the monitored object from the captured image.

[0181] For example, the worker 401 is located at the left front of the upper slewing body 3. A part of the worker 401 is included in the angle range between the two dashed lines corresponding to the horizontal shooting range of the camera 40L. Therefore, the display device 50 etc. can display a monitoring image including a part of the worker 401 based on the image shot by the camera 40L under the control of the controller 30 (display processing unit 301).

[0182] On the other hand, the worker 401 is located at the front end of the horizontal imaging range of the camera 40L. Therefore, the position of the worker 401 is not included in the detectable area of ​​the detection unit 304A, and the detection unit 304A cannot detect the worker 401 only partially reflected in the image captured by the camera 40L.

[0183] Similarly, the worker 402 is located at the right front position of the upper slewing body 3. A part of the worker 402 is included in the angle range between the two dot-dash lines corresponding to the horizontal shooting range of the camera 40R. Therefore, the display device 50 etc. can display a monitoring image including a part of the worker 402 based on the image shot by the camera 40R under the control of the controller 30 (display processing unit 301).

[0184] On the other hand, the worker 402 is located at the front end of the horizontal imaging range of the camera 40R. Therefore, the position of the worker 402 is not included in the detection area of ​​the detection unit 304A, and the detection unit 304A cannot detect the worker 402 only partially reflected in the image captured by the camera 40R.

[0185] Thus, the monitoring object located at the end of the viewing angle of the front side of the cameras 40L and 40R in the displayable area may not be detected by the detection unit 304A. Therefore, when only the camera device 40 (camera 40X) is used to detect the monitoring object around the shovel 100, there are monitoring objects that are not detected even though they are shown on the display device 50, etc., which may cause a sense of discomfort to the operator, manager, etc. of the shovel 100.

[0186] <First example of detectable area of ​​monitoring target>

[0187] Fig. 9 The diagram shows a first example of a detectable area using the detection unit 304A of the imaging device 40 and a detectable area using the detection unit 304B of the surrounding information acquisition device 45 when the shovel 100 is viewed from above.

[0188] like Fig. 9 As shown, the detectable area of ​​the detection unit 304A corresponds to the pear skin portion in the figure as described above. Therefore, as described above, the ends of the front side viewing angles of the cameras 40L and 40R within the displayable area defined by the angle range between the two single-point dashed lines extending from the camera 40X are excluded from the detectable area of ​​the detection unit 304A.

[0189] On the other hand, the detectable area of ​​the detection unit 304B is defined by the angle range in the horizontal direction between the two thick lines extending from the sensor 45X (i.e., the range of the detection angle in the horizontal direction). Specifically, the detectable area of ​​the detection unit 304B is represented as the union of the ranges of the detection angles in the horizontal direction of the sensors 45BL, 45BR, 45L, and 45R. The range of the detection angle in the horizontal direction defined by the two thick lines corresponds to, for example, the irradiation range of the infrared light of the LIDAR in the horizontal direction.

[0190] The range of the detection angle of the sensor 45L in the horizontal direction includes all the front ends of the angle of view of the camera 40L in the displayable area including the position of the operator 401 (i.e., the occupied range). Furthermore, the range of the detection angle of the sensor 45R in the horizontal direction includes all the front ends of the angle of view of the camera 40R in the displayable area including the position of the operator 402 (i.e., the occupied range). That is, the detectable area of ​​the detection unit 304B includes substantially all of the detectable area of ​​the detection unit 304A. Thus, the detection unit 304B can detect the monitoring object in the range outside the range of the detection area of ​​the detection unit 304A in the left front and right front of the upper slewing body 3. Therefore, the monitoring object (e.g., the operators 401 and 402) that is only partially included in the front ends of the displayable areas corresponding to the cameras 40L and 40R (i.e., the right and left ends of each captured image) can be detected.

[0191] For example, Fig.10 3 is a diagram showing a specific example (monitoring image 500) of a monitoring image displayed when a monitoring target is detected by the detection unit 304. Specifically, Fig.10 is a display device 50 as a monitoring image 500. Fig. 9 FIG. 4 is a diagram of an image captured by camera 40L (through image) under the condition of FIG.

[0192] In addition, the monitoring image 500 may be displayed on the output device 230 (display device) of the management device 200 or the output device 330 (display device) of the terminal device 300 instead of the display device 50 of the shovel 100, or in addition to the display device 50 of the shovel 100. Thus, the operator of the shovel 100 can remotely operate the shovel 100 while checking the monitoring image 500. In addition, the manager of the shovel 100 can remotely monitor the operating conditions of each of the plurality of shovels 100 operating using the automatic operation function through the monitoring image 500.

[0193] like Fig. 9 As shown in FIG. 4 , the front end of the displayable area of ​​the camera 40L includes a portion of the area where the operator 401 is located. Fig.10 As shown, a part of a worker 510 corresponding to the worker 401 is displayed (projected) at the right end of the monitoring image 500 (the image captured by the camera 40L) corresponding to the front end of the displayable area.

[0194] Furthermore, as described above, the detection unit 304 (detection unit 304B) can detect the worker 401 (worker 510) who is located at the end of the front side of the displayable area of ​​the camera 40L and only a part of whom is shown in the captured image (through image). Therefore, a frame 520 indicating that the worker 510 shown is detected by the detection unit 304 (detection unit 304B) is displayed in the monitoring image 500. Thus, even if the monitoring object (worker 510) is only partially shown in the monitoring image 500, the user such as the operator and the manager of the shovel 100 can understand the situation that can be detected by the detection unit 304 (detection unit 304B).

[0195] Similarly, in the monitoring image MP2 (reference Figure 7 ) is displayed on the display device 50 or the like, even if only a part of the workers 401 and 402 is shown at the left front and right front ends when the shovel image CG of the viewpoint conversion image EP is viewed, the detection unit 304B can detect the workers 401 and 402. Therefore, the workers 401 and 402 shown in the monitoring image MP2 including the viewpoint conversion image EP can be displayed in a manner of surrounding a frame, and the frame indicates that the workers 401 and 402 have been detected by the detection unit 304. Thus, as in the case of the monitoring image 500, even if only a part of the monitoring object is shown in the viewpoint conversion image EP (monitoring image MP2), the user such as the operator and the manager of the shovel 100 can understand the situation that can be detected by the detection unit 304.

[0196] In addition, the image showing the situation where a part of the monitoring object shown in the monitoring image (surrounding image such as the through image or the viewpoint conversion image EP) is detected may be a form different from the frame (for example, the frame 520). For example, the image showing the situation where a part of the monitoring object shown in the monitoring image (surrounding image) is detected may be the following Fig. 20 , Fig. 22 The method of indicating an image (for example, indicating image 913 or indicating images 934, 935) is described.

[0197] Thus, in this example, the detection unit 304 can use the surrounding information acquisition device 45 (sensor 45X) to supplement the range in which the monitoring object cannot be detected when using only the captured image of the camera device 40, and detect the monitoring object within the range. Therefore, the detection unit 304 can detect the monitoring object within the entire range in the circumferential direction of the surroundings when viewed from the shovel 100 in the monitoring image displayed on the display device 50 (that is, the entire relatively close range around the shovel 100). That is, the detection unit 304B can detect all the monitoring objects included in the capturing range of the cameras 40B, 40L, and 40R in the circumferential direction when viewed from the shovel 100, that is, all the monitoring objects displayed in the monitoring image displayed on the display device 50. For example, even when only a part of the monitoring object is displayed at the left and right ends of the captured image of the camera device 40, the detection unit 304B can detect the monitoring object. Therefore, it is possible to suppress the user from feeling a sense of discomfort due to the discrepancy between the display content of the display device 50 (the monitored object is displayed) and the detection result (the monitored object is not detected).

[0198] Return to Fig. 9 For example, the excavator 100 may sometimes operate in a state where the straight-moving direction (extending direction) of the lower traveling body 1 and the direction of the upper rotating body 3 are roughly consistent. Fig. 9 As shown, even if the workers 401 and 402 are relatively close to the left front or right front of the upper swing body 3, the risk of the workers 401 and 402 coming into contact with the excavator 100 is not high. This is because it is assumed that the excavator 100 can only move in the front-to-back direction of the upper swing body 3, and even if the lower traveling body 1 moves, the possibility of contact with the workers 401 and 402 at the left front and right front of the upper swing body 3 is low. In addition, it is assumed that the distance between the front end of the upper swing body 3 and the center of rotation is relatively close, so even if the upper swing body 3 rotates, it is not easy to shorten the distance between the front end of the upper swing body 3 and the workers 401 and 402.

[0199] On the other hand, the excavator 100 may sometimes operate while rotating the upper rotating body 3 at a relatively high frequency. Therefore, for example, the upper rotating body 3 may be rotated at a relatively large angle relative to the straight travel direction (extending direction) of the lower traveling body 1 (e.g., Fig. 9 The operation is performed in a state where the rotation angle is about 90 degrees, as in the case of the crawler 1CL and 1CR of the dotted line. Fig. 9As shown, the workers 401 and 402 at relatively close positions in front of the left or right side of the upper revolving body 3 may be in a state where the direction of travel faces the crawler 1C (crawler 1CL) on the front side when viewed from the upper revolving body 3. Therefore, if the lower traveling body 1 is traveling, the possibility of contact with the workers 401 and 402 increases. In addition, the space facing the crawler 1C (crawler 1CL) on the front side when viewed from the upper revolving body 3 is likely to become a blind spot when viewed from the cab 10, resulting in the operator of the cab 10 being unable to visually recognize it.

[0200] In contrast, in this example, Fig. 9 As shown, at least a portion of the workers 401 and 402 facing the crawler 1CL is included in the displayable area. Specifically, a portion of the workers 401 facing the crawler 1CL is included in the front end of the shooting range in the horizontal direction of the camera 40L. In addition, the workers 402 facing the crawler 1CL is included in the front end of the shooting range in the horizontal direction of the camera 40R. Therefore, even if the workers 401 and 402 facing the crawler 1CL are in a blind spot when viewed from the cab 10, the operator of the cab 10 can grasp the existence of the workers 401 and 402 through the monitoring image of the display device 50. In addition, the remote operator, manager, etc. of the excavator 100 using the management device 200 and the terminal device 300 can also grasp the existence of the workers 401 and 402 through the monitoring image of the output device 230 or the output device 330. Therefore, the safety of the excavator 100 can be improved.

[0201] Furthermore, a portion of the workers 401 and 402 facing the crawler 1C (crawler 1CL) on the front side when viewed from the upper revolving body 3 may be reflected on the left and right ends of the images captured by the cameras 40L and 40R, and displayed as a monitoring image on the display device 50. However, as described above, the detection unit 304A cannot recognize (detect) the workers 401 and 402 who are the monitoring objects and are only partially reflected in the images captured by the cameras 40L and 40R.

[0202] On the other hand, Fig. 9 As shown in FIG. 1 , the detectable area of ​​the detection unit 304B is defined by the infrared irradiation range of each of the sensors 45BL, 45BR, 45L, and 45R in the horizontal direction (the angle range between the two thick lines in the figure). This is because, unlike the case of image recognition, even when only a part of the reflected light from the monitored object can be received, the monitored object can be recognized (detected) based on the shape, size, or intensity of the reflected light of the object based on the light receiving information, as described above.

[0203] Specifically, the operator 401 is located at a position where the left end of the crawler 1C (crawler 1CL) on the front side faces the straight-moving direction on the left side when the upper rotating body 3 is rotated at an angle of about 90 degrees based on the straight-moving direction of the lower traveling body 1, and is included in the detectable area of ​​the detection unit 304B corresponding to the sensor 45L. Furthermore, the operator 402 is located at a position where the right end of the crawler 1C (crawler 1CL) on the front side faces the straight-moving direction on the right side when the upper rotating body 3 is rotated at an angle of about 90 degrees based on the straight-moving direction of the lower traveling body 1, and is included in the detectable area of ​​the detection unit 304B corresponding to the sensor 45R.

[0204] For example, as described above, if the lower traveling body 1 starts traveling in a state where the upper slewing body 3 is swung at an angle of about 90 degrees based on the straight traveling direction of the lower traveling body 1, it is possible that the lower traveling body 1 comes into contact with the workers 401 and 402. In addition, in this state, it is possible that the workers 401 and 402 are difficult to be observed with the naked eye from the cab 10.

[0205] On the other hand, as described above, the detection unit 304B can detect the monitoring object in the space in the straight direction of the crawler 1C facing the front side when the upper swing body 3 is viewed from the upper swing body 3 when the swing angle is about 90 degrees based on the straight direction of the lower traveling body 1. Therefore, the surrounding monitoring device 150 can activate safety functions such as the notification function and the action restriction function according to the detection of the workers 401 and 402. Therefore, the safety of the excavator 100 can be further improved.

[0206] Thus, in this example, when the orientation of the upper rotating body 3 deviates from the reference state that is consistent with the straight direction of the crawler 1C, the space in the straight direction of the crawler 1C facing the front side of the upper rotating body 3 is included in the displayable area of ​​the display device 50 or the like as the monitoring image. Also, in this example, when the orientation of the upper rotating body 3 deviates from the reference state that is consistent with the straight direction of the crawler 1C, the space in the straight direction of the crawler 1C facing the front side of the upper rotating body 3 is included in the detectable area of ​​the monitoring object based on the detection unit 304.

[0207] For example, as long as a camera 40X and a sensor 45X can be mounted on the lower traveling body 1, it is possible to more easily detect a monitored object at a specific position when observed from the lower traveling body 1 (track 1C) or display it on a display device 50, regardless of the rotation state of the upper rotating body 3.

[0208] However, in reality, if the working environment of the excavator 100 and the actual operation of the operator are taken into consideration, it is not realistic to mount the camera 40X and the sensor 45X on the lower traveling body 1. This is because the camera 40X and the sensor 45X are likely to fail to function due to the adhesion of sand and the like or to be damaged due to contact with the ground. Also, since the operator often uses the attachments of the upper rotating body 3 to perform operations, the operator needs information related to the positional relationship of the monitored object with the upper rotating body 3 as a reference. Therefore, it is necessary to mount the camera 40X and the sensor 45X on the upper rotating body 3. Therefore, if the relationship between the orientation of the upper rotating body 3 and the crawler 1C deviates from the reference state, the camera 40X and the sensor 45X may not be able to properly obtain information related to the positional relationship between the lower traveling body 1 and the surrounding objects when traveling.

[0209] Specifically, when the lower traveling body 1 is traveling in the same direction as the upper swing body 3, the operator of the cab 10 can directly visually recognize the operator facing the front of the crawler 1C when the lower traveling body 1 is traveling in front of the upper swing body 3. In addition, when remote operation or remote monitoring is performed, the operator can also confirm the operator facing the front of the crawler 1C based on the display content because the image captured by the front camera for remote operation is displayed on the display device for remote operation. In addition, in the same state, when the lower traveling body 1 is traveling behind the upper swing body 3, the operator, manager, etc. of the excavator 100 can confirm the operator facing the rear of the crawler 1C based on the image captured by the camera 40B displayed on the display device 50, etc.

[0210] On the other hand, if the traveling direction of the lower traveling body 1 deviates from the state in which the direction of the upper rotating body 3 is consistent, the information of the space facing the lower traveling body 1 (track 1C) may not be properly acquired depending on the number or configuration of the cameras 40X and the sensors 45X. For example, since a relatively wide range spanning between the left and right sides centered on the rear of the upper rotating body 3 becomes a blind spot for the operator of the cab 10, relatively more cameras 40X and sensors 45X corresponding to the range may be set. In contrast, since the operator, manager, etc. can directly visually recognize or confirm the front of the upper rotating body 3 through the display image of the front camera, relatively fewer cameras 40X and sensors 45X corresponding to the range spanning from the right front to the left front of the upper rotating body 3 may be set. Therefore, when the moving direction of the lower walking body 1 deviates from the state consistent with the orientation of the upper rotating body 3, if there is a space facing the track 1C in the left front or right front of the upper rotating body 3, the camera 40X and the sensor 45X may not be able to properly obtain information about the space.

[0211] In contrast, in this example, when the orientation of the upper swing body 3 deviates from the state in which the crawler 1C is aligned with the straight-moving direction, the space in the straight-moving direction of the crawler 1C facing the front side of the upper swing body 3 can be included in the displayable area of ​​the display device 50 or the like as the monitoring image. Furthermore, when the orientation of the upper swing body 3 deviates from the state in which the crawler 1C is aligned with the straight-moving direction, the space in the straight-moving direction of the crawler 1C facing the front side of the upper swing body 3 can be included in the detectable area of ​​the monitoring object by the detection unit 304. Therefore, the safety of the lower traveling body 1 of the shovel 100 during traveling can be improved.

[0212] In addition, the space in the straight-moving direction of the crawler 1C facing the front side of the upper rotating body 3 may be included only in a certain area of ​​the displayable area and the detectable area. Furthermore, as long as the entire displayable area in the circumferential direction around the excavator 100 is included in the detectable area of ​​the detection unit 304B in the circumferential direction when observed from the excavator 100, the number of sensors 45X may be arbitrary. The configuration of the sensor 45X is also the same. Furthermore, instead of using the detection units 304A and 304B to redundantly output two detection results related to the monitored object, a structure may be adopted in which the outputs of the two devices, the camera device 40 and the surrounding information acquisition device 45, are integrated to output one detection result related to the monitored object. In this case, the same effect is also exerted. The same is true for the second example described below.

[0213] <Second example of detectable area of ​​monitoring object>

[0214] Fig.11 It is a diagram showing a second example of the detectable area using the detection unit 304A of the imaging device 40 and the detectable area using the detection unit 304B of the surrounding information acquisition device 45 when the shovel 100 is viewed from above.

[0215] like Fig.11 As shown, in this example, the difference is that sensors 45LF and 45LR are provided on the upper surface of the upper rotating body 3 instead of the sensor 45L, and 45RF and 45RR are provided instead of the sensor 45R. That is, in this example, the surrounding information acquisition device 45 includes sensors 45BL, 45BR, 45LF, 45LR, 45RF, and 45RR. Hereinafter, in this example, the description will be centered on the parts that are different from the above-mentioned first example, and the description of the same parts and corresponding parts may be omitted in some cases. In addition, in this example, sensors 45BL, 45BR, 45LF, 45LR, 45RF, and 45RR are sometimes collectively referred to as "sensors 45X".

[0216] For example, the shovel 100 may be in a state where the upper swing body 3 is relatively largely swung within a range of a swing angle less than 90 degrees based on the straight travel direction of the crawler 1C (the front and rear axes of the crawler 1C). Fig.11 Specifically, in this example, the excavator 100 is in a state where the upper swing body 3 is turned 75 degrees to the right based on the straight direction of the crawler 1C.

[0217] In this state, the operator 403 is located at a position facing the straight-moving direction at the front end of the crawler 1C (crawler 1CR) on the front side when viewed from the upper swing body 3. At this time, since the operator 403 is located at a position deviated from the front of the cab 10, it may be difficult for the operator of the cab 10 to see the operator 403 with naked eyes. Similarly, the remote operator, manager, etc. of the excavator 100 may not be able to confirm the operator 403 through the image of the remote operation front camera displayed on the output device 230 (display device) of the management device 200 and the output device 330 (display device) of the terminal device 300. In addition, the space facing the straight-moving direction at the front end of the crawler 1CR is greatly deviated from the detectable area of ​​the detection unit 304A (the pear-skin surface portion in the figure), and is not included in the shooting range of the camera 40L (the angle range between the two single-point chain lines in the figure).

[0218] In contrast, in this example, two sensors 45X (sensors 45LF and 45LR) are provided for the left side of the upper rotating body 3. Specifically, the sensor 45LF is provided at a front position of the left end of the upper surface of the upper rotating body 3 so that the optical axis is slightly oriented to the left front when viewed from above. Also, the sensor 45LR is provided at a rear position of the left end of the upper surface of the upper rotating body 3 so that the optical axis is slightly oriented to the left rear when viewed from above. Thus, while the sensor 45LR is used to maintain the left side from the left front to the left rear of the upper rotating body 3 as the irradiation range of infrared rays, the sensor 45LF is used to include the space facing the straight-ahead direction at the front end of the crawler 1CR in the irradiation range. Therefore, the detection unit 304B can detect the operator 403 who is located at the front end of the crawler 1CR facing the straight-ahead direction when the upper rotating body 3 is relatively rotated within a range of a rotation angle of less than 90 degrees based on the straight-ahead direction of the crawler 1C, and the operator 403 who is located at the front end of the crawler 1CR facing the straight-ahead direction can be detected.

[0219] Furthermore, in this example, two sensors 45X (sensors 45RF and 45RR) are provided to be responsible for the right side of the upper rotating body 3. Specifically, the sensor 45RF is provided at a position forward of the right end of the upper surface of the upper rotating body 3 so that the optical axis is slightly toward the right front when viewed from above. Furthermore, the sensor 45RR is provided at a position backward of the right end of the upper surface of the upper rotating body 3 so that the optical axis is slightly toward the right rear when viewed from above. Fig.11 On the contrary, in the case where the front and rear axes of the crawler 1C are facing the upper left, when the front and rear axes of the crawler 1C are facing the upper right, the sensor 45RR can be used to maintain the right side from the right front to the right rear of the upper swing body 3 as the infrared irradiation range, and the sensor 45RF can be used to include the space facing the straight direction at the rear end of the crawler 1CR in the irradiation range. Therefore, in this state, the detection unit 304B can detect the operator who is located at the rear end of the crawler 1CR facing the straight direction.

[0220] Thus, in this example, when the upper swing body 3 is relatively largely rotated with respect to the straight direction of the crawler 1C, the space facing the straight direction of the crawler 1C is included in the range (detectable area) in which the monitoring object can be detected by the detection unit 304. Specifically, the number or arrangement of the sensors 45X is appropriately adjusted so that when the upper swing body 3 is relatively largely rotated within a range of a rotation angle less than 90 degrees with respect to the straight direction of the crawler 1C, the detection unit 304B can detect the monitoring object (for example, the operator 403) existing in the position facing the straight direction of the crawler 1C on the front side when viewed from the upper swing body 3. As a result, the safety of the excavator 100 can be further improved.

[0221] In addition, when the upper slewing body 3 is relatively largely swung with reference to the straight-moving direction of the crawler 1C, the space facing the straight-moving direction of the crawler 1C may be included in the range (displayable area) that can be displayed as a monitoring image on the display device 50 or the like by appropriately adjusting the number or arrangement of the cameras 40X instead of including the space facing the straight-moving direction of the crawler 1C in the detectable area of ​​the detection unit 304, or in addition to including the space facing the straight-moving direction of the crawler 1C in the detectable area of ​​the detection unit 304, the space may be included in the range (displayable area) that can be displayed as a monitoring image on the display device 50 or the like by appropriately adjusting the number or arrangement of the cameras 40X. Thus, the operator or the like can grasp the state of the space facing the straight-moving direction of the crawler 1C through the monitoring image of the display device 50 or the like.

[0222] [Detection method of monitoring object]

[0223] Next, refer to Figure 12 to Figure 15A specific example of a method of detecting a monitoring target by the detection unit 304 will be described.

[0224] <Specific example of the method of identifying the monitoring object>

[0225] Figure 12 to Figure 15 1 is a diagram showing a specific example of a person (i.e., a worker) at a work site that is a monitoring target. Specifically, Fig.12 This is a diagram showing a first example of a worker (worker W1 ). Fig.13 This is a diagram showing a second example of a worker (worker W2). Fig.14 This is a diagram showing a third example of a worker (worker W3). Fig.15 This is a diagram showing a fourth example of a worker (worker W4).

[0226] like Fig.12 As shown, the worker W1 wears a helmet HMT and a vest with a reflective material (hereinafter referred to as a "reflective vest") RV. The reflective material is a member with extremely high retro-reflectivity (performance of reflecting irradiated light toward a light source).

[0227] like Fig.13 As shown, operator W2 wears a safety helmet HMT but does not wear a reflective vest RV.

[0228] like Fig.14 As shown, worker W3 is wearing a reflective vest RV but not a safety helmet HMT.

[0229] like Fig.15 As shown, operator W4 wore neither a safety helmet HMT nor a reflective vest RV.

[0230] When detecting a person (worker) who is a monitoring target, the detection unit 304A attempts to recognize both the entire person (worker) and the helmet from the captured image of the camera device 40. Thus, even if the entire person cannot be recognized with an accuracy exceeding a predetermined standard due to the person's orientation or posture, the detection unit 304A can detect the worker when the helmet can be recognized with an accuracy exceeding a predetermined standard. That is, among the workers W1 to W4, the detection unit 304A can detect the workers W1 and W2 wearing the helmet HMT with a relatively high probability compared to the other workers W3 and W4.

[0231] On the other hand, when detecting a person (operator) who is the monitoring object, the detection unit 304B attempts to identify (detect) an object with a relatively high (specifically, higher than a specified reference) light intensity of the received reflected light (infrared rays) based on the light receiving information. The light receiving intensity of the reflected light is obtained based on the light receiving intensity information. In addition, the higher the retro-reflectivity of the surface of the object, the higher the light receiving intensity of the reflected light from the object. On the other hand, the longer the distance from the object to the sensor 45X, the lower the light receiving intensity of the reflected light from the object. Therefore, the detection unit 304B can consider the distance to the object based on the TOF information and correct the light receiving intensity based on the light receiving intensity information, for example, to the light receiving intensity when the object is assumed to be at a specified distance. At this time, the specified reference is pre-set in a manner that can distinguish between reflective materials and objects other than these. Thus, when an object with a relatively high light receiving intensity of the reflected light is identified, the detection unit 304B can determine that it is an operator wearing clothes with reflective materials such as a reflective vest, thereby detecting the operator. In addition to reflective vests, clothes with reflective materials include any clothes with reflective materials installed, such as jackets with reflective materials, pants with reflective materials, etc. That is, among workers W1 to W4, the detection unit 304B can detect workers W1 and W3 wearing reflective vests RV with a relatively high probability compared to other workers W2 and W4.

[0232] In addition, the detection unit 304B can identify the shape or size of an object based on the light receiving information corresponding to the reflected light received from multiple irradiation directions. Therefore, when the detection unit 304B detects a person (operator) as a monitoring object, it can also use the shape or size of the object based on the light receiving information to identify the person (operator).

[0233] As described above, the detection unit 304 uses the two detection units 304A and 304B to detect the monitored object. Thus, the detection unit 304 detects the worker who wears both a helmet and clothes with reflective materials (e.g., worker W1) with the highest probability. Furthermore, the detection unit 304 can detect the worker who wears only a helmet or only clothes with reflective materials (e.g., workers W2 and W3) with a certain degree of probability lower than the case of the worker who wears both a helmet and clothes with reflective materials. Furthermore, the detection unit 304 detects the worker who wears neither a helmet nor clothes with reflective materials (e.g., worker W4) with the lowest probability. In other words, the detection unit 304 is configured as follows: it is easiest to detect a person who wears both a helmet and clothes with reflective materials, it is second easiest to detect a person who wears only a helmet or only clothes with reflective materials, and it is most difficult to detect a person who wears neither a helmet nor clothes with reflective materials.

[0234] For example, when a person as a surveillance target is recognized (detected) using only the image captured by the imaging device 40, the person may not be recognized as a person or an object other than a person may be recognized as a person depending on the direction or posture of the person.

[0235] In contrast, in this example, in addition to the detection unit 304A that uses the output (captured image) of the camera 40 (camera 40X), a detection unit 304B that uses the output (light receiving information) of the surrounding information acquisition device 45 (sensor 45X) is provided. As a result, the surrounding monitoring device 150 can use not only the detection result of the detection unit 304A but also the detection result of the detection unit 304B. Therefore, even in the case where a person who should be detected by the detection unit 304A is not detected, or in the case where an object that is not a person is detected as a person, the surrounding monitoring device 150 can use the detection result of the detection unit 304B to suppress non-detection or erroneous detection. Therefore, the safety of the excavator 100 can be further improved. In addition, in this example, the detection unit 304 can detect the operator wearing a helmet as safety equipment and clothes with reflective material as safety equipment with relatively high accuracy. Therefore, from the perspective of ensuring the safety of the workers themselves and, in turn, the safety of the work site, the perimeter monitoring device 150 can encourage the workers to wear helmets or clothes with reflective materials.

[0236] In addition, in this example, the detection unit 304 is configured to be able to identify the helmet worn by the worker or the clothes with reflective material worn by the worker, but it can also detect (identify) the person (worker) who is the monitoring object in the form of identifying other safety equipment. The safety equipment that is the identification object can include, for example, safety belts, earplugs for noise scenes, safety boots, gloves for arc welding or vibration, dustproof or gas masks, dustproof or light-shielding glasses, welding masks, ID (Identifier) ​​tags that output alarms when construction machinery approaches, etc. Thus, the surrounding monitoring device 150 can encourage workers to wear more safety equipment.

[0237] <Method for determining the location of the monitoring object>

[0238] The detection unit 304A determines the position of the worker (for example, the position of the sole of the foot) when viewed from the shovel 100 based on the size or position of the partial image corresponding to the recognized whole person (worker) or helmet in the captured image. This is because the size of the worker in the captured image changes depending on the distance of the worker from the camera 40X. In addition, the position shown in the captured image changes depending on the distance and direction of the worker from the camera 40X.

[0239] On the other hand, the detection unit 304B determines the position of the operator when viewed from the shovel 100 based on the TOF information in the light receiving information. This is because the TOF information includes the irradiation direction of light when viewed from the sensor 45X and the time from irradiation of light in the irradiation direction to reception of reflected light, so that the direction of the object with respect to the sensor 45X as a reference can be determined based on the former, and the distance between the sensor 45X and the object can be determined based on the latter.

[0240] For example, as described above, the longer the distance between the object and the sensor 45X, the lower the light intensity of the reflected light (infrared) from the object. Therefore, if this property is used, the position (distance) of the monitored object can also be determined based on the light intensity information. However, the light intensity of the reflected light from the object will also change according to the properties of the surface of the object (specifically, retro-reflectivity or reflectivity, etc.). Therefore, in the case where only the light intensity information is used to determine the position of the monitored object, the accuracy of the determined position will be relatively reduced, and the surrounding monitoring device 150 may not be able to accurately grasp the position of the surrounding monitored objects. For example, in the case where the deviation of the retro-reflectivity of the clothes with reflective material worn by the operator is large, the range of the light intensity of the reflected light is relatively wide, which may cause the accuracy of the distance information based on the light intensity information to be reduced. Therefore, in the case where safety functions such as the notification function or the action restriction function are activated according to the position of the monitored object, the action conditions of the notification function or the action restriction function may be biased.

[0241] In contrast, the detection unit 304B identifies (detects) the monitoring object based on the light intensity information and the TOF information, thereby determining the position of the monitoring object. Then, when the safety function control unit 305 detects a monitoring object whose intensity of reflected light received by the detection unit 304B is relatively high and whose distance to the shovel 100 is relatively close (i.e., within the notification range or the action restriction range), it activates at least one of the notification function and the action restriction function. That is, the safety function control unit 305 outputs an alarm or restricts the action of the shovel 100 based on the output of the sensor 45X (the light intensity information and the TOF information). As a result, the surrounding monitoring device 150 can use the TOF information, and thus can determine the position of the monitoring object around the shovel 100 with relatively high accuracy compared to the case where only the light intensity information is used. Therefore, when the safety function such as the notification function or the action restriction function is activated based on the position of the monitoring object, the surrounding monitoring device 150 can suppress the deviation of the action conditions of the notification function or the action restriction function. Therefore, the surrounding monitoring device 150 can improve the safety of the shovel 100.

[0242] In addition, in this example, the detection unit 304B may determine the position of the monitored object using other information related to the distance between the shovel 100 and the surrounding objects instead of the TOF information, or may determine the position of the monitored object using other information related to the distance between the shovel 100 and the surrounding objects in addition to the TOF information. For example, the detection unit 304B may determine the position of the monitored object using information related to the distance to the surrounding objects acquired from the captured image of the stereo camera as the imaging device 40 (camera 40X).

[0243] [Safety function operation control method]

[0244] Next, a specific example of the operation control method of the safety function (notification function and operation restriction function) by the safety function control unit 305 will be described.

[0245] In this example, the safety function control unit 305 changes the operation mode of the safety function between a case where the monitoring target is detected by the detection unit 304A and a case where the monitoring target is detected by the detection unit 304B.

[0246] <Example 1 of the safety function operation control method>

[0247] When the monitoring object is detected by the detection unit 304A, the safety function control unit 305 activates only the notification function among the safety functions. That is, when the monitoring object is detected by the detection unit 304A, the safety function control unit 305 does not activate the action restriction function regardless of the distance between the monitoring object and the shovel 100. Specifically, when the monitoring object is detected by the detection unit 304A within the notification range (hereinafter referred to as the "first notification range"), the safety function control unit 305 activates the notification function and does not set the action restriction range (hereinafter referred to as the "first action restriction range") corresponding to the detection of the monitoring object by the detection unit 304A.

[0248] On the other hand, when the monitoring object is detected by the detection unit 304B in the action restriction range (hereinafter referred to as the "second action restriction range"), the safety function control unit 305 activates the action restriction function. Furthermore, when the monitoring object is detected by the detection unit 304B in the notification range (hereinafter referred to as the "second notification range"), the safety function control unit 305 may also activate the notification function. The first notification range and the second notification range may be the same or different.

[0249] For example, even if the detection accuracy related to the monitored object is relatively low, the action restriction function is still activated based on its detection result. In this case, the operator may feel uncomfortable or the working efficiency of the shovel 100 may be reduced, compared with the case where the notification function is activated.

[0250] In contrast, in this example, the surrounding monitoring device 150 allows the action restriction function to be activated when the monitoring object is detected by only the detection unit 304B of the detection units 304A and 304B. This is because the detection accuracy when the monitoring object is detected based on the light receiving information of the surrounding information acquisition device 45 tends to be relatively higher than the case where the monitoring object is detected by image recognition of the image captured by the camera device 40. In other words, the detection accuracy of the monitoring object by the detection unit 304B tends to be higher than that of the detection unit 304A. As a result, the surrounding monitoring device 150 can improve the safety of the shovel 100 while suppressing the discomfort given to the operator or the reduction in the working efficiency of the shovel 100.

[0251] <Second example of how to control the operation of safety functions>

[0252] The safety function control unit 305 changes the notification mode (i.e., notification method) when the notification function is activated between the case where the monitoring object is detected by the detection unit 304A in the first notification range and the case where the monitoring object is detected by the detection unit 304B in the second notification range. Thus, the operator, manager, and surrounding workers of the shovel 100 can easily understand whether the monitoring object is detected by the imaging device 40 (camera 40X) or the surrounding information acquisition device 45 (sensor 45X).

[0253] For example, the safety function control unit 305 may change the timbre, volume, pattern, ringing cycle, content of the sound output from the sound output device 54 between the case where the monitoring object is detected by the detection unit 304A and the case where the monitoring object is detected by the detection unit 304B. In addition, the same may be applied to the case where the output device 230 (sound output device) of the management device 200 and the output device 330 (sound output device) of the terminal device 300 are operated by the remote notification function. Specifically, when the monitoring object is detected by the detection unit 304B, the safety function control unit 305 may adopt a timbre, volume, pattern, ringing cycle, content of the sound, etc. that indicates a higher urgency (risk) than when the monitoring object is detected by the detection unit 304A. This is because, as described above, the detection accuracy of the detection unit 304B with respect to the monitoring object tends to be higher than that of the detection unit 304A.

[0254] Furthermore, for example, the safety function control unit 305 may change the display method related to the notification function displayed on the display device 50 or the like between the case where the monitoring object is detected by the detection unit 304A and the case where the monitoring object is detected by the detection unit 304B (for example, refer to Fig. 20 ).

[0255] [Safety function operation control method when optional equipment is installed]

[0256] Next, refer to Fig.16 A specific example of the operation control method of the safety function by the safety function control unit 305 when an optional device (hereinafter simply referred to as “optional device”) that increases the occupied volume of the shovel 100 is installed will be described.

[0257] Fig.16 This diagram shows the range of motion limitations when optional equipment is installed.

[0258] like Fig.16 As shown, in this example, the excavator 100 is equipped with a blade 90 as an optional equipment on the front side of the lower traveling body 1. Specifically, the blade 90 is installed in a manner protruding from the front side in the straight-moving direction of the lower traveling body 1 (crawler 1CL, 1CR) (white arrow in the figure). Therefore, when the lower traveling body 1 moves in the direction (right side in the figure) in which the blade 90 is installed (for example, straight-moving or slow-speed rotating walking), the distance between the surrounding workers 610 and the excavator 100 (blade 90) is shortened compared to a state in which the blade 90 is not installed. In addition, when the lower traveling body 1 performs a pivoting turn or a rotation turn, the blade 90 will also rotate around the excavator 100 along with the movement of the lower traveling body 1, so the rotation radius of the blade 90 increases by the amount of protrusion from the front side of the lower traveling body 1. Therefore, compared with a state in which the blade 90 is not attached to the shovel 100 , the distance between the surrounding worker 610 and the shovel 100 (blade 90 ) may be shortened.

[0259] In contrast, when the blade 90 moves toward the vicinity of the excavator 100 as the excavator 100 moves (i.e., the walking movement of the lower traveling body 1), the safety function control unit 305 makes the safety function easier to operate than in other cases. Other cases include, for example, a case where the lower traveling body 1 moves to the rear where the blade 90 is not installed, thereby moving the blade 90 away from the vicinity of the excavator 100. In this example, the outer edge of the movement restriction range of the blade 90 in the direction toward the vicinity of the excavator 100 (i.e., the front of the lower traveling body 1) is set so that the blade 90 installed on the lower traveling body 1 is farther away from the excavator 100 (e.g., the dotted line 630 in the figure) than the case where the blade 90 is not installed on the lower traveling body 1 (e.g., the dotted line 620 in the figure). This setting (change) can be performed by the setting unit 302, and the setting content can be registered in the storage unit 303. In addition, the outer edge of the notification range corresponding to the notification function can also be the same. Thus, when the excavator 100 moves toward the front side (right side in the figure) of the lower traveling body 1 to which the blade 90 is installed, the safety function control unit 305 can activate the safety function (alarm function or action restriction function) earlier based on the detection of the operator 610 based on the detection unit 304. Therefore, the safety function control unit 305 can activate the safety function more appropriately in accordance with the installation of the optional equipment. Therefore, the surrounding monitoring device 150 can further improve the safety of the excavator 100. In other words, when the optional equipment (blade 90) is installed, the safety function control unit 305 can activate the safety function in a manner that makes the safety of the excavator 100 relatively higher than when the optional equipment is not installed.

[0260] Furthermore, instead of setting the outer edge of the notification range or the action restriction range in a direction away from the excavator 100, other conditions for making the safety function work may be relaxed so as to make it easy for the safety function to work. For example, in the case where a condition related to the degree of accuracy of the monitoring object detected by the detection unit 304 (hereinafter referred to as "the accuracy of the monitoring object") is included as an action condition of the safety function, the condition may be relaxed. Information related to the accuracy of the monitoring object is output from the detection unit 304. Specifically, in the case where the blade 90 moves around the excavator 100 as the excavator 100 works, the setting unit 302 may set the threshold corresponding to the condition of the accuracy of the monitoring object to be relatively lower than other cases. Furthermore, in the case where the blade 90 is installed, the setting unit 302 may set the threshold corresponding to the condition of the accuracy of the monitoring object to be relatively lower than the case where the blade 90 is not installed. Thus, the safety function control unit 305 can make the safety function work more appropriately in accordance with the installation of the optional equipment.

[0261] Furthermore, the safety function to be operated may be switched to a relatively high safety specification instead of mitigating the operating condition of the safety function. For example, the setting unit 302 may be set to enable the notification function and the action restriction function as a safety function when the blade 90 moves around the excavator 100 as the excavator 100 moves, and to enable only the notification function in other cases. Furthermore, the setting unit 302 may be set to enable the notification function and the action restriction function as a safety function when the blade 90 is installed, and to enable only the notification function as a safety function when the blade 90 is not installed. This is because the action restriction function is relatively safer than the notification function from the perspective of being able to forcibly restrict the action of the excavator 100 regardless of the intention of the operator or the like. Furthermore, the setting unit 302 may be set to enable the action restriction function to be activated with a relatively high degree of restriction when the blade 90 moves around the excavator 100 as the excavator 100 moves, and to enable the action restriction function to be activated with a relatively low degree of restriction in other cases. Furthermore, the setting unit 302 may be set to enable the action restriction function to function with a relatively high degree of restriction when the blade 90 is installed, and to enable the action restriction function to function with a relatively low degree of restriction when the blade 90 is not installed. Furthermore, the setting unit 302 may be set to enable the notification function to function in a manner indicating a relatively high degree of urgency (risk) when the blade 90 moves around the excavator 100 as the excavator 100 moves, and to enable the notification function to function in a manner indicating a relatively low degree of urgency (risk) in other cases. Furthermore, the setting unit 302 may be set to enable the notification function to function in a manner indicating a relatively high degree of urgency (risk) when the blade 90 is installed, and to enable the notification function to function in a manner indicating a relatively low degree of urgency (risk) when the blade 90 is not installed. Thus, the safety function control unit 305 can more appropriately enable the safety function in accordance with the installation of the optional equipment.

[0262] Furthermore, in the case where other types of optional equipment are installed in place of the blade 90 or in addition to the blade 90, the safety function can also be activated in a relatively safe manner. Other types of optional equipment may include, for example, an additional counterweight installed at the rear of the upper rotating body 3 or a large counterweight that is heavier than the counterweight of the standard equipment. Furthermore, other types of optional equipment may also include a relatively long boom installed in place of the boom 5. Furthermore, other types of optional equipment may also include, for example, a spare attachment (for example, a crusher or a mixer, etc.) installed as an end attachment in place of the bucket 6 or a spare attachment (for example, a quick coupling or a tilt rotator, etc.) installed between the end attachment and the boom 5.

[0263] Furthermore, in the case where the optional equipment can be loaded and unloaded, the safety function control unit 305 can determine whether the optional equipment exists according to the type of the optional equipment, and change the operation mode of the safety function as described above according to the result of its determination. For example, the setting unit 302 performs settings related to the installation status of the optional equipment (for example, whether each type of optional equipment is installed, etc.) according to the specified input made by the operator or the like through the input device 52. Then, the safety function control unit 305 can determine whether the optional equipment exists according to its setting content registered in the storage unit 303. Furthermore, for example, the safety function control unit 305 can automatically determine whether the optional equipment is installed according to the type of the optional equipment based on the output of a sensor (for example, a camera device 40 or a switch provided in the installation portion of the optional equipment) that can obtain information related to the installation status of the optional equipment.

[0264] Furthermore, when the state in which the optional device is not installed is changed to the state in which the optional device is installed, the display processing unit 301 may display a notification related to the change of the operation mode of the safety function (hereinafter referred to as "change notification") on the display device 50 or the like. The change notification includes a notification indicating that the operation object of the safety function is automatically changed. Thus, the surrounding monitoring device 150 can enable the operator, the manager, etc. to grasp the situation that the operation mode of the safety function is automatically changed due to the installation of the optional device. Furthermore, the change notification may also be a notification to promote whether to allow the change of the operation mode of the safety function. At this time, when the change notification is displayed on the display device 50 or the like, the user such as the operator and the manager can allow or refuse the change of the operation mode of the safety function through the input device 52, 240, 340. Therefore, the intention of the user such as the operator and the manager can be reflected in the change of the operation mode of the safety function corresponding to the installation status of the optional device. Furthermore, the change notification may also be a notification to promote manual change of the operation mode of the safety function. At this time, the user can manually change the operation mode of the safety function according to the change notification. That is, the setting unit 302 can change the operation mode of the safety function according to the manual setting operation performed by the operator, administrator, etc. through the input device 52, 240, 340. Therefore, the intention of the user such as the operator can be reflected in the change of the operation mode of the safety function corresponding to the installation status of the optional equipment.

[0265] [External notification function action control method]

[0266] Next, a specific example of the operation control method based on the external notification function of the safety function control unit 305 is described.

[0267] As described above, the safety function control unit 305 is configured to activate the external notification function when the monitoring target is detected by the detection unit 304 .

[0268] For example, the positional relationship between the excavator 100 and the surrounding workers caused by the actions of the workers who are the monitored objects changes relatively slowly, so it is considered that the possibility of a sharp decrease in safety is not high. Therefore, in a situation where the safety is not so reduced, the necessity of activating the external notification function is relatively low. On the contrary, if the external notification function is activated, it may scare the workers and cause work errors. Therefore, in this example, when the monitored object is detected by the detection unit 304 (within the notification range), the safety function control unit 305 sometimes activates the external notification function and sometimes does not activate the external notification function.

[0269] On the other hand, for example, it is also conceivable that the safety of the excavator 100 is drastically reduced due to its operation, depending on its operating conditions or the conditions of the terrain. Therefore, it is considered that even in a situation where the safety is not so reduced, it is necessary to notify the operator inside the cab 10 or the operator who is operating remotely of the existence of the monitoring object such as the surrounding workers. In addition, when the excavator 100 is operating through the automatic operation function, the manager of the excavator 100, etc., sometimes monitors the operating conditions of each excavator 100 while observing the monitoring image displayed on the output device 230 (display device) of the management device 200. Similarly, the manager of the excavator 100, etc., sometimes uses the terminal device 300 to monitor its operating conditions. Therefore, it is considered that even in a situation where the safety is not so reduced, it is necessary to notify the manager who is remotely monitoring the operating conditions of the excavator 100, etc., of the existence of the monitoring object such as the surrounding workers. Therefore, when the monitoring target is detected by the detection unit 304 within the notification range, the safety function control unit 305 can activate the internal notification function or the remote notification function even when the external notification function is not activated.

[0270] For example, the safety function control unit 305 can switch between activating the external notification function and not activating the external notification function, depending on which of the detection units 304A and 304B detects the monitoring object. Specifically, the safety function control unit 305 can be configured so that the external notification function cannot be activated when the monitoring object is detected by the detection unit 304A, and on the other hand, the external notification function can be activated when the monitoring object is detected by the detection unit 304B. For example, if the external notification function is activated, the work of the operator around the excavator 100 may be stopped, and if it is an erroneous notification, the operation efficiency may be reduced due to the unnecessary operation of the external notification function. In contrast, as described above, by setting the external notification function to be activated only when the monitoring object is detected by the detection unit 304B, which has a relatively high detection accuracy among the detection units 304A and 304B, a balance between safety and operation efficiency around the excavator 100 can be achieved.

[0271] Furthermore, for example, when the monitoring object is detected by the detection unit 304, the safety function control unit 305 may switch between activating the external notification function and not activating the external notification function by linking with the operation status of the action restriction function. Specifically, even when the monitoring object is detected by the detection unit 304 within the notification range, when the action restriction function (for example, the action deceleration function) is not activated (for example, as described later, when the monitoring object is unlikely to contact the shovel 100, etc.), the safety function control unit 305 may not activate the external notification function. On the other hand, when the monitoring object is detected by the detection unit 304 within the notification range and the action restriction function (for example, the action deceleration function) is activated, the safety function control unit 305 may activate the external notification function. Thus, the surrounding monitoring device 150 can ensure the safety of the shovel 100 while suppressing the situation in which the external notification function is activated when the surrounding monitoring object is unlikely to contact the shovel 100, thereby preventing the workers around the shovel 100 from stopping their work. Therefore, it is possible to suppress a decrease in the work efficiency at the work site while taking into account the safety of the shovel 100. Furthermore, if the action limiting function of the shovel 100 is activated, it will attract the attention of workers and the like around the shovel 100, so there is a high possibility that the work around the shovel 100 will be stopped or delayed from the beginning. Therefore, even if the external notification function activated in this situation is an erroneous notification, since the work around the shovel 100 will be stopped or delayed from the beginning, the problem of reduced work efficiency is unlikely to occur.

[0272] Furthermore, for example, when the monitoring object is detected within the notification range by the detection unit 304, the safety function control unit 305 may switch between activating and activating the external notification function according to whether the monitoring object exists in the direction in which the excavator 100 moves. Specifically, the safety function control unit 305 may determine the direction in which the excavator 100 moves in accordance with the operation of the driven elements (e.g., the lower traveling body 1, the upper swing body 3, the attachment, etc.) based on the operation information of the excavator 100, and not activate the external notification function when there is no monitoring object in the direction in which the excavator 100 moves. On the other hand, the safety function control unit 305 may determine the direction in which the excavator 100 moves in accordance with the operation information of the excavator 100, and activate the external notification function when there is a monitoring object in the direction in which the excavator 100 moves. Thus, the surrounding monitoring device 150 can ensure the safety of the shovel 100, and prevent the external notification function from being activated when there is a low possibility that the surrounding monitoring object will come into contact with the shovel 100, thereby preventing the workers around the shovel 100 from stopping their work. Therefore, it is possible to prevent the work efficiency at the work site from being reduced while taking the safety of the shovel 100 into consideration.

[0273] Furthermore, for example, when a monitored object is detected within the notification range by the detection unit 304, the safety function control unit 305 may switch between enabling and disabling the external notification function depending on the level of risk based on the positional relationship between the excavator 100 and the monitored object.

[0274] Specifically, the safety function control unit 305 can activate the external notification function when the risk (e.g., the possibility of contact, etc.) based on the positional relationship between the shovel 100 and the monitored object is relatively high, and on the other hand, the external notification function is not activated when the risk based on the positional relationship between the shovel 100 and the monitored object is relatively low. The risk can be acquired (calculated) based on the distance between the monitored object and the shovel 100, for example. In addition, the risk can also be acquired (calculated) based on the monitored object and the movement direction of the driven element corresponding to the operation of the shovel 100 (e.g., the travel direction of the lower traveling body 1, the rotation direction of the upper rotating body 3, the moving direction of the attachment, etc.). As a result, the surrounding monitoring device 150 can ensure the safety of the shovel 100 while suppressing the situation where the external notification function is activated when the possibility of contact between the surrounding monitored objects and the shovel 100 is low, thereby preventing the workers around the shovel 100 from stopping their work. Therefore, it is possible to suppress the reduction of work efficiency at the work site while taking into account the safety of the shovel 100.

[0275] Furthermore, for example, when the monitoring object is detected within the notification range by the detection unit 304, the safety function control unit 305 may also switch between enabling and disabling the external notification function according to the operation state of the door lock rod. Specifically, when the door lock rod is in the released state, the safety function control unit 305 may enable the external notification function. This is because, when the door lock rod is in the released state, the excavator 100 is more likely to operate according to the operator's operation or the operation instruction corresponding to the automatic operation function. On the other hand, when the door lock rod is in the locked state, the safety function control unit 305 may not enable the external notification function. This is because, when the door lock rod is in the locked state, the operator's operation or the operation instruction corresponding to the automatic operation function is invalidated, and the excavator 100 does not operate. As a result, it is possible to suppress the reduction of the operating efficiency at the work site while taking into account the safety of the excavator 100.

[0276] Furthermore, for example, when the monitoring object is detected within the notification range by the detection unit 304, the safety function control unit 305 may also switch between enabling and disabling the external notification function according to whether the operation of the shovel 100 is stopped when other operation conditions are satisfied. Specifically, when the operation of the shovel 100 is not stopped (prohibited) correspondingly when other operation conditions are satisfied, the safety function control unit 305 may enable the external notification function. This is because the shovel 100 may operate according to the operation of the operator or the operation instruction corresponding to the automatic operation function. On the other hand, when the operation of the shovel 100 is stopped (prohibited) correspondingly when other operation conditions are satisfied, the safety function control unit 305 may not enable the external notification function. This is because the shovel 100 may stop regardless of the operation of the operator or the operation instruction corresponding to the automatic operation function. Thus, it is possible to suppress the reduction of the work efficiency at the work site while taking into account the safety of the shovel 100.

[0277] Furthermore, for example, when the monitoring object is detected by the detection unit 304 within the notification range, the safety function control unit 305 may switch between activating the external notification function and not activating the external notification function according to the duration of the stop state of the shovel 100. Specifically, when the duration of the stop state of the shovel 100 is longer than a predetermined threshold, the safety function control unit 305 may activate the external notification function. This is because, if the duration of the stop state of the shovel 100 is relatively long, the workers around the shovel 100 may have a stronger idea that the shovel 100 will not move, and the sense of safety may be reduced. On the other hand, when the duration of the stop state of the shovel 100 is less than the predetermined threshold (including when the shovel 100 is not stopped), the safety function control unit 305 may not activate the external notification function. This is because, on the contrary, it is believed that when the duration of the stop state of the shovel 100 is relatively short and the shovel 100 is continuously moving, it is easy to arouse the attention of the workers to the movement of the shovel 100, and the sense of safety of the workers may be relatively high. Thereby, it is possible to suppress a decrease in the work efficiency at the work site while taking the safety of the shovel 100 into consideration.

[0278] Furthermore, when the monitoring object is detected within the notification range by the detection unit 304, the safety function control unit 305 may switch between activating the external notification function and not activating the external notification function according to the sign of the start of the action in the stopped state of the shovel 100. Specifically, when the shovel 100 is stopped and there is a sign of the shovel 100 starting to act, the safety function control unit 305 may activate the external notification function. Furthermore, when the duration of the stop of the shovel 100 is greater than a predetermined threshold and there is a sign of the shovel 100 starting to act, the safety function control unit 305 may also activate the external notification function. This is because the possibility of the shovel 100 starting to act is high. On the other hand, when the shovel 100 is not stopped or there is no sign of the shovel 100 starting to act while it is stopped, the safety function control unit 305 may not activate the external notification function. Furthermore, when the duration of the stop of the excavator 100 is below a predetermined threshold value (including when the excavator 100 is not stopped) or when there is no sign of the excavator 100 starting to move, the safety function control unit 305 may not enable the external notification function. This is because, as described above, it is believed that when the excavator 100 is moving, it is easy to arouse the attention of the operator to the excavator 100, so that the operator's awareness of safety will be relatively high. And because, in a situation where the excavator 100 is unlikely to start to move, the necessity of the external notification function will be relatively reduced. As a result, it is possible to suppress the reduction of the work efficiency at the work site while taking into account the safety of the excavator 100.

[0279] Whether the excavator 100 has a sign of starting to move can be determined, for example, based on the operation state of the door lock rod. This is because, when the door lock rod is in a locked state, the excavator 100 is unlikely to start moving. On the other hand, when the door lock rod is operated from a locked state to a released state, the excavator 100 is more likely to start moving. In addition, whether the excavator 100 has a sign of starting to move can also be determined, for example, based on whether the operator has a sign of starting to operate. Whether the operator has a sign of starting to operate can be identified, for example, based on an image captured by a camera installed in the cab 10 or a camera that captures an operator performing remote operation. The camera that captures the operator performing remote operation is installed, for example, in the management device 200 or the terminal device 300. In addition, whether the operator has a sign of starting to operate can also be identified, for example, based on the contact state (holding state) of the operator on the operating device 26 or the operating device for remote operation. For example, when the operating device 26 or the operating device for remote operation is not held by the operator, its output is stabilized in a state equivalent to zero. On the other hand, if the operator holds the operating device 26 or the operating device for remote operation, the output of the operating device 26 or the operating device for remote operation jumps to a state indicating a small amount of operation before starting the operation. Therefore, by monitoring the output (pressure signal or electrical signal) of the operating device 26 or the operating device for remote operation, it is possible to grasp the contact state between the operator's hand and the operating device or the operating device for remote operation, thereby identifying whether the operator has signs of starting the operation. In addition, whether the excavator 100 has signs of starting to move can also be determined based on the movement pattern of the excavator 100. The movement pattern of the excavator 100 can be identified, for example, based on the content of the operator's operation or the history of the content of the operation instruction corresponding to the automatic operation function.

[0280] Furthermore, for example, the setting unit 302 can switch the setting of the external notification function between on (validation) and off (invalidation) according to a predetermined operation performed by the operator or the like through the input device 52. At this time, when the monitoring object is detected within the notification range by the detection unit 304, the safety function control unit 305 can switch between enabling and disabling the external notification function according to the setting status related to whether the external notification function is operated or not. Specifically, when the monitoring object is detected within the notification range by the detection unit 304, the safety function control unit 305 can enable the external notification function when the external notification function is set to on (validation), and disabling the external notification function when the external notification function is set to off (invalidation). Thus, the surrounding monitoring device 150 can reflect the intention of the operator or the like in the judgment of whether to enable the external notification function.

[0281] As described above, the safety function control unit 305 can activate the external notification function through the sound output device 54. As described above, the sound output device 54 includes, for example, a speaker 54a or a walking alarm 54b. Thus, the safety function control unit 305 can use the existing speaker 54a or walking alarm 54b to realize the external notification function. Therefore, the cost for realizing the external notification function can be suppressed.

[0282] For example, the horn 54a may be a structure that can sound by energizing the coil and closing the relay 62 according to a control instruction (control current) corresponding to the operation of the external notification function from the controller 30 regardless of the on operation of the knob switch 64. The controller 30 can output sound from the horn 54a with a prescribed sounding pattern by appropriately controlling the opening / closing pattern of the horn relay. At this time, the sounding pattern of the horn 54a corresponding to the external notification function can usually be set to a pattern different from the sounding pattern when the operator or the like starts walking. Thus, the surrounding monitoring device 150 can enable the operators and the like around the excavator 100 to recognize the notification (signal) when the excavator 100 starts walking and the operation of the external notification function.

[0283] Furthermore, for example, the travel alarm 54b outputs sound in a prescribed sounding pattern according to the control instruction of the controller 30 corresponding to the operation of the external notification function. At this time, the sounding pattern of the travel alarm corresponding to the external notification function can be set to a different pattern from that when the lower traveling body 1 is normally traveling (for example, a buzzer sound that repeatedly sounds "beep, beep, beep, ..."). Thus, similar to the case of the speaker 54a, the surrounding monitoring device 150 can enable the workers and the like around the excavator 100 to recognize the notification when the excavator 100 is traveling and the operation of the external notification function.

[0284] Furthermore, as described above, the safety function control unit 305 may activate the external notification function by using a lighting device such as a headlight provided in the storage portion of the upper swing body 3 or the like, or an external display device.

[0285] For example, the lighting device is turned on or repeatedly turned on and off in a predetermined pattern according to a control instruction of the controller 30 corresponding to the operation of the external notification function.

[0286] Furthermore, for example, the external display device displays character information that urges workers, supervisors, and the like around the shovel 100 to evacuate the area around the shovel 100 in accordance with a control command of the controller 30 corresponding to the operation of the external notification function.

[0287] [Motion control method of the motion restriction function]

[0288] Next, refer to Figure 17 to Figure 19A specific example of the motion control method based on the motion limiting function of the safety function control unit 305 will be described.

[0289] Fig.17 1 is a diagram for explaining a method of determining whether or not there is contact with a monitoring object around the shovel 100. Specifically, Fig.17 2 shows a working state 700 in which the lower traveling body 1 (crawler tracks 1CL, 1CR) of the shovel 100 is traveling straight to the right in the figure. Fig.18 , Fig.19 1 is a diagram for explaining the motion control method of the motion limiting function. Specifically, Fig.18 , Fig.19 These are diagrams showing the motion control method of the motion limiting function in the working conditions 810 and 820 of the shovel 100 , respectively.

[0290] In this example, when the monitoring object is detected within the motion restriction range by the detection unit 304, the safety function control unit 305 determines whether the monitoring object comes into contact with the shovel 100 based on the motion of the shovel 100 corresponding to the operation of the shovel 100 (i.e., the operation of the operating device 26, the remote operation) or the operation command corresponding to the automatic operation function. Then, when it is determined that the monitoring object comes into contact with the shovel 100, the safety function control unit 305 activates the motion stop function to stop the motion of the shovel 100 in response to the operation of the shovel 100.

[0291] Specifically, the safety function control unit 305 can determine the direction of the driven element's movement based on the operation information of the driven element as the operation object. Then, the safety function control unit 305 can predict whether the shovel 100 is in contact with the monitored object based on the information related to the positional relationship between the outer surface of the shovel 100 and the monitored object (detected by the detection unit 304) and the determined direction of the shovel 100 movement. The information related to the positional relationship between the outer surface of the shovel 100 and the monitored object may include, for example, the output information of the camera 40 (camera 40X) or the surrounding information acquisition device 45 (sensor 45X) or the detection result of the detection unit 304. In addition, the information related to the positional relationship between the outer surface of the shovel 100 and the monitored object may include, for example, information related to the installation position of the camera 40X or the sensor 45X on the shovel 100 (upper swing body 3). This is because it is possible to determine which part of the outer surface of the shovel 100 the monitored object is close to based on the installation position of the camera 40X or the sensor 45X that acquires information related to the monitored object. Furthermore, the information related to the positional relationship between the outer surface of the shovel 100 and the monitored object may include information related to the body structure (e.g., specifications such as width or shape, etc.) of the shovel 100 relative to the direction in which the shovel 100 moves. This is because the positional relationship between the outer surface of the shovel 100 and the monitored object may change depending on the body structure (occupied width) of the shovel 100 relative to the direction in which the shovel 100 moves.

[0292] For example, when the monitoring object is detected by the detection unit 304, the safety function control unit 305 determines the direction of the excavator 100 movement accompanying the movement of the driven element based on the operation information of the driven element at a predetermined period. Thus, the determination result of the direction of the excavator 100 movement is updated in accordance with the change of the operation information of the driven element. Similarly, the detection unit 304 performs the process of detecting the monitoring object around the excavator 100 at a predetermined period and outputs the position information of the detected monitoring object. Thus, the relative position information between the monitoring object continuously detected by the detection unit 304 and the excavator 100 is sequentially updated. Then, the safety function control unit 305 can determine whether the excavator 100 and the monitoring object will come into contact in the period until the predetermined time passes thereafter based on the direction of the excavator 100 movement determined by itself and the position information of the monitoring object output from the detection unit 304. Thus, the safety function control unit 305 can sequentially update the judgment result of whether or not there is contact between the shovel 100 and the surrounding monitoring object according to the sequentially updated direction of the shovel 100 movement and the position of the monitoring object when viewed from the shovel 100. Therefore, even in a situation where the operation content related to the driven element changes (for example, even in the case of a walking operation in which the operation content related to the lower traveling body 1 is changed from straight walking to slow turning walking, etc.), it is possible to appropriately judge whether or not there is contact between the shovel 100 and the monitoring object.

[0293] Furthermore, for example, the safety function control unit 305 may also determine whether the shovel 100 is in contact with the monitored object using a predetermined range (i.e., a motion stop range) set based on information related to the positional relationship between the outer surface of the shovel 100 and the monitored object and the determined direction of the shovel 100. Specifically, the motion stop range may be set by the setting unit 302 at a predetermined period as a range in which the shovel 100 and the monitored object may come into contact.

[0294] For example, in Fig.17 In the working condition 700, in the excavator 100 in which the upper rotating body 3 and the attachment are represented by solid lines (hereinafter referred to as "the excavator 100 with solid lines"), the upper rotating body 3 (attachment) is rotated 90 degrees to the left relative to the straight direction of the lower traveling body 1 (the right side in the figure), so as to face the upper side in the figure. Furthermore, in the excavator 100 in which the upper rotating body 3 and the attachment are represented by single-dot chain lines (hereinafter referred to as "the excavator 100 with single-dot chain lines"), the upper rotating body 3 is rotated to the left relative to the straight direction of the lower traveling body 1 (the right side in the figure), so as to face the upper right side in the figure.

[0295] When the lower traveling body 1 of the shovel 100 in the solid line moves to the right side in the figure, the machine body (lower traveling body 1 and upper swing body 3) passes through the width (range) between the dotted lines 710 defined by the left end (outer end) of the crawler 1CL and the rear end of the upper swing body 3 in a plan view. Therefore, in the state of the shovel 100 in the solid line, when the lower traveling body 1 moves, the safety function control unit 305 can determine whether there is contact between the monitoring object and the shovel 100 (machine body) based on the positional relationship between the left end (outer end) of the crawler 1CL and the rear end of the upper swing body 3 and the surrounding monitoring object (for example, the operator 730). Furthermore, the safety function control unit 305 can also determine whether the monitored object is in contact with the shovel 100 (body) based on the body structure of the shovel 100 body (the width between the dotted lines 710) relative to the straight-moving direction of the shovel 100 (the right side in the figure) and the relative position of the monitored object around the shovel 100. For example, the safety function control unit 305 can determine that the shovel 100 is in contact with the monitored object when the monitored object is detected within the action stop range defined by adding a predetermined margin (for example, 1 meter to 1.5 meters) to the width between the dotted lines 710.

[0296] On the other hand, when the lower traveling body 1 of the excavator 100 of the single-dot chain line moves to the right side in the figure, the machine body (lower traveling body 1 and upper swing body 3) passes through the width (range) between the single-dot chain line 720 defined by the left front corner and the right rear corner of the upper swing body 3 in a plan view. Therefore, in the state of the excavator 100 of the single-dot chain line, when the lower traveling body 1 moves, the safety function control unit 305 can determine whether there is contact between the monitoring object and the excavator 100 (machine body) based on the positional relationship between the left front corner and the right rear corner of the upper swing body 3 and the surrounding monitoring object (for example, the operator 730). Furthermore, the safety function control unit 305 can also determine whether or not the monitored object is in contact with the shovel 100 (body) based on the body structure of the shovel 100 body relative to the straight-moving direction of the shovel 100 (the right side in the figure) (the width between the dashed lines 720) and the relative position of the monitored object around the shovel 100. For example, the safety function control unit 305 can determine that the shovel 100 is in contact with the monitored object when the monitored object is detected within the motion stop range defined by the width obtained by adding a predetermined margin to the width between the dashed lines 720.

[0297] As described above, in the case of the state of the excavator 100 of the solid line, the action stop range is equivalent to the range between the dotted lines 710, and in the case of the state of the excavator 100 of the single-dot chain line, the action stop range is equivalent to the range between the single-dot chain line 720. That is, the setting unit 302 can change the action stop range according to the rotation angle of the upper rotating body 3 relative to the lower walking body 1. Thus, the setting unit 302 can set the action stop range more appropriately corresponding to the rotation angle of the upper rotating body 3. Information related to the rotation angle of the upper rotating body 3 can be obtained, for example, based on the image captured by the camera 40X. Specifically, the rotation angle of the upper rotating body 3 can be determined based on the appearance of the crawler track reflected in the image captured by the camera 40X or which camera 40X it is reflected in, etc. Furthermore, the excavator 100 may be provided with a sensor capable of acquiring information related to the rotation angle of the upper rotating body 3 (for example, a rotary encoder mounted on the upper rotating body 3, an acceleration sensor, an angular velocity sensor, a six-axis sensor, an IMU (Inertial Measurement Unit), etc.

[0298] And, in Fig.17 In the working condition 700, it is assumed that the lower walking body 1 walks straight, but in addition to the straight walking, the lower walking body 1 also has walking modes such as slow turning walking, pivoting and turning, and rotating turning. Therefore, the safety function control unit 305 can judge whether it is a walking operation corresponding to the straight walking of the lower walking body 1, a walking operation corresponding to the slow turning walking, a turning operation corresponding to the pivoting and turning, or a turning operation corresponding to the rotating and turning according to the operation information input from the operation information output device 29. That is, the operation information related to the lower walking body 1 may include at least one of information indicating a walking operation corresponding to the straight walking, information indicating a walking operation corresponding to the slow turning walking, information indicating a walking operation corresponding to the pivoting and turning, and information indicating a walking operation corresponding to the rotating and turning.

[0299] In this way, the safety function control unit 305 can determine whether the excavator 100 is in contact with the surrounding monitoring object detected by the detection unit 304 by considering the body structure (occupied width) of the fuselage and the like relative to the movement direction of the driven element of the excavator 100. Specifically, the safety function control unit 305 can determine whether the monitoring object and the excavator 100 are in contact with the movement of the driven element based on the operation information related to the driven element of the excavator 100 and the information related to the positional relationship between the outer surface of the excavator 100 and the monitoring object. In addition, in the case of the walking operation of the lower walking body 1, the information related to the walking operation of the lower walking body 1 includes at least one of information indicating the walking operation corresponding to straight walking, information indicating the walking operation corresponding to slow turning walking, information indicating the walking operation corresponding to pivoting steering, and information indicating the walking operation corresponding to rotating steering. As a result, the safety function control unit 305 can grasp the travel direction of the lower walking body 1 in more detail, thereby more appropriately determining whether the excavator 100 is in contact with the monitoring object.

[0300] For example, in Fig.18 In the working condition 810, the excavator 100 is in a state where the rotation angle is 90 degrees to the left based on the straight direction (right side in the figure) of the lower traveling body 1. In addition, there is an operator 811 detected by the detection unit 304 in front of the lower traveling body 1 (right side in the figure).

[0301] At this time, the safety function control unit 305 allows the lower walking body 1 to move (backward) to the rear (left side in the figure) according to the walking operation of the lower walking body 1. This is because, since the excavator 100 moves in a direction away from the operator 811, the excavator 100 and the operator 811 will not come into contact. On the other hand, the safety function control unit 305 prohibits the lower walking body 1 from moving (advancing) in front of the lower walking body 1 (right side in the figure) according to the walking operation of the lower walking body 1, and activates the action stop function. This is because, since the operator 811 is in front of the direction of travel of the excavator 100, if it moves as it is, the excavator 100 will approach the operator 811, and contact will occur between the excavator 100 and the operator 811.

[0302] And, in Fig.19 Under the operating condition 820, Fig.18 The excavator 100 is in a state where the rotation angle is 90 degrees to the left based on the straight direction (right side in the figure) of the lower traveling body 1. In addition, there is an operator 821 detected by the detection unit 304 at the right oblique rear of the lower traveling body 1 and the right oblique front of the crawler 1CR.

[0303] At this time, the safety function control unit 305 allows the lower walking body 1 to move (backward) to the rear (left side in the figure) according to the walking operation of the lower walking body 1. This is because, since the excavator 100 moves in a direction away from the operator 811, there is no contact between the excavator 100 and the operator 811. In addition, the safety function control unit 305 also allows the lower walking body 1 to move (advance) in front of the lower walking body 1 (right side in the figure) according to the walking operation of the lower walking body 1. This is because, although the excavator 100 approaches the operator 812 as the lower walking body 1 moves, it will be away from the end of the occupied width of the excavator 100 (single-dotted line 822) to a certain extent, so even if it moves as it is (straight), there is no contact between the excavator 100 and the operator 811.

[0304] In addition, Fig.19 When the lower traveling body 1 performs a walking operation corresponding to the slow swing walking to the right side under the working condition 820, the excavator 100 moves toward the operator 812 while swinging to the right. Therefore, when the walking operation of the lower traveling body 1 is the slow swing walking to the right side, the safety function control unit 305 can prohibit the excavator 100 from moving and activate the motion stop function. Thus, the safety function control unit 305 can activate the motion stop function more appropriately according to the content of the walking operation of the lower traveling body 1.

[0305] In this way, when the shovel 100 is likely to come into contact with the monitoring object detected by the detection unit 304 due to the operation of the shovel 100, the safety function control unit 305 activates the operation stop function to stop the operation of the shovel 100 for the operation. On the other hand, even when the shovel 100 approaches the monitoring object detected by the detection unit 304 due to the operation of the shovel 100, when the monitoring object and the shovel 100 are not in contact, the safety function control unit 305 allows the operation of the shovel 100 for the operation. Specifically, the safety function control unit 305 determines whether the shovel 100 as a driven element and the monitoring object detected by the detection unit 304 will come into contact, and when it is determined that the shovel 100 and the monitoring object are not in contact, the operation of the shovel 100 is allowed for the operation of the shovel 100. At this time, the safety function control unit 305 may allow the movement of the shovel 100 while activating the movement deceleration function to reduce the movement speed of the shovel 100, or may allow the movement of the shovel 100 without activating the movement restriction function of the shovel 100. Thus, the surrounding monitoring device 150 can allow the movement of the shovel 100 without contact between the shovel 100 and the monitored object, thereby achieving a balance between the safety and working efficiency of the shovel 100. Therefore, the surrounding monitoring device 150 can stop the movement of the shovel 100 more appropriately according to the presence of obstacles (monitoring objects) around the shovel 100.

[0306] [Display control method of monitoring image when monitoring object is detected]

[0307] Next, refer to Figure 20 to Figure 22 A specific example of a method for controlling the display of a monitoring image by the display processing unit 301 when the monitoring target is detected by the detection unit 304 will be described.

[0308] Figure 20 to Figure 22 3 is a diagram showing a specific example of a monitoring image when the monitoring object is detected by the detection unit 304. Specifically, Figure 20 to Figure 22 These are diagrams showing first to third examples (monitoring images 910 to 930 ) of monitoring images displayed on the display device 50 when the monitoring target is detected by the detection unit 304 .

[0309] In addition, Fig. 22 In FIG. 1 , instruction images 934 and 935 that are not actually displayed are depicted with dotted lines.

[0310] First, if Fig. 20 As shown, under the control of the display processing unit 301 , the image captured by the camera 40B (through image) is displayed on the display device 50 as a monitoring image 910 .

[0311] A worker 911 wearing a helmet and a reflective vest is shown in monitoring image 910. In this example, worker 911 wears both a helmet and a reflective vest, and therefore can be detected by both detection unit 304A and detection unit 304B.

[0312] Furthermore, a frame 912 indicating that the worker 911 has been detected by the detection unit 304A is superimposed and displayed on the monitoring image 910. This allows an operator or the like to easily recognize the presence and position of the worker 911 detected by the detection unit 304A.

[0313] Furthermore, the display mode of the frame 912 can be changed according to the distance between the monitored object (operator 911) detected by the detection unit 304 (detection unit 304A) and the excavator 100. For example, the frame 912 can be displayed in yellow when the distance between the monitored object detected by the detection unit 304A and the excavator 100 is relatively far, and can be displayed in red when the distance between the monitored object and the excavator 100 is relatively close. Thus, the operator and the like can understand the positional relationship (distance) between the excavator 100 and the monitored object through the display mode of the frame 912. Fig.10 Frame 520 and the following Fig. 22 Frame 932 may also be the same.

[0314] In addition, an indication image 913 indicating that the monitoring object (operator 911) detected by the detection unit 304B is present in the monitoring image 910 is superimposed on the monitoring image 910. Thus, the operator or the like can recognize that the monitoring object detected by the detection unit 304B is present in the currently displayed monitoring image 910.

[0315] The indication image 913 has an arrowhead shape and is displayed at the lower end of the monitoring image 910, that is, at the end of the body (upper swing body 3) of the shovel 100 in the vertical direction of the monitoring image 910. The indication image 913 points from the lower end of the monitoring image 910 corresponding to the body side of the shovel 100 toward the inside of the monitoring image 910, that is, corresponding to the upper side of the inner side of the monitoring image 910. Thus, the indication image 913 can specifically indicate that the monitoring object (operator 911) detected by the detection unit 304B exists in the monitoring image 910.

[0316] Furthermore, the indication image 913 is formed into an arrow shape by a plurality of circular images (specifically, 14 circular images), and the monitoring image 910 is exposed from the gaps between the circular images. Thus, even when the indication image 913 is displayed overlappingly on the monitoring image 910 (the image captured by the camera 40X), the operator or the like can properly grasp the state of the vicinity of the excavator 100 corresponding to the portion of the monitoring image 910 on which the indication image 913 is displayed. Furthermore, even when the indication image 913 is displayed overlappingly on an image portion corresponding to the monitoring object, the image portion corresponding to the monitoring object and the image portion corresponding to the background can be exposed as is through the gaps between the indication image 913. Therefore, the user such as the operator can easily distinguish between the image portion corresponding to the monitoring object on which the indication image 913 is overlapped and the image portion corresponding to the background in the monitoring image 910. The following will describe the following. Fig.21 Indicative images 921, 922 and Fig. 22 The same is true for the instruction images 933 to 935 .

[0317] In addition, the instruction image 913 may be composed of a continuous arrow shape without a gap. In this case, the instruction image 913 (continuous arrow shape) may be displayed in a manner that makes the overlapped back image portion transparent and visible. Thus, even when the instruction image 913 is displayed overlapped on the monitoring image 910 (the image captured by the camera 40X), the operator or the like can appropriately grasp the state of the shovel 100 surrounding the portion of the monitoring image 910 where the instruction image 913 is displayed.

[0318] Furthermore, the display mode of the indication image 913 can be changed according to the distance between the monitored object (operator 911) detected by the detection unit 304 (detection unit 304B) and the excavator 100. For example, the indication image 913 can be displayed in yellow when the distance between the monitored object detected by the detection unit 304B and the excavator 100 is relatively far, and can be displayed in red when the distance between the monitored object and the excavator 100 is relatively close. Thus, the operator and the like can understand the positional relationship (distance) between the excavator 100 and the monitored object through the display mode of the indication image 913. Fig.21 Indicative images 921, 922 and Fig. 22 The instruction images 933 to 935 may also be the same.

[0319] In addition, in this example, the monitoring image 910 is displayed in the entire display area of ​​the display device 50, but the monitoring image 910 may be displayed in a manner that leaves a blank in the display area of ​​the display device 50. In this case, the instruction image 913 may be displayed in the blank portion. Fig.21The indication images 921 and 922 may be the same. Furthermore, the indication image 913 may be displayed not only when the monitoring object is detected by the detection unit 304B, but also when the monitoring object is detected by the detection unit 304A. Fig.21 Indicative images 921, 922 and Fig. 22 The indication images 933 to 935 may also be the same. Furthermore, in place of the indication image 913, an indication image of another form that can indicate the presence of a monitoring object detected by the detection unit 304B may be displayed in the monitoring image 910. For example, an indication image having an upward arrow shape may be displayed superimposed on the image captured by the camera 40B at the lower end of the monitoring image 910. Furthermore, for example, an indication image having an isosceles triangle (or equilateral triangle) having an upward vertex may be displayed superimposed on the image captured by the camera 40B at the lower end of the monitoring image 910. At this time, the indication image of the arrow shape or the isosceles triangle may be in the same manner as the indication image 913, such as having a gap portion that exposes the overlapped back image portion as it is, or in the same manner as the indication image 913, such as making the overlapped back image portion transparent and visible. Thus, as in the case of the indication image 913, even when the indication image is displayed superimposed on the monitoring image 910, the operator or the like can appropriately grasp the state of the surroundings of the excavator 100 corresponding to the portion of the monitoring image 910 where the indication image is displayed. The following will be described later. Fig.21 Indicative images 921, 922 and Fig. 22 The instruction images 933 to 935 may also be the same.

[0320] In this way, the monitoring image 910 (through image) is displayed with the indication image 913 indicating the direction in which the monitoring object (operator 911) detected by the detection unit 304B exists when the range displayed in the captured image is viewed from the shovel 100 (upper swing body 3). Thus, the operator can easily understand in which direction the monitoring object detected by the detection unit 304B exists based on the range around the shovel 100 in the captured image displayed on the display device 50 through the indication image 913.

[0321] Then, if Fig.21 As shown, the image captured by the camera 40B (through image) is displayed on the display device 50 as a monitoring image 920 .

[0322] In this example, the monitoring target (worker) is detected by the detection unit 304B outside the range around the shovel 100 corresponding to the monitoring image 920 .

[0323] Indication images 921 and 922 indicating that the monitoring target is detected by the detection unit 304B outside the range around the shovel 100 corresponding to the monitoring image 920 are superimposed on the monitoring image 920. Thus, the operator or the like can recognize that the monitoring target is detected outside the range around the shovel 100 corresponding to the monitoring image 920.

[0324] and Fig. 20 Similar to the indication image 913 of the monitoring image 920, the indication image 921 has an arrowhead shape and is overlapped and displayed near the center in the up-down direction of the left end of the monitoring image 920. The indication image 921 points to the outside of the monitoring image 920, that is, it is equivalent to the left side of the outside of the monitoring image 920. Thus, the indication image 921 can indicate that the monitoring object detected by the detection unit 304B exists in a range (for example, the shooting range of the camera 40R) on the left side of the range around the excavator 100 corresponding to the monitoring image 920 in the circumferential direction. That is, the indication image 921 points to the direction in which the monitoring object detected by the detection unit 304B exists, based on the monitoring image 920. Thus, the operator and the like can intuitively understand in which direction the monitoring object exists based on the monitoring image 920 currently displayed on the display device 50. Therefore, the operator or the like can switch the display content of the display device 50 through the input device 52, so that the image captured by the camera 40R including the monitoring object detected by the detection unit 304B within the imaging range is displayed on the display device 50 as a monitoring image.

[0325] Similar to the indication image 921, the indication image 922 has an arrow shape and is displayed in a superimposed manner near the center in the up-down direction of the right end of the monitoring image 920. The indication image 922 points to the outside of the monitoring image 920, that is, it corresponds to the right side of the outside of the monitoring image 920. Thus, the indication image 922 can indicate that the monitoring object detected by the detection unit 304B exists in a range (for example, the shooting range of the camera 40L) that is further to the right side than the range around the shovel 100 corresponding to the monitoring image 920 in the circumferential direction. That is, the indication image 922 points to the direction in which the monitoring object detected by the detection unit 304B exists with reference to the monitoring image 920. Thus, the operator can intuitively understand in which direction the monitoring object exists with reference to the monitoring image 920 currently displayed on the display device 50. Therefore, the operator can perform an operation of switching the display content of the display device 50 through the input device 52, so that the image captured by the camera 40L including the monitoring object detected by the detection unit 304B in the shooting range is displayed on the display device 50 as the monitoring image.

[0326] In addition, when the monitoring object is detected by the detection unit 304 outside the range around the shovel 100 corresponding to the monitoring image displayed on the display device 50, the display processing unit 301 can maintain the state of displaying the monitoring image on the display device 50 unless the operation of switching the display content is performed through the input device 52. This is because the operator can understand the position where the detected monitoring object exists through the indication images 921 and 922. Thus, the surrounding monitoring device 150 can give priority to the user's intention related to the display content of the display device 50. In addition to the display device 50, other display devices can be provided in the cab 10. In this case, when the monitoring object is detected by the detection unit 304 outside the range around the shovel 100 corresponding to the monitoring image displayed on the display device 50, the display processing unit 301 can automatically display the image captured by the camera 40X corresponding to the position where the monitoring object exists on other display devices. Thus, the convenience of the operator is improved. Furthermore, when the monitoring object is detected by the detection unit 304 outside the range around the shovel 100 corresponding to the monitoring image displayed on the display device 50, the controller 30 may also communicate the content to the operator or the like by other methods instead of communicating the content through the display device 50, or communicate the content to the operator or the like by other methods in addition to communicating the content through the display device 50. For example, when the monitoring object is detected outside the range around the shovel 100 corresponding to the monitoring image displayed on the display device 50, the controller 30 may communicate the content to the operator or the like by outputting a predetermined sound or voice using the voice output device 54.

[0327] In this way, the monitoring image 920 (through image) includes indication images 921 and 922 indicating the direction in which the monitoring object detected by the detection unit 304B exists when the range displayed in the captured image is viewed from the shovel 100 (upper swing body 3). Thus, the operator can easily understand in which direction the monitoring object detected by the detection unit 304B exists based on the range around the shovel 100 in the captured image displayed on the display device 50 through the indication images 921 and 922.

[0328] Then, if Fig. 22 As shown, a monitoring image 930 including a viewpoint conversion image EP based on images captured by the cameras 40B, 40L, and 40R and a shovel image CG is displayed on the display device 50 .

[0329] In this example, a worker 931 wearing a helmet and a reflective vest is shown in the monitoring image 930 at a position corresponding to the rear of the upper swing body 3. In this example, since the worker 931 is wearing both a helmet and a reflective vest, he can be detected by both the detection unit 304A and the detection unit 304B.

[0330] Furthermore, a frame 932 indicating that the worker 931 has been detected by the detection unit 304A is superimposed and displayed on the monitoring image 930. This allows an operator or the like to easily recognize the presence and position of the worker 931 detected by the detection unit 304A.

[0331] In addition, an indication image 933 indicating that the monitoring object (operator 931) detected by the detection unit 304B is present in the monitoring image 910 is displayed in the monitoring image 930. Thus, the operator or the like can recognize that the monitoring object detected by the detection unit 304B is present in the monitoring image 930 currently displayed.

[0332] The instruction image 933 is displayed in a superimposed manner on the outer edge portion of the rear side when the shovel 100 (shovel image CG) is viewed from the viewpoint conversion image EP, that is, the portion of the horizontal image HVP. The instruction image 933 is directed inward from the outer edge of the viewpoint conversion image EP (the portion of the horizontal image HVP), that is, from the outer edge of the viewpoint conversion image EP toward the top of the shovel 100 (shovel image CG). Thus, the instruction image 933 can specifically indicate that the monitoring object (operator 931) detected by the detection unit 304B exists in the monitoring image 930. Furthermore, the instruction image 933 is superimposed on the portion of the viewpoint conversion image EP (horizontal image HVP) corresponding to the rear of the upper swing body 3. Thus, it is possible to indicate that the monitoring object (operator 931) detected by the detection unit 304B exists behind the shovel 100 (upper swing body 3).

[0333] Furthermore, although not detected in this example, when the monitoring object is detected by the detection unit 304B on the left side of the upper swing body 3 (for example, the shooting range of the camera 40L), the indication image 934 (dotted line) is displayed on the portion of the viewpoint conversion image EP (horizontal image HVP) corresponding to the left side of the upper swing body 3. The indication image 934 is displayed overlapping the outer edge portion of the left side when the shovel 100 (shovel image CG) in the viewpoint conversion image EP is viewed, that is, the portion of the horizontal image HVP. The indication image 934 points to the inside from the outer edge of the viewpoint conversion image EP (the portion of the horizontal image HVP), that is, from the outer edge of the viewpoint conversion image EP toward the right side of the shovel 100 (shovel image CG). Thus, the indication image 934 can specifically indicate that the monitoring object detected by the detection unit 304B exists in the monitoring image 930. Furthermore, the indication image 934 is arranged overlapping the portion of the viewpoint conversion image EP (horizontal image HVP) corresponding to the left side of the upper swing body 3. This can indicate that the monitoring target detected by the detection unit 304B is present on the left side of the shovel 100 (the upper swing body 3 ).

[0334] Similarly, although not detected in this example, when the monitoring object is detected by the detection unit 304B on the right side of the upper swing body 3 (for example, the shooting range of the camera 40R), the indication image 935 (dotted line) is displayed on the portion of the viewpoint conversion image EP (horizontal image HVP) corresponding to the right side of the upper swing body 3. The indication image 935 is displayed overlapping the outer edge portion of the right side when the shovel 100 (shovel image CG) is viewed from the viewpoint conversion image EP, that is, the portion of the horizontal image HVP. The indication image 935 points inward from the outer edge of the viewpoint conversion image EP (the portion of the horizontal image HVP), that is, from the outer edge of the viewpoint conversion image EP toward the left side of the shovel 100 (shovel image CG). Thus, the indication image 935 can specifically indicate that the monitoring object detected by the detection unit 304B exists in the monitoring image 930. Furthermore, the indication image 935 is arranged overlapping the portion of the viewpoint conversion image EP (horizontal image HVP) corresponding to the right side of the upper swing body 3. This can indicate that the monitoring target detected by the detection unit 304B is present on the right side of the shovel 100 (the upper swing body 3 ).

[0335] In addition, the instruction images 933 to 935 may be displayed in a blank area around the viewpoint conversion image EP in the monitoring image 930 .

[0336] Furthermore, the same monitoring images as the monitoring images 910, 920, and 930 of this example can be displayed on the output device 230 (display device) of the management device 200 and the output device 330 (display device) of the terminal device 300. Thus, the operator can remotely operate the shovel 100 while confirming the same monitoring image displayed on the display devices of the management device 200 and the terminal device 300. Furthermore, when the shovel 100 is operating by the automatic operation function, the manager of the shovel 100 can monitor the operating conditions of each of the plurality of shovels 100 while observing the same monitoring image displayed on the output device 230 (display device) of the management device 200. Similarly, the manager of the shovel 100 can monitor the operating conditions of each shovel 100 while observing the same monitoring image displayed on the output device 330 (display device) of the terminal device 300, for example.

[0337] [effect]

[0338] Next, the operation of the shovel management system 1000 (the shovel 100 , the management device 200 , and the terminal device 300 ) according to the present embodiment will be described.

[0339] In the present embodiment, when the detection unit 304 detects a predetermined object (monitoring target) outside the range of the displayed surrounding image (through image, viewpoint conversion image EP), the display device 50 displays that the predetermined object is detected outside the range of the displayed surrounding image by the detection unit 304. The same is true for the output device 230 (display device) of the management device 200 and the output device 330 (display device) of the terminal device 300.

[0340] Thus, when the monitoring target detected by the detection unit 304 is not displayed in the displayed surrounding image, the operator, manager, etc. of the shovel 100 can easily understand that the monitoring target is detected outside the range of the displayed surrounding image.

[0341] Furthermore, in this embodiment, when a predetermined object is detected outside the range of the displayed surrounding image by the detection unit 304, the display device 50 may display an image (arrow images 921, 922) indicating the direction in which the predetermined object exists based on the displayed surrounding image. Furthermore, the output device 230 (display device) of the management device 200 and the output device 330 (display device) of the terminal device 300 may be the same.

[0342] Thus, the operator, manager, etc. of the shovel 100 can easily grasp the position of the monitoring object detected by the detection unit 304 outside the range of the surrounding image based on the range of the displayed surrounding image.

[0343] Furthermore, in this embodiment, when a predetermined object is detected within the range of the displayed surrounding image by the detection unit 304, the display device 50 may display an image indicating the direction in which the predetermined object exists within the displayed surrounding image. Furthermore, the output device 230 (display device) of the management device 200 and the output device 330 (display device) of the terminal device 300 may be the same.

[0344] This allows the operator, manager, etc. of the shovel 100 to easily (intuitively) understand that the monitoring target has been detected in the displayed surrounding image.

[0345] Furthermore, in this embodiment, the display device 50 may display an image (indication images 913, 921, 922, 933 to 935) indicating the direction in which the predetermined object detected by the detection unit 304 exists, superimposed on the displayed surrounding image. Furthermore, the output device 230 (display device) of the management device 200 and the output device 330 of the terminal device 300 may be the same.

[0346] This allows the operator, manager, or the like of the shovel 100 to easily recognize that the arrow image 913 or the like is the display content for the displayed through image or viewpoint conversion image EP.

[0347] Furthermore, in this embodiment, the image indicating the direction in which the predetermined object detected by the detection unit 304 exists may include a gap portion that exposes the surrounding image. Furthermore, the output device 230 (display device) of the management device 200 and the output device 330 (display device) of the terminal device 300 may be the same.

[0348] This makes it easy for the operator, manager, or the like of the shovel 100 to understand the state of the periphery of the shovel 100 corresponding to the portion of the through image or the viewpoint conversion image EP on which the instruction image 913 or the like is superimposed.

[0349] Furthermore, in the present embodiment, the image indicating the direction in which the predetermined object detected by the detection unit 304 exists can be displayed in a different manner according to the distance of the predetermined object from the shovel 100. Furthermore, the output device 230 (display device) of the management device 200 and the output device 330 of the terminal device 300 can also be the same.

[0350] Thus, the operator, manager, or the like of the shovel 100 can easily grasp the degree to which the monitoring target detected by the detection unit 304 approaches the shovel 100 through the arrow image 913 or the like.

[0351] Furthermore, in this embodiment, when a predetermined object is detected outside the range of the displayed surrounding image by the detection unit 304, the display device 50 may maintain the state of displaying the surrounding image unless an operation to change the displayed content is performed. Furthermore, the output device 230 (display device) of the management device 200 and the output device 330 of the terminal device 300 may also be the same.

[0352] Thus, the shovel management system 1000 can give priority to the intention of the user such as the operator or the manager regarding the display content of the display device 50 , the output device 230 , or the output device 330 .

[0353] [Transformation / Change]

[0354] Although the embodiments have been described in detail above, the present invention is not limited to the specific embodiments, and various modifications and changes can be made within the scope of the gist described in the claims.

[0355] For example, the perimeter monitoring device 150 according to the above embodiment may be mounted on any construction machine other than the excavator 100. For example, the perimeter monitoring device 150 may be mounted on a lifting magnet device equipped with a lifting magnet, a bulldozer, a wheel loader, an asphalt roller, a forestry machine, etc. as a terminal attachment.

[0356] Furthermore, the function of the detection unit 304 of the above-mentioned embodiment and variant examples (i.e., a detection device (recognition device) capable of identifying the wearing status of safety equipment) may also be provided at a structure, other construction machinery, work vehicle, building, etc. at the work site instead of the excavator 100, or may be provided at a structure, other construction machinery, work vehicle, building, etc. at the work site in addition to the excavator 100. The structure at the work site includes, for example, a door at an entrance, etc. Other construction machinery at the work site includes, for example, a bulldozer, a crane, etc. Work vehicles at the work site include, for example, trucks, etc. Buildings at the work site include, for example, temporary offices, etc. Thus, for example, managers at the work site, etc. can use the detection results of the detection device to grasp the wearing status of safety equipment of workers at the work site within a wide range of the work site. Therefore, managers, etc. can use the detection results of the detection device to perform safety management at the work site. Furthermore, information related to the detection results of the detection device may be recorded in a storage unit. At this time, the detection device and hardware that implements the structure of the storage unit or recording unit (for example, a terminal device or a server device) may be set at the same location, or may be set at different locations in a manner that allows wired or wireless communication.

[0357] For example, Fig.23 It is a block diagram showing an example of the structure of the work site safety management system 2000.

[0358] like Fig.23 As shown, the work site safety management system 2000 includes a control device 2100 , a camera 2200 , a sensor 2300 , an input device 2400 , an output device 2500 , and a communication device 2600 .

[0359] The control device 2100 performs control related to the work site safety management system 2000. In the work site, the control device 2100 can be installed at the same place as the place where the camera 2200 or the sensor 2300 is installed, or relatively close to the place where the camera 2200 or the sensor 2300 is installed, or it can be installed at a place different from the camera 2200 or the sensor 2300. The control device 2100 can be installed, for example, in a structure or building at the work site. In addition, the control device 2100 can also be installed, for example, in a construction machine or work vehicle including the excavator 100 at the work site. When the control device 2100 is installed in the excavator 100, it is equivalent to the controller 30 of the above-mentioned embodiment.

[0360] The functions of the control device 2100 can be implemented by any hardware or any combination of hardware and software. The control device 2100 can be configured with a computer including a CPU, a memory device such as RAM, an auxiliary storage device such as ROM, and an interface device for external input or output.

[0361] The control device 2100 includes, for example, a detection unit 2110, a setting unit 2120, and a log output unit 2150 as functional units, which are implemented by downloading a program installed in an auxiliary storage device to a memory device and executing it by a CPU. In addition, the control device 2100 uses, for example, storage units 2130 and 2140, which are implemented by a storage area specified in an internal memory such as an auxiliary storage device of the control device 2100 or an external storage device capable of communicating with the control device 2100.

[0362] The camera 2200 captures the state of the work site. The camera 2200 is connected to the control device 2100 so as to be able to communicate, and its output (captured image) is input to the control device 2100. The camera 2200 can be installed, for example, in a structure or building at the work site. In addition, the camera 2200 can also be installed, for example, in a construction machine or work vehicle including the excavator 100 at the work site. When installed in the excavator 100, the camera 2200 is equivalent to the imaging device 40 (camera 40X) of the above-mentioned embodiment.

[0363] The sensor 2300 acquires information related to the conditions of the work site. The sensor 2300 is connected to the control device 2100 so as to be able to communicate, and its output is input to the control device 2100. The sensor 2300 may be, for example, a LIDAR like the surrounding information acquisition device 45 (sensor 45X). Furthermore, the sensor 2300 may also be, for example, a millimeter wave radar or an ultrasonic sensor like the surrounding information acquisition device 45 (sensor 45X). The following description will focus on the case where the sensor 2300 is a LIDAR.

[0364] The imaging range of the camera 2200 and the information acquisition range of the sensor 2300 are set to overlap at least partially. Thus, in both the case of using the output of the camera 2200 and the case of using the output of the sensor 2300, the detection unit 2110 can detect the same monitoring object (operator) existing in the overlapping range.

[0365] The input device 2400 receives various inputs from users such as managers, supervisors, and operators at the work site. The input device 2400 is connected to the control device 2100 so as to be able to communicate, and signals corresponding to the various inputs received are input to the control device 2100. The input device 2400 includes, for example, an operation input device that receives operation inputs from users. In addition, the input device 2400 may include, for example, a voice input device, a gesture input device, etc. that receive voice inputs, gesture inputs, etc. from users.

[0366] The output device 2500 is connected to the control device 2100 so as to be able to communicate, and outputs information to users such as managers, supervisors, and workers at the work site based on data input from the control device 2100. The output device 2500 includes, for example, a display device or a lighting device that outputs visual information. In addition, the output device 2500 includes, for example, a sound output device that outputs auditory information.

[0367] The communication device 2600 is installed at the same place as the control device 2100, and communicates with the outside of the place where the control device 2100 is installed (for example, the management device 200, the terminal device 300, etc.).

[0368] The detection unit 2110 detects a worker at the work site based on the output of the camera 2200 or the sensor 2300. The detection unit 2110 includes detection units 2110A and 2110B.

[0369] The detection unit 2110A detects a person (operator) in a specified monitoring area of ​​the work site based on the output (captured image) of the camera 2200 in the same manner as the detection unit 304A of the above-mentioned embodiment. Thus, the detection unit 2110A can detect the operator in the monitoring area and determine the position of the operator, or determine which of the multiple pre-registered registered operators the detected operator corresponds to. Furthermore, the detection unit 2110A can detect an operator wearing a helmet with a relatively high probability (detection accuracy) by identifying a helmet on the image captured by the camera 2200. That is, the detection unit 2110A can identify whether an operator detected based on features other than a helmet or an operator detected by the detection unit 2110B is wearing a helmet.

[0370] The detection unit 2110B detects a person (operator) in the same monitoring area as the case of the detection unit 2110A at the work site based on the output of the sensor 2300 in the same method as the detection unit 304B of the above-mentioned embodiment. Thus, the detection unit 2110B can detect the operator in the monitoring area and determine the position of the operator. In addition, the detection unit 2110B can detect the operator wearing clothes with reflective material with a relatively high probability (detection accuracy) by using the light intensity information. That is, the detection unit 2110B can identify whether the operator detected based on features other than clothes with reflective material or the operator detected by the detection unit 2110A is wearing clothes with reflective material.

[0371] Furthermore, in the case where a worker is detected, the detection unit 2110 can determine the wearing status of multiple (types) of safety equipment of the detected worker. The multiple safety equipment, for example, is the safety equipment that must be worn at the work site or is recommended to be worn at the work site according to the regulations of the management object of the work site safety management system 2000. As described above, the multiple safety equipment, for example, may include a safety helmet or clothes with reflective materials. Furthermore, as described above, the multiple safety equipment, for example, may also include a safety belt, earplugs for noise sites, safety boots, etc. Furthermore, the multiple safety equipment, for example, may also include gloves for arc welding or vibration prevention, dust-proof or gas-proof masks, glasses for dust-proof or light-shielding, welding masks, etc. Furthermore, the multiple safety equipment, for example, may also include an ID tag that outputs an alarm when construction machinery, etc. approaches.

[0372] When a worker is detected, the detection unit 2110 can determine whether the worker is wearing a helmet and clothes with reflective material based on the detection results of the detection units 2110A and 2110B, for example.

[0373] Furthermore, when a worker is detected, the detection unit 2110 may also determine the wearing status of multiple safety equipment by comprehensively considering the output (captured image) of the camera 2200 and the output (light receiving information) of the sensor 2300, for example. Specifically, the detection unit 2110 may determine whether each of the multiple safety equipment is worn based on the image feature quantity of the captured image of the camera 2200 and the three-dimensional shape of the worker's wearing object based on the light receiving information. At this time, a database (hereinafter referred to as a "wearing status judgment database") related to the image feature quantity and light receiving information used to judge whether each of the multiple safety equipment is worn can be constructed in the storage unit 2130. The wearing status judgment database can be pre-constructed by, for example, machine learning (reinforcement learning) using the results of experiments or simulation tests. Then, the detection unit 2110 can compare the image feature quantity of the captured image of the camera 2200 and the light receiving information of the sensor 2300 with the content of the wearing status judgment database to determine whether each of the multiple safety equipment of the detected worker is worn.

[0374] Furthermore, the detection unit 2110 may output information related to the determination result of the wearing status of the plurality of safety equipments of the detected worker to the users including the detected worker through the output device 2500 .

[0375] For example, when it is determined that one of the multiple safety equipment is not worn, the detection unit 2110 can output an alarm (warning) indicating that the safety equipment is not worn through the output device 2500. And, for example, when multiple safety equipment are worn, the detection unit 2110 can output a notification indicating that all safety equipment has been worn to the operator through the output device 2500. Thus, the work site safety management system 2000 can encourage the operator to wear multiple safety equipment that must be worn at the work site or is recommended to be worn at the work site.

[0376] The setting unit 2120 performs settings related to the work site safety management system 2000 based on a predetermined input from the user received through the input device 2400. Thus, the user can use the input device 2400 to cause the control device 2100 to perform settings related to the work site safety management system 2000 in a desired manner.

[0377] The setting unit 2120 can perform settings such as adding or deleting safety equipment to be recognized, for example, based on a predetermined input through the input device 2400. Furthermore, the setting unit 2120 can also perform settings related to the wearing condition determination database, that is, registering a new wearing condition determination database or updating the wearing condition determination database, for example, based on a predetermined input through the input device 2400. The contents of the settings (changes) performed by the setting unit 2120 are registered (stored) in the storage unit 2130.

[0378] The storage unit 2130 stores various settings related to the work site safety management system 2000. As described above, the storage unit 2130, for example, stores settings of various safety equipment that specify recognition targets. Also, as described above, the storage unit 2130, for example, has a database for determining a wearing condition constructed therein.

[0379] The storage unit 2140 stores (records) log information related to the judgment result based on the detection unit 2110, and the judgment result is related to the wearing status of multiple safety equipment of the operator. For example, if the operator is detected by the detection unit 2110, the log information (record) including the detection date and time of the operator, the identification information of the detected operator (for example, the unique ID of each of the multiple registered operators), and the information indicating whether each of the multiple safety equipment is worn is recorded in the storage unit 2140. In this way, a database related to the judgment result of the wearing status of multiple safety equipment based on the record data group can be constructed in the storage unit 2140.

[0380] The log output unit 2150 outputs log information related to the determination result of the wearing status of the plurality of safety equipments of the worker recorded in the storage unit 2140 to the outside.

[0381] The log output unit 2150 outputs the log information of the storage unit 2140 to the user through the output device 2500, for example, based on a predetermined input through the input device 2400. Thus, the user can confirm the history of the wearing status of the safety equipment of the workers in the past through the output device 2500 or analyze the wearing status of the safety equipment in the work site based on the history.

[0382] Furthermore, the log output unit 2150 may output (send) the log information of the storage unit 2140 to the outside of the control device 2100 (e.g., the management device 200, the terminal device 300) through the communication device 2600, for example, according to a predetermined input through the input device 2400. Thus, for example, the user can retrieve the log information to the outside of the management device 200, the terminal device 300, etc. to confirm the history of the wearing status of the safety equipment of the workers in the past or perform analysis related to the wearing status of the safety equipment in the work site based on the history.

[0383] Thus, in this example, the work site safety management system 2000 includes multiple sensors of different types (e.g., camera 2200 and sensor 2300). Furthermore, the detection unit 2110 detects workers at the work site based on the outputs of the multiple sensors, and determines (identifies) the wearing status of each of the multiple safety equipments that are recommended or required to be worn at the work site by the workers.

[0384] Thus, the work site safety management system 2000 can grasp the wearing status of the safety equipment of the workers at the work site. Therefore, the manager of the work site can, for example, promote specific workers who have not worn safety equipment to wear safety equipment based on the judgment result (recognition result) of the wearing status of multiple safety equipment of the workers detected at the work site. Therefore, the work site safety management system 2000 can improve the safety of the work site.

[0385] Furthermore, in this example, the storage unit 2140 may record information related to the determination result (recognition result) of the wearing conditions of the plurality of safety equipments of the worker detected by the detection unit 2110 .

[0386] Thus, the user can confirm the log information (history) of the judgment result related to the wearing status of the safety equipment after the fact, for example, through an external device (e.g., management device 200, terminal device 300, etc.) connected to the output device 2500 or the control device 2100 so as to be able to communicate. Therefore, the user can study measures to improve the wearing rate of safety equipment at the work site by confirming or analyzing the log information after the fact. Therefore, the work site safety management system 2000 can further improve the safety of the work site.

[0387] Finally, this application claims priority based on Japanese Patent Application No. 2019-086881 filed on April 26, 2019, and the entire contents of the Japanese patent application are incorporated herein by reference.

[0388] Explanation of symbols

[0389] 1-lower walking body, 1C, 1CL, 1CR-crawler, 3-upper rotating body, 4-boom, 5-arm, 6-bucket, 10-cab, 30-controller, 40-camera, 40B, 40L, 40R-camera, 45-surrounding information acquisition device, 45BL, 45BR, 45L, 45LF, 45LR, 45R, 45RF, 45RR-sensor, 50-display device, 52-input device, 54-sound output device, 54a-speaker, 54b-travel alarm, 56-hydraulic control valve, 70-communication device, 100-excavator, 150-surrounding monitoring device, 200-management device (information processing device), 210-control device, 220-communication device, 230-output device (display device), 240-input device (operating device) , 300-terminal device (information processing device), 301-display processing unit, 302-setting unit, 303-storage unit, 304-detection unit, 304A-detection unit, 304B-detection unit, 305-safety function control unit, 310-control device, 320-communication device, 330-output device (display device), 340-input device (operating device), 1000-excavator management system, 2000-work site safety management system, 2100-control device, 2110-detection unit, 2120-setting unit, 2130-storage unit, 2140-storage unit, 2150-log output unit, 2200-camera, 2300-sensor, 2400-input device, 2500-output device, 2600-communication device, HMT-safety helmet, RV-reflective vest.

Claims

1. A display device, wherein: When an image captured by one of a plurality of cameras capturing images of the periphery of the excavator is displayed and a predetermined object is detected within a capturing range of another camera among the plurality of cameras different from the one camera, the detection of the predetermined object outside the range of the captured image of the one camera being displayed is displayed, If the specified object does not exist in the captured image, the direction in which the specified object exists is displayed. When the specified object does not exist in the captured image of the one of the multiple cameras, and the specified object exists in the capturing range of the other cameras among the multiple cameras that are different from the one camera, and the captured image of the other cameras is not displayed, the direction in which the specified object exists is displayed in the captured image of the one camera that is not an overhead image.

2. The display device according to claim 1, wherein: When the predetermined object is detected within the imaging range of the other camera, an image indicating the direction in which the predetermined object exists is displayed based on the displayed imaging image of the one camera.

3. The display device according to claim 1 or 2, wherein: When the predetermined object is detected within the displayed imaging range of the one camera, an image showing a direction in which the predetermined object exists within the image captured by the one camera is displayed.

4. The display device according to claim 2, wherein: An image indicating the detected direction in which the predetermined object exists is superimposed and displayed on the image captured by the one camera.

5. The display device according to claim 2, wherein: The image indicating the direction in which the detected predetermined object exists includes a gap portion through which the image captured by the one camera is exposed.

6. The display device according to claim 2, wherein: The image indicating the direction in which the detected predetermined object exists is displayed in a different manner according to the distance of the predetermined object from the shovel.

7. The display device according to any one of claims 1 to 6, wherein: When the predetermined object is detected within the imaging range of the other camera, the image captured by the one camera is kept displayed unless an operation to change the displayed content is performed.

8. An excavator, wherein: The display device according to any one of claims 1 to 7 is provided inside a cab.

9. An information processing device comprising: The display device according to any one of claims 1 to 7; and An operating device is used for remotely operating the excavator.

Images