Excavator

Abstract

To provide a technique for further improving the estimating accuracy of the position of an object to be detected around a shovel.SOLUTION: A shovel 100 includes an undercarriage 1, a super structure 3 turnably mounted on the undercarriage, an image pick-up device 40 mounted on the super structure 3 for picking up an image around the super structure 3, an object detection part 302 for detecting a predetermined object (an object to be monitored) around a shovel 100 from an input image based on an image picked up by the image pick-up device 40, and a position estimation part 303 for estimating the existing position of the predetermined object around the shovel 100 on the basis of information about a detection position on the input image of the predetermined object and information abound a detection range on the input image of the predetermined object.SELECTED DRAWING: Figure 23

Description

Technical Field

[0001] The present disclosure relates to an excavator.

Background Art

[0002] Conventionally, a technique is known in which a predetermined object (for example, a person) within a predetermined range around an excavator (upper swing body) is detected from a captured image of an imaging device mounted on the upper swing body, and a safety function (for example, output of an alarm) is activated (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, for example, only a part of a predetermined object on the captured image of the imaging device may be detected (recognized) by blending in with the background or being hidden by other objects. Therefore, for example, when estimating the position of an actual predetermined object around the excavator from the detection position of the predetermined object on the captured image, the estimation accuracy may decrease, and as a result, the safety function may not be properly activated.

[0005] Therefore, in view of the above problems, an object is to provide a technique capable of further improving the estimation accuracy of the position of an object to be detected around an excavator.

Means for Solving the Problems

[0006] To achieve the above object, in one embodiment of the present disclosure, a lower traveling body, an upper swing body rotatably mounted on the lower traveling body, An imaging device mounted on the upper swing body and imaging the periphery of the upper swing body; A detection unit that detects a predetermined object around the excavator from an input image based on an imaging image of the imaging device; Based on information regarding the vertical detection position of the predetermined object on the input image and information regarding the detection range of the predetermined object on the input image, Whether the relationship between the vertical detection position of the predetermined object on the input image and the detection range of the predetermined object on the input image has consistency when it is assumed that the entire predetermined object is shown in the input image An estimation unit that estimates the position of the predetermined object around the excavator, and is provided with an excavator. An excavator is provided.

Advantages of the Invention

[0007] According to the above-described embodiment, it is possible to further improve the estimation accuracy of the position of the object to be detected around the excavator.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments will be described with reference to the drawings.

[0010] [Outline of Excavator] With reference to FIGS. 1 to 3, the outline of the excavator will be described.

[0011] FIG. 1 and FIG. 2 are a side view and a top view showing an example of the excavator 100 according to the present embodiment. FIG. 3 is a diagram showing an example of the excavator management system SYS.

[0012] As shown in FIGS. 1 and 2, the excavator 100 includes a lower traveling body 1, an upper swing body 3 that is swingably mounted on the lower traveling body 1 via a swing mechanism 2, an attachment AT for performing various operations, and a cabin 10. Hereinafter, the front of the excavator 100 (upper swing body 3) corresponds to the direction in which the attachment extends with respect to the upper swing body 3 when the excavator 100 is viewed in plan (top view) from directly above along the swing axis of the upper swing body 3.

[0013] The lower traveling body 1 includes, for example, a pair of left and right crawlers 1C. The lower traveling body 1 travels the excavator 100 by hydraulically driving each crawler 1C with a left traveling hydraulic motor 1ML and a right traveling hydraulic motor 1MR.

[0014] The upper swing body 3 swings with respect to the lower traveling body 1 when the swing mechanism 2 is hydraulically driven by a swing hydraulic motor 2M.

[0015] The attachment AT includes a boom 4, an arm 5, and a bucket 6 as driven elements.

[0016] The boom 4 is pivotally attached to the center of the front portion of the upper swing body 3 so as to be able to pitch. An arm 5 is pivotally attached to the tip of the boom 4 so as to be able to rotate up and down, and a bucket 6 is pivotally attached to the tip of the arm 5 so as to be able to rotate up and down.

[0017] The bucket 6 is an example of an end attachment. The bucket 6 is used, for example, for excavation work and the like.

[0018] Further, at the tip of the arm 5, depending on the work content and the like, instead of the bucket 6, other end attachments may be attached. The other end attachments may be, for example, other types of buckets such as a large bucket, a slope bucket, a dredging bucket, etc. Further, the other end attachments may be end attachments of types other than buckets such as a stirrer, a breaker, a grapple, etc. Further, between the arm 5 and the end attachment, a preliminary attachment such as a quick coupling or a tilt rotator may be interposed.

[0019] Further, a hook for crane work may be attached to the bucket 6. The hook is rotatably connected at its base end to a bucket pin that connects between the arm 5 and the bucket 6. Thereby, when work other than crane work (lifting work) such as excavation work is performed, the hook is stored in the space formed between the two bucket links.

[0020] The boom 4, the arm 5, and the bucket 6 are each hydraulically driven by a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9 as hydraulic actuators.

[0021] Note that in the excavator 100, some or all of the various hydraulic actuators may be replaced with electric actuators. That is, the excavator 100 may be a hybrid excavator or an electric excavator.

[0022] The cab 10 is an operator's cab and is mounted, for example, on the front left side of the upper swing body 3.

[0023] Note that when the operator does not board the excavator 100 and operate it, and as described later, when the excavator 100 operates solely with a remote operation or a fully automatic operation function, the cab 10 may be omitted.

[0024] Further, as shown in FIG. 3, the excavator 100 may be included in the excavator management system SYS together with the management device 200 and may be capable of communicating with the management device 200 via a predetermined communication line NW. Thereby, the excavator 100 can transmit (upload) various information to the management device 200 or receive various signals (for example, information signals and control signals) from the management device 200.

[0025] The communication line NW includes, for example, a wide area network (WAN: Wide Area Network). The wide area network may include, for example, a mobile communication network having a base station as an end point. Further, the wide area network may include, for example, a satellite communication network using a communication satellite above the excavator 100. Further, the wide area network may include, for example, the Internet network. Further, the communication line NW may include, for example, a local area network (LAN: Local Area Network) of a facility where the management device 200 is installed. The local area network may be a wireless line, a wired line, or a line including both of them. Further, the communication line NW may include, for example, a short-distance communication line based on a predetermined wireless communication method such as WiFi or Bluetooth (registered trademark).

[0026] The excavator management system SYS performs management related to the excavator 100 and support related to the excavator 100 in the management device 200.

[0027] For example, the excavator management system SYS may perform management (monitoring) of the operating state and operation status of the excavator 100 based on various information uploaded from the excavator 100 in the management device 200. Further, as will be described later, the excavator management system SYS may assist in remotely operating the excavator 100 in the management device 200. Also, as will be described later, when the excavator 100 performs work by fully automatic operation, the excavator management system SYS may assist in remotely monitoring the work by the fully automatic operation of the excavator 100 in the management device 200, for example. In this case, the excavator management system SYS may assist in operating the excavator 100 by the intervention of the monitor for the fully automatic operation function in the management device 200.

[0028] The excavator 100 included in the excavator management system SYS may be one or a plurality. Also, the management device 200 included in the excavator management system SYS may be one or a plurality. That is, a plurality of management devices 200 may perform distributed processing related to the excavator management system SYS. For example, a plurality of management devices 200 may communicate with each other with a part of the excavators 100 in charge among all the excavators 100 included in the excavator management system SYS, and execute processing for a part of the excavators 100.

[0029] The management device 200 may be, for example, an on-premises server or a cloud server installed in a management center or the like outside the work site where the excavator 100 performs work. Also, the management device 200 may be, for example, an edge server arranged within the work site where the excavator 100 performs work or in a location relatively close to the work site. Further, the management device 200 may be a stationary terminal device or a portable terminal device (mobile terminal) arranged in a management office or the like within the work site of the excavator 100. The stationary terminal device may include, for example, a desktop PC (Personal Computer). Also, the portable terminal device may include, for example, a smartphone, a tablet terminal, a laptop PC, etc.

[0030] The excavator 100 may drive a driven element such as the lower traveling body 1, the upper swing body 3, and the attachment AT by operating an actuator (for example, a hydraulic actuator) according to the operation of an operator who boards the cabin 10.

[0031] Further, the excavator 100 may be configured to be remotely operable from a predetermined external device (for example, the management device 200). When the excavator 100 is remotely operated, the inside of the cabin 10 may be unmanned. Hereinafter, the operation of the operator includes both the operation of the operating device 26 by the operator in the cabin 10 and the remote operation by an external operator.

[0032] Specifically, the excavator 100 is operated by an input from a user (operator) regarding the actuator of the excavator 100 performed by the management device 200. In this case, the excavator 100 transmits data of an image (hereinafter, "peripheral image") representing the state around the excavator 100, such as an image captured by the imaging device 40 described later or a processed image (for example, a viewpoint-converted image) based on the captured image, to the external device. Then, the peripheral image is displayed on a display device for remote operation (hereinafter, "remote operation display device") provided in the management device 200. Also, various information images (information screens) displayed on the display device 50A (see FIG. 6) in the cabin 10 of the excavator 100 may be similarly displayed on the remote operation display device of the management device 200. Thereby, the operator can remotely operate the excavator 100 while checking the display contents such as the peripheral image representing the state around the excavator 100 and various information images displayed on the remote operation display device. Then, the excavator 100 can operate the actuator and drive driven elements such as the lower traveling body 1, the upper swing body 3, and the attachment AT according to a remote operation signal received from the management device 200 and representing the content of the remote operation.

[0033] Further, the excavator 100 may automatically operate the actuator regardless of the content of the operator's operation. Thereby, the excavator 100 can realize a function of automatically operating at least a part of the driven elements such as the lower traveling body 1, the upper revolving body 3, and the attachment AT, that is, a so-called "automatic operation function" or "Machine Control (MC) function".

[0034] The automatic operation function may include a function of automatically operating a driven element (actuator) other than the driven element (actuator) to be operated in accordance with the operator's operation, that is, a so-called "semiautomatic operation function" or "operation support type MC function". Further, the automatic operation function may include a function of automatically operating at least a part of a plurality of driven elements (hydraulic actuators) on the premise that there is no operator operation, that is, a so-called "fully automatic operation function" or "fully automatic type MC function". In the excavator 100, when the fully automatic operation function is valid, the inside of the cab 10 may be unmanned. Further, the semiautomatic operation function, the fully automatic operation function, etc. may include a mode in which the operation content of the driven element (actuator) to be automatically operated is automatically determined according to a rule defined in advance. Further, the semiautomatic operation function, the fully automatic operation function, etc. may include a so-called "autonomous operation function" in which the excavator 100 autonomously makes various determinations, and the operation content of the driven element (hydraulic actuator) to be automatically operated is determined autonomously according to the determination result.

[0035] [Hardware Configuration of Excavator] Next, with reference to FIG. 4, the hardware configuration of the excavator 100 will be described.

[0036] FIG. 4 is a diagram showing an example of the hardware configuration of the excavator 100.

[0037] Note that in FIG. 4, the path through which mechanical power is transmitted is shown by a double line, the path through which high-pressure hydraulic oil for driving the hydraulic actuator flows is shown by a solid line, the path through which the pilot pressure is transmitted is shown by a broken line, and the path through which the electric signal is transmitted is shown by a dotted line.

[0038] The hydraulic excavator 100 includes respective components such as a hydraulic drive system related to the hydraulic drive of a driven element, an operation system related to the operation of the driven element, a user interface system related to the information exchange with the user, a communication system related to the communication with the outside, and a control system related to various controls.

[0039] <Hydraulic drive system> As shown in FIG. 4, the hydraulic drive system of the hydraulic excavator 100 according to the present embodiment includes, as described above, hydraulic actuators HA that hydraulically drive respective driven elements such as the lower traveling body 1 (left and right crawlers 1C), the upper swing body 3, and the attachment AT. Further, the hydraulic drive system of the hydraulic excavator 100 according to the present embodiment includes an engine 11, a regulator 13, a main pump 14, and a control valve 17.

[0040] As shown in FIG. 6, the hydraulic actuator HA includes traveling hydraulic motors 1ML, 1MR, a swing hydraulic motor 2M, a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, and the like.

[0041] The engine 11 is the prime mover of the hydraulic excavator 100 and is the main power source in the hydraulic drive system. The engine 11 is, for example, a diesel engine that uses light oil as fuel. The engine 11 is mounted, for example, at the rear of the upper swing body 3. The engine 11 rotates at a constant speed at a preset target rotational speed under the direct or indirect control of a controller 30 to be described later, and drives the main pump 14 and the pilot pump 15.

[0042] In addition or alternatively to the engine 11, other prime movers (for example, an electric motor) may be mounted on the hydraulic excavator 100.

[0043] The regulator 13 controls (adjusts) the discharge amount of the main pump 14 under the control of the controller 30. For example, the regulator 13 adjusts the angle of the swash plate (hereinafter, "tilt angle") of the main pump 14 according to a control command from the controller 30.

[0044] The main pump 14 supplies hydraulic oil to the control valve 17 through the high-pressure hydraulic line. The main pump 14 is mounted, for example, at the rear of the upper swing body 3, like the engine 11. The main pump 14 is driven by the engine 11 as described above. The main pump 14 is, for example, a variable displacement hydraulic pump. As described above, under the control of the controller 30, the tilt angle of the swash plate is adjusted by the regulator 13, so that the stroke length of the piston is adjusted and the discharge flow rate (discharge pressure) is controlled.

[0045] The control valve 17 is a hydraulic control device that controls the hydraulic actuator HA according to the operation of the operator's operating device 26, the content of the remote operation, or the operation command related to the automatic operation function output from the controller 30. The operation command corresponding to the automatic operation function may be generated by the controller 30 or by another control device (arithmetic unit) that performs control related to the automatic operation function. The control valve 17 is mounted, for example, at the center of the upper swing body 3. The control valve 17 is connected to the main pump 14 via the high-pressure hydraulic line as described above, and selectively supplies the hydraulic oil supplied from the main pump 14 to each hydraulic actuator according to the operation of the operator or the operation command corresponding to the automatic operation function. Specifically, the control valve 17 includes a plurality of control valves (also referred to as "direction switching valves") that control the flow rate and flow direction of the hydraulic oil supplied from the main pump 14 to each of the hydraulic actuators HA.

[0046] <Operating system> As shown in FIG. 4, the operating system of the excavator 100 according to the present embodiment includes a pilot pump 15, an operating device 26, a hydraulic control valve 31, a shuttle valve 32, and a hydraulic control valve 33.

[0047] The pilot pump 15 supplies pilot pressure to various hydraulic devices via the pilot line 25. The pilot pump 15 is mounted, for example, at the rear of the upper swing body 3, similar to the engine 11. The pilot pump 15 is, for example, a fixed displacement hydraulic pump and is driven by the engine 11 as described above.

[0048] Incidentally, the pilot pump 15 may be omitted. In this case, the hydraulic oil with a relatively low pressure after being decompressed by a predetermined pressure reducing valve from the relatively high pressure hydraulic oil discharged from the main pump 14 is supplied as the pilot pressure to various hydraulic devices.

[0049] The operating device 26 is provided near the driver's seat in the cabin 10 and is used for the operator to operate various driven elements. In other words, the operating device 26 is used for the operator to operate the hydraulic actuators HA that drive the respective driven elements. The operating device 26 includes a pedal device and a lever device for operating the respective driven elements (hydraulic actuators HA).

[0050] For example, as shown in FIG. 4, the operating device 26 is of a hydraulic pilot type. Specifically, the operating device 26 utilizes the hydraulic oil supplied from the pilot pump 15 through the pilot line 25 and the pilot line 25A branched therefrom, and outputs a pilot pressure corresponding to the operation content to the secondary pilot line 27A. The pilot line 27A is connected to one inlet port of the shuttle valve 32 and is connected to the control valve 17 via the pilot line 27 connected to the outlet port of the shuttle valve 32. Thereby, a pilot pressure corresponding to the operation content regarding various driven elements (hydraulic actuators) in the operating device 26 can be input to the control valve 17 via the shuttle valve 32. Therefore, the control valve 17 can drive the respective hydraulic actuators HA according to the operation content of the operator on the operating device 26.

[0051] Furthermore, the operating device 26 may be electric. In this case, the pilot line 27A, the shuttle valve 32, and the hydraulic control valve 33 are omitted. Specifically, the operating device 26 outputs an electric signal (hereinafter, “operation signal”) corresponding to the operation content, and the operation signal is taken into the controller 30. Then, the controller 30 outputs a control command corresponding to the content of the operation signal, that is, a control signal corresponding to the operation content for the operating device 26, to the hydraulic control valve 31. Thereby, a pilot pressure corresponding to the operation content of the operating device 26 is input from the hydraulic control valve 31 to the control valve 17, and the control valve 17 can drive each hydraulic actuator HA according to the operation content of the operating device 26. Also, the control valve (direction switching valve) for driving each hydraulic actuator incorporated in the control valve 17 may be of an electromagnetic solenoid type. In this case, the operation signal output from the operating device 26 may be directly input to the control valve 17, that is, to the electromagnetic solenoid type control valve. Further, when the excavator 100 is remotely operated or operates by an automatic operation function, the operating device 26 may be omitted.

[0052] The hydraulic control valve 31 is provided for each driven element (hydraulic actuator HA) to be operated by the operating device 26. The hydraulic control valve 31 may be provided, for example, in the pilot line 25B between the pilot pump 15 and the control valve 17, and may be configured to be able to change its flow passage area (i.e., the cross-sectional area through which the hydraulic oil can flow). Thereby, the hydraulic control valve 31 can output a predetermined pilot pressure to the secondary pilot line 27B by using the hydraulic oil of the pilot pump 15 supplied through the pilot line 25B. Therefore, as shown in FIG. 4, the hydraulic control valve 31 can indirectly apply a predetermined pilot pressure corresponding to the control signal from the controller 30 to the control valve 17 through the shuttle valve 32 between the pilot line 27B and the pilot line 27. Further, as shown in FIG. 5, the hydraulic control valve 31 can directly apply a predetermined pilot pressure corresponding to the control signal from the controller 30 to the control valve 17 through the pilot line 27B and the pilot line 27. Therefore, the controller 30 can supply a pilot pressure corresponding to the operation content of the electric operating device 26 from the hydraulic control valve 31 to the control valve 17, and realize the operation of the excavator 100 based on the operation of the operator.

[0053] Further, the controller 30 may, for example, control the hydraulic control valve 31 to realize an automatic operation function. Specifically, the controller 30 outputs a control signal corresponding to an operation command related to the automatic operation function to the hydraulic control valve 31 regardless of the presence or absence of an operation of the operating device 26. Thereby, the controller 30 can supply a pilot pressure corresponding to the operation command related to the automatic operation function from the hydraulic control valve 31 to the control valve 17, and realize the operation of the excavator 100 based on the automatic operation function.

[0054] Further, the controller 30 may control, for example, the hydraulic control valve 31 to realize remote operation of the excavator 100. Specifically, the controller 30 outputs, to the hydraulic control valve 31, a control signal corresponding to the content of the remote operation specified by the remote operation signal received from the management device 200 by the communication device 60. Thereby, the controller 30 can cause the hydraulic control valve 31 to supply a pilot pressure corresponding to the content of the remote operation to the control valve 17, and realize the operation of the excavator 100 based on the remote operation of the operator.

[0055] The shuttle valve 32 has two inlet ports and one outlet port, and outputs the hydraulic oil having the higher pilot pressure among the pilot pressures input to the two inlet ports to the outlet port. The shuttle valve 32 is provided for each driven element (hydraulic actuator HA) to be operated by the operating device 26. Further, the shuttle valve 32 is provided for each moving direction of the driven element (for example, the raising direction and the lowering direction of the boom 4). One of the two inlet ports of the shuttle valve 32 is connected to the secondary pilot line 27A of the operating device 26 (specifically, the above-mentioned lever device or pedal device included in the operating device 26), and the other is connected to the secondary pilot line 27B of the hydraulic control valve 31. The outlet port of the shuttle valve 32 is connected to the pilot port of the corresponding control valve of the control valve 17 through the pilot line 27. The corresponding control valve is a control valve that drives the hydraulic actuator that is the operation target of the above-mentioned lever device or pedal device connected to one inlet port of the shuttle valve 32. Therefore, each of these shuttle valves 32 can apply the higher one of the pilot pressure of the secondary pilot line 27A of the operating device 26 and the pilot pressure of the secondary pilot line 27B of the hydraulic control valve 31 to the pilot port of the corresponding control valve. That is, the controller 30 can control the corresponding control valve regardless of the operation of the operator on the operating device 26 by outputting a pilot pressure higher than the secondary pilot pressure of the operating device 26 from the hydraulic control valve 31. Therefore, the controller 30 can control the operation of the driven elements (lower traveling body 1, upper slewing body 3, attachment AT) regardless of the operation state of the operator on the operating device 26, and realize the automatic operation function.

[0056] The hydraulic control valve 33 is provided in a pilot line 27A that connects the operating device 26 and the shuttle valve 32. The hydraulic control valve 33 is configured, for example, to be able to change its flow passage area. The hydraulic control valve 33 operates in response to a control signal input from the controller 30. Thereby, when the operating device 26 is operated by the operator, the controller 30 can forcibly reduce the pilot pressure output from the operating device 26. Therefore, even when the operating device 26 is being operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator corresponding to the operation of the operating device 26. Further, the controller 30 can, for example, reduce the pilot pressure output from the operating device 26 to be lower than the pilot pressure output from the hydraulic control valve 31 even when the operating device 26 is being operated. Therefore, by controlling the hydraulic control valve 31 and the hydraulic control valve 33, the controller 30 can surely cause a desired pilot pressure to act on the pilot port of the control valve in the control valve 17, for example, regardless of the operation content of the operating device 26. Thus, the controller 30 can more appropriately realize the automatic operation function and the remote operation function of the excavator 100, for example, by controlling the hydraulic control valve 33 in addition to the hydraulic control valve 31.

[0057] <User interface system> As shown in FIG. 4, the user interface system of the excavator 100 according to the present embodiment includes an operating device 26, an output device 50, and an input device 52.

[0058] The output device 50 outputs various types of information to the user of the excavator 100 (for example, the operator in the cab 10 or a work vehicle around the excavator 100).

[0059] For example, the output device 50 includes lighting equipment, a display device 50A (see FIG. 6), etc., which output various types of information in a visual manner. The lighting equipment is, for example, a warning light or the like. The display device 50A is, for example, a liquid crystal display, an organic EL (Electroluminescence) display, or the like. The lighting equipment and the display device 50A may be provided, for example, inside the cabin 10 and output various types of information to an operator or the like inside the cabin 10 in a visual manner. Also, the lighting equipment and the display device 50A may be provided, for example, on the side surface of the upper swing body 3 or the like and output various types of information to workers or the like around the excavator 100 in a visual manner.

[0060] Also, for example, the output device 50 includes a sound output device 50B (see FIG. 6) that outputs various types of information in an auditory manner. The sound output device 50B includes, for example, a buzzer, a speaker, or the like. The sound output device 50B may be provided on at least one of the inside and outside of the cabin 10 and output various types of information to an operator inside the cabin 10 or a person (such as a worker) around the excavator 100 in an auditory manner.

[0061] Also, for example, the output device 50 may include a device that outputs various types of information in a tactile manner such as vibration of the operator's seat.

[0062] The input device 52 receives various inputs from the user of the excavator 100, and a signal corresponding to the received input is taken into the controller 30. The input device 52 is provided, for example, inside the cabin 10 and receives inputs from an operator or the like inside the cabin 10. Also, the input device 52 may be provided, for example, on the side surface of the upper swing body 3 or the like and receive inputs from workers or the like around the excavator 100.

[0063] For example, the input device 52 includes an operation input device that receives operation inputs. The operation input device may include a touch panel mounted on the display device, a touch pad installed around the display device, a button switch, a lever, a toggle, a knob switch provided on the operation device 26 (lever device), or the like.

[0064] Further, for example, the input device 52 may include a voice input device that receives a user's voice input. The voice input device includes, for example, a microphone.

[0065] Further, for example, the input device 52 may include a gesture input device that receives a user's gesture input. The gesture input device includes, for example, an imaging device that images the state of a gesture made by the user.

[0066] Further, for example, the input device 52 may include a biological input device that receives a user's biological input. The biological input includes, for example, the input of biological information such as a user's fingerprint and iris.

[0067] <Communication system> As shown in FIG. 4, the communication system of the excavator 100 according to the present embodiment includes a communication device 60.

[0068] The communication device 60 is connected to a communication line NW and communicates with a device (for example, a management device 200) provided separately from the excavator 100. The device provided separately from the excavator 100 may include a device outside the excavator 100 and a portable terminal device brought into the cab 10 by the user of the excavator 100. The communication device 60 includes, for example, a mobile communication module compliant with standards such as 4G (4 th Generation) and 5G (5 th Generation). Further, the communication device 60 may include, for example, a satellite communication module. Further, the communication device 60 may include, for example, a WiFi communication module, a Bluetooth (registered trademark) communication module, or the like.

[0069] <Control system> As shown in FIG. 4, the control system of the excavator 100 according to the present embodiment includes a controller 30. The control system of the excavator 100 according to the present embodiment also includes an operation pressure sensor 29, an imaging device 40, an irradiation device 70, a boom angle sensor S1, an arm angle sensor S2, a bucket angle sensor S3, a machine body attitude sensor S4, and a swing angle sensor S5.

[0070] The controller 30 performs various controls regarding the excavator 100.

[0071] The functions of the controller 30 may be realized by any hardware, or any combination of hardware and software, etc. For example, as shown in FIG. 4, the controller 30 includes an auxiliary storage device 30A, a memory device 30B, a CPU (Central Processing Unit) 30C, and an interface device 30D, which are connected by a bus B1.

[0072] The auxiliary storage device 30A is a non-volatile storage means, which stores the installed programs and also stores necessary files, data, etc. The auxiliary storage device 30A is, for example, a flash memory or the like.

[0073] The memory device 30B, for example, loads the program in the auxiliary storage device 30A so that the CPU 30C can read it when there is a program start instruction. The memory device 30B is, for example, an SRAM (Static Random Access Memory).

[0074] The CPU 30C, for example, executes the program loaded in the memory device 30B and realizes various functions of the controller 30 according to the instructions of the program.

[0075] The interface device 30D is used, for example, as an interface for connecting to the internal communication line of the excavator 100. The interface device 30D may include a plurality of different types of interface devices according to the type of the communication line to be connected.

[0076] A program that realizes various functions of the controller 30 is provided, for example, by a portable recording medium. In this case, the interface device 30D functions as an interface for reading data from the recording medium and writing data to the recording medium. The recording medium is, for example, a dedicated tool connected by a cable detachable from a connector installed inside the cabin 10. Further, the recording medium may be a general-purpose recording medium such as an SD memory card or a USB (Universal Serial Bus) memory. Further, the program may be downloaded from another computer (for example, the management device 200) outside the excavator 100 through a predetermined communication line and installed in the auxiliary storage device 30A.

[0077] Some of the functions of the controller 30 may be realized by other controllers (control devices). That is, the functions of the controller 30 may be realized in a distributed manner by a plurality of controllers. For example, the function related to the image processing of the captured image of the imaging device 40, the function related to the detection of objects around the excavator 100, and the function related to ensuring the safety of the excavator 100 may be realized by different controllers. The function related to the image processing of the captured image of the imaging device 40 includes, for example, the functions of the display processing unit 301 and the image correction unit 306 described later. The function related to the detection of objects around the excavator 100 includes, for example, the functions of the object detection unit 302 and the detection determination unit 307 described later. The function related to ensuring the safety of the excavator 100 includes, for example, the function of the safety control unit 304 described later. Further, the function related to the detection of objects around the excavator 100 may be mounted in a distributed manner on the controller that realizes the function related to the image processing of the captured image of the imaging device 40 and the controller that realizes the function related to ensuring the safety of the excavator 100. Specifically, among the functions related to the detection of objects around the excavator 100, functions with a high degree of relevance to image processing, such as the function of recognizing an object from the captured image of the imaging device 40 (the function of the object detection unit 302), may be mounted on the former controller. On the other hand, among the functions related to the detection of objects around the excavator 100, functions with a high degree of relevance to ensuring the safety of the excavator 100, such as the function of finally determining the presence or absence of the detected object (the function of the detection determination unit 307), may be mounted on the latter controller.

[0078] The operation pressure sensor 29 detects the pilot pressure on the secondary side (pilot line 27A) of the hydraulic pilot type operation device 26, that is, the pilot pressure corresponding to the operation state of each driven element (hydraulic actuator) in the operation device 26. The detection signal of the pilot pressure corresponding to the operation state of each driven element (hydraulic actuator HA) in the operation device 26 by the operation pressure sensor 29 is taken into the controller 30.

[0079] When the operating device 26 is electric, the operation pressure sensor 29 is omitted. This is because the controller 30 can grasp the operating state of each driven element through the operating device 26 based on the operation signal captured from the operating device 26.

[0080] The imaging device 40 acquires an image for interpolating blind spots and hard-to-see locations as viewed by the operator around the excavator 100. The output (imaging image) of the imaging device 40 is captured by the controller 30.

[0081] The imaging device 40 is, for example, a monocular camera, a stereo camera, a depth camera, or the like. Also, the imaging device 40 may acquire three-dimensional data (e.g., point cloud data or surface data) representing the positions and outer shapes of objects around the excavator 100 within a predetermined imaging range (angle of view) based on the imaging image.

[0082] For example, as shown in FIGS. 1 and 2, the imaging device 40 includes a camera 40F that images the front of the upper swing body 3, a camera 40B that images the rear of the upper swing body 3, a camera 40L that images the left side of the upper swing body 3, and a camera 40R that images the right side of the upper swing body 3. Thereby, the operator can visually recognize peripheral images such as the imaging images of the cameras 40B, 40L, 40R and processed images generated based on the imaging images through the display device 50A or the remote operation display device, and confirm the states of the left side, right side, and rear of the upper swing body 3. Also, the operator can remotely operate the excavator 100 while confirming the operation of the attachment AT including the bucket 6 by visually recognizing peripheral images such as the imaging image of the camera 40F and processed images generated based on the imaging image through the remote operation display device. Hereinafter, the cameras 40F, 40B, 40L, 40R may be collectively or individually referred to as "camera 40X".

[0083] The imaging device 40 (camera 40X) includes an image sensor 41 and an image processing engine 42, respectively. Based on the output (electrical signal) of the image sensor 41, the camera 40X generates a captured image by the image processing engine 42 and outputs the generated image (captured image). The image processing engine 42 is a dedicated computer for image processing that includes, for example, a CPU, a memory device, and an auxiliary storage device, etc. By executing a program installed in the auxiliary storage device on the CPU, various image processes are realized. Hereinafter, the function of the image processing engine 42 to generate a captured image is referred to as a captured image generation function, and the captured image of the imaging device 40 means an image generated by the captured image generation function.

[0084] Note that the two-dot chain line in FIG. 2 represents the viewing angle (imaging range) in the top view of the cameras 40F, 40B, 40L, and 40R.

[0085] The irradiation device 70 irradiates a predetermined light to the imaging range of the imaging device 40 under the control of the controller 30. The predetermined light is, for example, visible light. Also, the predetermined light may be, for example, infrared light. As long as the entire imaging range of the imaging device 40 can be irradiated, the irradiation device 70 may be one or a plurality. For example, the irradiation device 70 is provided for each camera 40X and is installed on the upper swivel body 3 so as to be close to the camera 40X.

[0086] Note that the irradiation device 70 may be omitted.

[0087] The boom angle sensor S1 acquires detection information regarding the attitude angle (hereinafter, "boom angle") of the boom 4 with respect to a predetermined reference (for example, a horizontal plane or a state at either end of the movable angle range of the boom 4, etc.). The boom angle sensor S1 may include, for example, a rotary encoder, an acceleration sensor, an angular velocity sensor, a six-axis sensor, an IMU (Inertial Measurement Unit), etc. Also, the boom angle sensor S1 may include a cylinder sensor capable of detecting the extension and retraction position of the boom cylinder 7.

[0088] The arm angle sensor S2 acquires detection information regarding the attitude angle of the arm 5 (hereinafter, "arm angle") with respect to a predetermined reference (for example, a straight line connecting the connection points at both ends of the boom 4, or any state at both ends of the movable angle range of the arm 5, etc.). The arm angle sensor S2 may include, for example, a rotary encoder, an acceleration sensor, an angular velocity sensor, a six-axis sensor, an IMU, etc. Further, the arm angle sensor S2 may include a cylinder sensor capable of detecting the extension and retraction position of the arm cylinder 8.

[0089] The bucket angle sensor S3 acquires detection information regarding the attitude angle of the bucket 6 (hereinafter, "bucket angle") with respect to a predetermined reference (for example, a straight line connecting the connection points at both ends of the arm 5, or any state at both ends of the movable angle range of the bucket 6, etc.). The bucket angle sensor S3 may include, for example, a rotary encoder, an acceleration sensor, an angular velocity sensor, a six-axis sensor, an IMU, etc. Further, the bucket angle sensor S3 may include a cylinder sensor capable of detecting the extension and retraction position of the bucket cylinder 9.

[0090] The machine body attitude sensor S4 acquires detection information regarding the attitude state of the machine body of the excavator 100 including the lower traveling body 1 and the upper slewing body 3. The machine body attitude sensor S4 is mounted on, for example, the upper slewing body 3 and acquires detection information regarding the inclination angle of the upper slewing body 3 with respect to the horizontal plane and the attitude angle around the slewing axis (that is, the direction of the upper slewing body 3 with respect to the ground). The machine body attitude sensor S4 may include, for example, an acceleration sensor (tilt sensor), an angular velocity sensor, a six-axis sensor, an IMU, etc.

[0091] The slewing angle sensor S5 acquires detection information regarding the slewing angle of the upper slewing body 3 with respect to the lower traveling body 1 (that is, the direction of the upper slewing body 3). The slewing angle sensor S5 includes, for example, a potentiometer, a rotary encoder, a resolver, etc.

[0092] Further, for example, the excavator 100 may include a positioning device capable of measuring the absolute position of the machine itself. The positioning device is, for example, a GNSS (Global Navigation Satellite System) sensor. Thereby, the estimation accuracy of the attitude state of the excavator 100 can be improved.

[0093] Further, for example, in addition to the imaging device 40, the excavator 100 may include a distance sensor that detects the distance to an object around the excavator 100. The distance sensor includes, for example, LIDAR (Light Detecting and Ranging), millimeter-wave radar, ultrasonic sensor, infrared sensor, distance image sensor, and the like. Thereby, the controller 30 can detect an object around the excavator 100 using the output of the distance sensor in addition to the output of the imaging device 40, for example.

[0094] Note that some or all of the boom angle sensor S1, arm angle sensor S2, bucket angle sensor S3, machine body attitude sensor S4, and slewing angle sensor S5 may be omitted. For example, when a remote operation function or an automatic driving function is not adopted, it may not be necessary to estimate the attitude state of the attachment AT or the machine body (upper slewing body 3) of the excavator 100. Also, for example, it may be possible to estimate the attitude state of the excavator 100 from the information around the excavator 100 acquired by the imaging device 40 or the distance sensor described later. Specifically, the information around the excavator 100 acquired by the imaging device 40 or the distance sensor described later may include information regarding the surrounding objects, the position and shape of the attachment, etc. seen from the machine body (upper slewing body 3). In this case, the controller 30 can estimate the attitude state of the attachment AT or the machine body (upper slewing body 3) from the information depending on the required accuracy.

[0095] [Hardware Configuration of Management Device] Next, with reference to FIG. 5, the hardware configuration of the management device 200 will be described.

[0096] FIG. 5 is a diagram showing an example of the hardware configuration of the management device 200 according to the present embodiment.

[0097] The functions of the management device 200 are realized by any hardware, or any combination of hardware and software, etc. For example, as shown in FIG. 5, the management device 200 includes an external interface 201, an auxiliary storage device 202, a memory device 203, a CPU 204, a communication interface 206, an input device 207, and a display device 208, which are connected by a bus B2.

[0098] The external interface 201 functions as an interface for reading data from the recording medium 201A and writing data to the recording medium 201A. The recording medium 201A includes, for example, a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disc), a BD (Blu-ray (registered trademark) Disc), an SD memory card, a USB memory, etc. Thereby, the management device 200 can read various data used in processing through the recording medium 201A, store it in the auxiliary storage device 202, or install a program for realizing various functions.

[0099] Note that the management device 200 may acquire various data and programs from an external device through the communication interface 206.

[0100] The auxiliary storage device 202 stores various installed programs and also stores files, data, etc. necessary for various processes. The auxiliary storage device 202 includes, for example, an HDD (Hard Disc Drive), an SSD (Solid State Drive), a flash memory, etc.

[0101] When there is an instruction to start a program, the memory device 203 reads and stores the program from the auxiliary storage device 202. The memory device 203 includes, for example, DRAM (Dynamic Random Access Memory) and SRAM (Static Random Access Memory).

[0102] The CPU 204 executes various programs loaded from the auxiliary storage device 202 into the memory device 203, and realizes various functions related to the management device 200 according to the programs.

[0103] The high-speed arithmetic unit 205 operates in conjunction with the CPU 204 and performs arithmetic processing at a relatively high speed. The high-speed arithmetic unit 205 includes, for example, GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), etc.

[0104] Note that the high-speed arithmetic unit 205 may be omitted according to the speed of the required arithmetic processing.

[0105] The communication interface 206 is used as an interface for communicably connecting to an external device. Thereby, the management device 200 can communicate with an external device such as the excavator 100 through the communication interface 206. Further, the communication interface 206 may have a plurality of types of communication interfaces depending on the communication method with the connected device and the like.

[0106] The input device 207 receives various inputs from the user. For example, the input device 207 includes an input device (remote operation operating device) for an operator to perform remote operations.

[0107] The input device 207 includes, for example, an operation input device that receives mechanical operation inputs from a user. The operation input device includes, for example, buttons, toggles, levers, etc. Further, the operation input device includes, for example, a touch panel mounted on the display device 208, a touch pad provided separately from the display device 208, etc.

[0108] Also, the input device 207 includes, for example, a voice input device that can receive voice inputs from a user. The voice input device includes, for example, a microphone that can collect the user's voice.

[0109] Also, the input device 207 includes, for example, a gesture input device that can receive gesture inputs from a user. The gesture input device includes, for example, a camera that can image the state of the user's gestures.

[0110] Also, the input device 207 includes, for example, a biometric input device that can receive biometric inputs from a user. The biometric input device includes, for example, a camera that can acquire image data containing information about the user's fingerprint or iris.

[0111] The display device 208 displays an information screen or an operation screen for the user. For example, the display device 208 includes the above-described display device for remote operation. The display device 208 is, for example, a liquid crystal display, an organic EL (Electroluminescence) display, or the like.

[0112] [An Example of the Functional Configuration of an Excavator] Next, with reference to FIG. 6, an example of the functional configuration of the excavator 100 will be described.

[0113] FIG. 6 is a functional block diagram showing an example of the functional configuration of the excavator 100 according to the present embodiment.

[0114] As shown in FIG. 6, the controller 30 includes, as functional units, a display processing unit 301, an object detection unit 302, a position estimation unit 303, a safety control unit 304, and an irradiation control unit 305. Further, the controller 30 may include, as a functional unit, an image correction unit 306. The functions of the display processing unit 301, the object detection unit 302, the position estimation unit 303, the safety control unit 304, and the irradiation control unit 305 are realized, for example, by loading a program installed in the auxiliary storage device 30A into the memory device 30B and executing it on the CPU 30C.

[0115] Based on the captured image input from the imaging device 40, the display processing unit 301 causes the display device 50A inside the cabin 10 to display a peripheral image.

[0116] The display processing unit 301 may display a peripheral image corresponding to the imaging range of any one of the cameras 40F, 40B, 40L, and 40R on the display device 50A, or may display the imaging ranges of any two or more of the cameras 40F, 40B, 40L, and 40R on the display device 50A. For example, the display processing unit 301 displays, on the display device 50A, a peripheral image including at least the imaging ranges of the cameras 40B and 40R among the cameras 40F, 40B, 40L, and 40R. This is because the rear and the right side of the upper swing body 3 corresponding to the imaging ranges of the cameras 40B and 40R are likely to be blind spots as viewed by the operator in the cabin 10.

[0117] The object detection unit 302 (an example of a detection unit) detects a predetermined object (hereinafter, "monitored object") of a monitored object around the excavator 100 based on the output (captured image) of the imaging device 40. Specifically, the object detection unit 302 may recognize the monitored object from the captured image of the imaging device 40 and specify the position (area) where the monitored object appears. Further, the object detection unit 302 may recognize the monitored object from an image (hereinafter, "corrected captured image") after the captured image of the imaging device 40 (camera 40X) is corrected by the image correction unit 306, and specify the position (area) where the monitored object appears. That is, the detection of the monitored object may mean recognizing the monitored object reflected in the captured image of the imaging device 40 and specifying the position (area) in the input image including the monitored object. Hereinafter, the image input to the object detection unit 302 (the captured image which is the output of the imaging device 40 itself, or the corrected captured image output from the image correction unit 306) may be referred to as "input image" for convenience.

[0118] The monitored object may include a person such as an operator working around the excavator 100 or a supervisor at the work site. Further, the monitored object may include any object (obstacle) other than a person at the work site. Obstacles other than a person at the work site may include, for example, specific terrains such as holes, ditches, and mounds of earth and sand, road cones, fences, utility poles, temporarily placed materials, and temporary offices at the work site (i.e., obstacles that do not move by themselves). Further, obstacles other than a person at the work site may include, for example, movable obstacles such as other construction machines and work vehicles. The number of types of monitored objects to be detected by the object detection unit 302 may be one or plural. Hereinafter, the case where the monitored object is a person will be mainly described.

[0119] Details of the object detection method by the object detection unit 302 will be described later.

[0120] Further, the object detection unit 302 may have its function switched between ON (enabled) / OFF (disabled) in response to a predetermined input by an operator or the like to the input device 52. Further, the function of the object detection unit 302 may be transferred to the camera 40X, and the detection result may be transmitted to the controller 30. Further, the function of the object detection unit 302 may be transferred to the display device 50A, and the detection result may be transferred to the controller 30. In this case, the captured image of the camera 40X is directly input to the display device 50A. Therefore, in this case, the function of the display processing unit 301 may also be transferred to the display device 50A. Further, the function of the object detection unit 302 may be transferred to the imaging device 40. In this case, the function of the object detection unit 302 is divided into object detection units that detect a monitoring object based on the captured images of the cameras 40F, 40B, 40L, and 40R, respectively, and is incorporated in each of the cameras 40F, 40B, 40L, and 40R (see FIG. 25 described later).

[0121] When a monitoring object is detected from the captured image of the imaging device 40 by the object detection unit 302, the position estimation unit 303 (an example of an estimation unit) estimates the position where the detected monitoring object actually exists, that is, the position of the monitoring object around the excavator 100 (hereinafter, "actual existence position").

[0122] Specifically, the position estimation unit 303 estimates the actual existence position of the monitoring object based on the detection position (detection region) of the monitoring object in the captured image specified by the object detection unit 302. When a plurality of monitoring objects are detected by the object detection unit 302, the position estimation unit 303 estimates the actual existence position for each of the plurality of monitoring objects.

[0123] For example, when the camera 40X is a monocular camera, the position estimation unit 303 identifies a reference point (detection position) in the detection area of the monitored object in the input image, and estimates the direction as seen from the excavator 100 (upper swing body 3) based on the left-right position of the reference point in the input image. Further, the position estimation unit 303 estimates the distance from the excavator 100 to the monitored object (specifically, the distance in the direction along the working plane where the excavator 100 is located) based on the size of the detection area of the monitored object in the input image (for example, the size in the vertical direction). This is because there is a correlation relationship such that the size of the recognized monitored object on the input image becomes smaller as the monitored object moves away from the excavator 100 (upper swing body 3). Specifically, since the monitored object has an assumed size range (for example, the assumed height range of a person), the correlation relationship between the position of the monitored object as seen from the excavator 100 within the assumed size range and its size on the input image can be defined in advance. Therefore, the object detection unit 302 can estimate the actual position of the recognized monitored object based on, for example, a map or conversion formula representing the correlation relationship between the size of the monitored object on the input image and the distance as seen from the upper swing body 3, which is stored in advance in the auxiliary storage device 30A or the like. Thus, the position estimation unit 303 can estimate the position of the recognized monitored object by estimating the direction and distance of the monitored object as seen from the excavator 100 (upper swing body 3).

[0124] Further, for example, when the camera 40X is a monocular camera, the position estimation unit 303 may estimate its actual position by performing projective transformation or the like of the input image onto the plane on which the monitored object exists on the same plane as the lower traveling body 1. Specifically, the position estimation unit 303 may identify a reference point (detection position) representing the contact point of the monitored object with the ground, and calculate the actual position of the monitored object by projective transformation defined according to the installation position and installation angle of the camera 40X with respect to the upper swing body 3 for the reference point. In this case, a certain part (for example, a certain pixel) constituting the input image is one-to-one corresponding to a certain position on the same plane as the excavator 100 (lower traveling body 1).

[0125] Further, for example, when the camera 40X is a stereo camera, the actual position of the monitored object is estimated based on the deviation (parallax) of the reference position in the detection area of the monitored object for each of the two captured images.

[0126] The safety control unit 304 (an example of the control unit performs control related to the functional safety of the excavator 100.

[0127] The safety control unit 304 activates the safety function, for example, when a monitored object is detected within a predetermined range around the excavator 100 by the object detection unit 302. Specifically, the safety control unit 304 may activate the safety function when the actual position (estimated value) of the monitored object estimated by the position estimation unit 303 is within a predetermined range around the excavator 100.

[0128] The safety function may include, for example, a notification function that outputs an alarm or the like to at least one of the inside of the cab 10, the outside of the cab 10, and a remote operator or monitor of the excavator 100 to notify the detection of the monitored object. Thereby, it is possible to prompt the operator inside the cab 10, the worker around the excavator 100, the operator or monitor who performs remote operation or remote monitoring of the excavator 100, etc. to pay attention to the presence of the monitored object in the monitoring area around the excavator 100. Hereinafter, the notification function to the inside of the cab 10 (operator, etc.) is referred to as the "internal notification function", the notification function to the outside of the excavator 100 (worker, etc.) is referred to as the "external notification function", and the notification function to the operator or monitor who performs remote operation or remote monitoring of the excavator 100 is referred to as the "remote notification function" and may be distinguished.

[0129] In addition, the safety function may include, for example, an operation restriction function that restricts the operation of the excavator 100 in response to an operation of the operation device 26, a remote operation, or an operation command corresponding to an automatic driving function. Thereby, the operation of the excavator 100 can be forcibly restricted, and the possibility of approach or contact between the excavator 100 and surrounding objects can be reduced. The operation restriction function may include an operation deceleration function that slows down the operation speed of the excavator 100 compared to normal in response to an operation of the operation device 26, a remote operation, or an operation command corresponding to an automatic driving function. Further, the operation restriction function may include an operation stop function that stops the operation of the excavator 100 and maintains the stopped state regardless of an operation of the operation device 26, a remote operation, or an operation command corresponding to an automatic driving function.

[0130] The safety control unit 304 activates the notification function, for example, when a monitoring object is detected by the object detection unit 302 within a predetermined range (hereinafter, "notification range") around the excavator 100. The notification range is, for example, a range where the distance D from a predetermined part of the excavator 100 is equal to or less than the threshold value Dth1. The predetermined part of the excavator 100 is, for example, the upper swing body 3. Also, the predetermined part of the excavator 100 may be, for example, the bucket 6 or the hook at the tip of the attachment AT. The threshold value Dth1 may be constant regardless of the direction seen from the predetermined part of the excavator 100, or may vary depending on the direction seen from the predetermined part of the excavator 100.

[0131] The safety control unit 304 activates an internal notification function or an external notification function by sound (i.e., an auditory method) for at least one of the inside and outside of the cab 10 by controlling the sound output device 50B. At this time, the safety control unit 304 may vary the pitch, sound pressure, tone color of the output sound, the sounding period when the sound is periodically sounded, the content of the voice, etc. according to various conditions.

[0132] Further, the safety control unit 304 activates, for example, an internal notification function by a visual method. Specifically, the safety control unit 304 may control the display device 50A inside the cabin 10 through the display processing unit 301 to cause the display device 50A to display, together with the surrounding image, an image indicating that a monitoring object has been detected. Further, the safety control unit 304 may emphasize, through the display processing unit 301, the monitoring object reflected in the surrounding image displayed on the display device 50A inside the cabin 10 or the position on the surrounding image corresponding to the detected monitoring object. More specifically, the safety control unit 304 may superimpose and display a frame surrounding the detected monitoring object on the surrounding image displayed on the display device 50A inside the cabin 10, or may superimpose and display a marker at the position on the surrounding image corresponding to the actual position of the detected monitoring object. Thereby, the display device 50A can realize a visual notification function for the operator. Further, the safety control unit 304 may use a warning light, a lighting device, etc. inside the cabin 10 to notify the operator, etc. inside the cabin 10 that a monitoring object has been detected.

[0133] Further, the safety control unit 304 may activate an external notification function by a visual method by controlling an output device 50 (for example, a lighting device such as a headlight or a display device 50A) provided on the side surface of the housing portion of the upper swing body 3 or the like. Further, the safety control unit 304 may activate an internal notification function by a tactile method by controlling, for example, a vibration generating device that vibrates the operator's seat. Thereby, the controller 30 can make the operator, workers and supervisors around the excavator 100 recognize that a monitoring object (for example, a person such as a worker) exists in a relatively close place around the excavator 100. Therefore, the controller 30 can prompt the operator to check the safety status around the excavator 100 or prompt the workers in the monitoring area to evacuate from the monitoring area.

[0134] Further, the safety control unit 304 may activate the remote notification function by, for example, transmitting a command signal indicating the activation of the notification function to the management device 200 through the communication device 60. In this case, when the management device 200 receives the command signal from the excavator 100 through the communication interface 206, it may output an alarm by a visual method or an auditory method through the display device 208 or the like. Thereby, an operator or a monitor who remotely operates or remotely monitors the excavator 100 through the management device 200 can grasp that a monitoring object has entered the notification range around the excavator 100.

[0135] In addition, the remote notification function of the safety control unit 304 may be transferred to the management device 200. In this case, the management device 200 receives information regarding the detection status of the monitoring object by the object detection unit 302 and the estimation result of the actual position of the monitoring object by the position estimation unit 303 from the excavator 100. Then, based on the received information, the management device 200 determines whether or not a monitoring object has entered the notification range, and activates the external notification function when a monitoring object exists within the notification range.

[0136] Further, the safety control unit 304 may vary the notification mode (that is, the way of notification) according to the positional relationship between the monitoring object detected within the notification range and the upper swing body 3.

[0137] For example, when the monitoring object detected within the notification range by the object detection unit 302 is located at a relatively far position with respect to a predetermined part of the excavator 100, the safety control unit 304 may output an alarm with a relatively low urgency level (hereinafter, "alarm at the caution level") that prompts attention to the monitoring object. Hereinafter, the range within the notification range that is relatively far from the predetermined part of the excavator 100, that is, the range corresponding to the alarm at the caution level, may be conveniently referred to as the "caution notification range". On the other hand, when the monitoring object detected within the notification range by the object detection unit 302 is located at a relatively close position with respect to a predetermined part of the excavator 100, the safety control unit 304 may output an alarm with a relatively high urgency level (hereinafter, "alarm at the warning level") that notifies that the monitoring object is approaching a predetermined part of the excavator 100 and the risk level is increasing. Hereinafter, the range within the notification range that is relatively close to the predetermined part of the excavator 100, that is, the range corresponding to the alarm at the warning level, may be referred to as the "warning notification range".

[0138] In this case, the safety control unit 304 may vary the pitch, sound pressure, timbre, beeping period, etc. of the sound output from the sound output device 50B between the alarm at the caution level and the alarm at the warning level. Also, the safety control unit 304 may vary the color, shape, size, presence or absence of blinking, blinking period, etc. of an image indicating that the monitoring object is detected on the display device 50A or an image (e.g., a frame, a marker, etc.) that emphasizes the monitoring object or the position of the monitoring object on the surrounding image displayed on the display device 50A between the alarm at the caution level and the alarm at the warning level. Thereby, the controller 30 can enable an operator or the like to grasp the urgency level, in other words, the degree of approach of the monitoring object to a predetermined part of the excavator 100, based on the difference in the notification sound (alarm sound) output from the sound output device 50B and the notification image displayed on the display device 50A.

[0139] After the operation start of the notification function, if the monitored object detected by the object detection unit 302 is no longer detected within the notification range, the safety control unit 304 may stop the notification function. Further, after the operation start of the notification function, if a predetermined input for canceling the operation of the notification function is received through the input device 52, the safety control unit 304 may also stop the notification function.

[0140] In addition, for example, when a monitored object is detected within a predetermined range (hereinafter referred to as "operation restriction range") around the excavator 100 by the object detection unit 302, the safety control unit 304 activates the operation restriction function. The operation restriction range is set to be the same as, for example, the above-mentioned notification range. Further, the operation restriction range may be set to a range in which, for example, the outer edge is relatively closer to a predetermined part of the excavator 100 than the notification range. Thereby, for example, when the monitored object enters the notification range from the outside, the safety control unit 304 first activates the notification function, and then, when the monitored object enters the inner operation restriction range, the safety control unit 304 can further activate the operation restriction function. Therefore, the controller 30 can gradually activate the notification function and the operation restriction function in accordance with the inward movement of the monitored object within the monitoring area.

[0141] Specifically, when a monitored object is detected within the operation restriction range where the distance D from a predetermined part of the excavator 100 is within the threshold value Dth2 (≤ Dth1), the safety control unit 304 may activate the operation restriction function. The threshold value Dth2 may be constant regardless of the direction seen from the predetermined part of the excavator 100, or may vary depending on the direction seen from the predetermined part of the excavator 100.

[0142] In addition, the operation restriction range includes at least one of an operation deceleration range that slows down the operation speed of the excavator 100 compared to normal for operations of the operation device 26, remote operations, and operation commands corresponding to the automatic driving function, and an operation stop range that stops the operation of the excavator 100 and maintains the stopped state regardless of operations of the operation device 26, remote operations, and operation commands corresponding to the automatic driving function. For example, when both the operation deceleration range and the operation stop range are included in the operation restriction range, the operation stop range is a range close to a predetermined part of the excavator 100 within the operation restriction range. And the operation deceleration range is a range set outside the operation stop range within the operation restriction range.

[0143] The safety control unit 304 activates an operation restriction function that restricts the operation of the excavator 100 by controlling the hydraulic control valve 31. In this case, the safety control unit 304 may restrict the operations of all driven elements (i.e., corresponding hydraulic actuators), or may restrict the operations of some driven elements (hydraulic actuators). Thereby, when there is a monitoring object around the excavator 100, the controller 30 can decelerate or stop the operation of the excavator 100. Therefore, the controller 30 can suppress the occurrence of contact between the monitoring object around the excavator 100 and the excavator 100 or the suspended load. In addition, the safety control unit 304 may activate the operation restriction function (operation stop function) by controlling an electromagnetic switching valve (not shown) of the pilot line 25 and shutting off the pilot line 25.

[0144] In addition, after the activation of the operation restriction function, the safety control unit 304 may stop the operation restriction function when the monitoring object detected by the object detection unit 302 is no longer detected within the operation restriction range. Also, after the activation of the operation restriction function, the safety control unit 304 may stop the operation restriction function when a predetermined input for releasing the activation of the operation restriction function is received through the input device 52. The content of the input for releasing the activation of the notification function for the input device 52 and the content of the input for releasing the activation of the operation restriction function may be the same or different.

[0145] Further, the safety control unit 304 may have its function switched between ON (enabled) / OFF (disabled) in response to a predetermined input by an operator or the like to the input device 52.

[0146] The irradiation control unit 305 performs control regarding the irradiation device 70.

[0147] In addition, as described above, when the irradiation device 70 is omitted, the irradiation control unit 305 is of course also omitted.

[0148] The image correction unit 306 performs predetermined correction on the output (captured image) of the imaging device 40 (camera 40X) and outputs the corrected captured image to the object detection unit 302.

[0149] [Outline of Object Detection Method] Next, with reference to FIGS. 7 to 15, an outline of the object detection method by the object detection unit will be described.

[0150] FIG. 7 is a diagram for explaining a specific example of the object detection method. FIGS. 8 to 13 are diagrams showing the first to sixth examples (images 800 to 1300) of the teacher data images. Specifically, FIGS. 8 to 13 are captured images of a camera of the same type as the camera 40X installed at the same position of an excavator of the same model as the excavator 100, and the side surface of the upper swing body 3 is reflected in the front side (lower end portion) of the captured image. FIGS. 14 and 15 are diagrams showing an example and another example of the operator W.

[0151] The object detection unit 302 detects a monitoring object from the captured image of the camera 40X by simply applying image processing techniques such as shape detection and pattern recognition (template matching).

[0152] Also, as shown in FIG. 7, the object detection unit 302 detects a monitoring object from the captured image of the camera 40X by applying machine learning in addition to, for example, image processing technology. Specifically, the object detection unit 302 uses a pre-trained model LM that has been machine-learned to identify the features of the monitoring object reflected in the input image, and outputs a rectangular frame (hereinafter, "detection frame") representing the area in the input image (the captured image of the camera 40X or its corrected captured image) where the monitoring object is reflected, and a label representing the type of the monitoring object. The label includes, for example, a label indicating the absence of a monitoring object and labels set for each type of monitoring object. Only one label for each type of monitoring object may be set, or multiple labels may be set as described later. Also, when there are multiple types of monitoring objects, a single pre-trained model LM may be configured to output labels for all types of monitoring objects, that is, to be able to detect all types of monitoring objects, or multiple pre-trained models LM that can detect only some of the types of all monitoring objects may be provided. For example, for each type of monitoring object, there may be a pre-trained model LM, and the label for each pre-trained model LM may be composed only of a label indicating the presence of a certain type of monitoring object and a label indicating the absence of that type of monitoring object.

[0153] The pre-trained model LM is generated by applying supervised learning to a base learning model. Specifically, the pre-trained model LM is generated by machine-learning a base learning model with a collection of training data (training dataset) consisting of a combination of an image as input and the correct answer (detection frame and label) as output. Also, the pre-trained model LM may be generated (updated) by additionally training an existing pre-trained model LM with a new training dataset. Naturally, both images containing (reflecting) a monitoring object and images not containing (not reflecting) a monitoring object are adopted as the images as input included in the training dataset.

[0154] The learned model LM is generated by an external device such as the management device 200, and is written to the auxiliary storage device 30A from a predetermined recording medium through the interface device 30D during the manufacture of the excavator 100. Further, the learned model LM may be downloaded from an external device such as the management device 200 to the excavator 100 through a predetermined communication line and registered in the auxiliary storage device 30A of the controller 30. Thereby, the object detection unit 302 can detect a monitoring object using the learned model LM registered in the auxiliary storage device 30A.

[0155] Further, the learned model LM may be updated by installing update data from a predetermined recording medium through the interface device 30D to the auxiliary storage device 30A. Further, the learned model LM may be updated by downloading update data from an external device such as the management device 200 to the excavator 100 through a predetermined communication line and installing it in the auxiliary storage device 30A. Thereby, the object detection unit 302 can detect a monitoring object using the updated and latest learned model LM.

[0156] For example, the object detection unit 302 uses a machine-learned support vector machine (SVM) to detect a monitoring object from an input image based on the tendency of the image feature amount of the monitoring object shown in the image. In this case, the learned model LM includes, as a preprocessing unit, a processing unit that extracts image feature amounts from the captured image of the camera 40X. The image feature amount is, for example, a HOG (Histogram of Oriented Gradients) feature amount.

[0157] Also, for example, the object detection unit 302 detects a monitored object from an input image using a learned model LM obtained by machine learning using a deep neural network (DNN), that is, deep learning (deep neural network). Specifically, the object detection unit 302 may detect a monitored object from an input image using a learned model LM obtained by deep learning using a convolutional neural network (CNN). A CNN is configured by connecting a plurality of combinations of a convolutional layer that performs convolutional processing and a pooling layer that performs pooling processing with an activation function, and the final fully connected layer makes a final determination based on feature amounts (feature maps). The activation function is, for example, ReLU (Rectified Linear Unit). Thereby, the learned model LM can handle the input image as it is without requiring preprocessing.

[0158] For example, the object detection unit 302 generates regions of candidates for a monitored object from an input image using a learned model LM based on CNN, and classifies these candidates into labels, thereby detecting the monitored object. That is, the learned model LM based on CNN may be, for example, a classification model that treats the detection of a monitored object from an input image as a classification problem, generates regions of candidates for the monitored object from a captured image of the camera 40X, and classifies these candidates into labels. The classification model may be, for example, R (Region-based)-CNN or its derivatives (Fast R-CNN, Faster R-CNN, etc.).

[0159] Also, for example, the object detection unit 302 detects a monitored object by simultaneously recognizing the monitored object and specifying its position (area) from an input image using a pre-trained model LM based on a CNN. That is, the pre-trained model LM based on a CNN may be a regression model that, for example, treats the detection of a monitored object from a captured image of a camera 40X as a regression problem and simultaneously recognizes the monitored object and specifies its position (area) from the input image. The regression model may be, for example, YOLO (You Only Look Once), SSD (Single Shot Detector), or the like.

[0160] The pre-trained model LM is generated, for example, by being machine-learned so that a base learning model or an existing pre-trained model LM can detect monitored objects in different poses.

[0161] For example, as shown in FIG. 8, in the image 800 of the training data, a worker W in an upright posture facing the camera is shown. Also, as shown in FIG. 9, in the image 900 of the training data, a worker W in a semi-crouched posture facing the camera is shown. Further, as shown in FIG. 10, in the image 1000 of the training data, a worker W in a crouched posture facing the camera is shown. The pre-trained model LM is machine-learned by a training data set including images in which monitored objects in different poses are shown, such as images 800 to 1000, so that monitored objects (in this example, people) in different poses can be detected from the input image.

[0162] Also, the pre-trained model LM may be generated, for example, by being machine-learned so that a base learning model or an existing pre-trained model LM can detect monitored objects in different orientations.

[0163] For example, as shown in FIG. 11, in the image 1100 of the teacher data, an operator W in an upright posture facing horizontally (right) with respect to the camera is shown. Also, as shown in FIG. 12, in the image 1200 of the teacher data, an operator W in a semi - crouched posture with his back to the camera is shown. Further, as shown in FIG. 13, in the image 1300 of the teacher data, an operator W in a crouched posture facing horizontally (left) with respect to the camera is shown. The learned model LM is machine - learned by a teacher data set including images in which monitoring objects with different orientations are shown, such as the images 800, 1100, the images 900, 1200, and the images 1000, 1300, so that monitoring objects with different orientations can be detected from the input image.

[0164] Also, the learned model LM may be generated, for example, by being machine - learned so that a base learning model or an existing learned model LM can detect monitoring objects with at least one of the orientation and the posture being different from each other. Specifically, the learned model LM is machine - learned by a teacher data set including images in which monitoring objects with different orientations and postures are shown, such as the images 800 to 1300, so that monitoring objects with different orientations and postures can be detected from the input image.

[0165] Also, when the monitoring object is a person, the learned model LM may be generated, for example, by being machine - learned so that it can detect (recognize) a person wearing an item of clothing with a relatively high wearing frequency by an operator or the like around the excavator 100 or the item of clothing. Thereby, even in the case of an imaging image in which it is difficult to distinguish between work clothes and the background, for example, the controller 30 can appropriately recognize the characteristics of the item of clothing, detect the person and the item of clothing, and grasp the presence of the person near the excavator 100.

[0166] For example, as shown in FIG. 14, the learned model LM may be generated by being machine - learned to detect (recognize) a person wearing a helmet (worker W) and the worn helmet HMT. Specifically, the learned model LM is machine - learned by a teacher dataset including a large number of images in which a worker W wearing a helmet HMT is shown, so that a person wearing a helmet and the worn helmet can be detected from the input image.

[0167] Also, for example, as shown in FIG. 15, the learned model LM may be generated by being machine - learned to detect (recognize) a person wearing highly visible safety clothing RV (worker W) and the worn highly visible safety clothing RV. The highly visible safety clothing RV is configured so that the visibility from around the wearer is relatively high. For the highly visible safety clothing RV, for example, a retro - reflective material is attached. Also, for the highly visible safety clothing RV, for example, a fluorescent fabric colored with fluorescent yellow or fluorescent green is used. Specifically, the learned model LM is machine - learned by a teacher dataset including a large number of images in which a worker W wearing highly visible safety clothing RV is shown, so that a person wearing highly visible safety clothing and the worn highly visible safety clothing can be detected from the input image.

[0168] Also, for example, as shown in FIG. 15, the learned model LM may be machine - learned to detect (recognize) a person (worker W) wearing both a helmet HMT and highly visible safety clothing RV, and both the worn highly visible safety clothing RV and the helmet HMT. In this case, the learned model LM may be able to output both a label indicating the presence of a helmet and a label indicating the presence of highly visible safety clothing. Specifically, the learned model LM is machine - learned by a teacher dataset including a large number of images in which a worker W wearing a helmet HMT and highly visible safety clothing RV is shown, so that a person wearing a helmet and highly visible safety clothing and the worn helmet and highly visible safety clothing can be detected from the input image.

[0169] In addition, the teacher dataset includes, for example, images in which a monitored object is shown and which are captured by a camera of the same model as the camera 40X. Further, the teacher dataset may include, for example, as shown in FIGS. 8 to 13, images in which a monitored object is shown and which are captured by a camera of the same model as the camera 40X and which are installed in substantially the same position and in substantially the same posture on a shovel of the same model as the shovel 100. Thereby, the learned model LM is machine-learned so as to be able to detect the monitored object from the captured image of the camera 40X in consideration of the way the monitored object appears in the captured image of the camera 40X. Therefore, the object detection unit 302 can detect the monitored object with higher accuracy.

[0170] In addition, the teacher dataset includes, for example, images in which a monitored object is shown in each of a plurality of different types of backgrounds. For example, the plurality of types of backgrounds include a dirt ground, an asphalt ground, a forest, a house, urban buildings, and the like. Thereby, the learned model LM is machine-learned so as to be able to detect the monitored object from the input image in consideration of the appearance of the monitored object due to the difference in the background. Therefore, the object detection unit 302 can detect the monitored object with higher accuracy.

[0171] In addition, the teacher dataset includes, for example, images in which a monitored object is shown and which are captured at each of a plurality of different time zones. The plurality of time zones include, for example, a morning time zone (for example, 6:00 to 10:00), a daytime time zone (for example, 10:00 to 15:00), an evening time zone (for example, 16:00 to 19:00), and a nighttime time zone (for example, 19:00 to 6:00 the next day). Thereby, the learned model LM is machine-learned so as to be able to detect the monitored object from the input image in consideration of the appearance of the monitored object due to the difference in the time zone. Therefore, the object detection unit 302 can detect the monitored object with higher accuracy.

[0172] In addition, the teacher dataset includes, for example, images in which a monitored object is captured under a plurality of different lighting conditions during the night time. The lighting conditions include, for example, the level of illuminance of the lighting and the color of the lighting. As a result, the learned model LM is machine-learned so as to be able to detect the monitored object from the input image while considering how the monitored object looks due to differences in lighting conditions. Therefore, the object detection unit 302 can detect the monitored object with higher accuracy.

[0173] In addition, the teacher dataset includes, for example, images in which a monitored object is captured under a plurality of different types of weather conditions. The plurality of types of weather conditions include, for example, clear, cloudy, rainy, snowy, and the like. As a result, the learned model LM is machine-learned so as to be able to detect the monitored object from the input image while considering how the monitored object looks due to differences in weather. Therefore, the object detection unit 302 can detect the monitored object with higher accuracy.

[0174] In addition, the teacher dataset includes, for example, a plurality of images in which a monitored object is captured and the positional relationship between the light source and the imaging range is different from each other. The light source is, for example, the sun or night-time lighting. The plurality of images include images corresponding to each of front light, semi-front light, side light, backlight, etc. in terms of the positional relationship between the light source and the imaging range. In addition, the plurality of images include, for example, images in the same front light, side light, backlight, etc. state but with different heights (elevation angles) of the sun from the ground. As a result, the teacher dataset is machine-learned so as to be able to detect the monitored object from the input image while considering how the monitored object looks due to differences in the position of the light source. Therefore, the object detection unit 302 can detect the monitored object with higher accuracy.

[0175] [Specific Example of Object Detection Method] Next, with reference to FIGS. 16 to 18, a specific example of the method for detecting a monitored object by the object detection unit 302 will be described.

[0176] [First Example] In this example, the object detection unit 302 detects a person (operator) around the excavator 100 by detecting (recognizing) a highly visible safety vest from the input image. As a result, the object detection unit 302 can more easily detect a person even from an input image in which it is difficult to distinguish the work clothes of a person around the excavator 100 from the background, and as a result, can detect a person as a monitoring object with higher accuracy.

[0177] Specifically, the object detection unit 302 may detect (recognize) a person wearing a highly visible safety vest from the input image, or may detect (recognize) the highly visible safety vest itself worn by a person, or may detect (recognize) both of them. As a result, when the object detection unit 302 detects a person wearing a highly visible safety vest or a highly visible safety vest (worn by a person), it can be determined that a person as a monitoring object has been detected.

[0178] For example, the object detection unit 302 detects a person around the excavator 100 by recognizing at least one of the shape, color (e.g., fluorescent yellow or fluorescent green), and brightness of the retroreflective material of the highly visible safety vest. Specifically, the object detection unit 302 may recognize at least one of the shape, color, and brightness of the retroreflective material of the highly visible safety vest by applying shape detection, pattern recognition, etc. to the shape, color, and brightness of the highly visible safety vest. Also, as described above, the object detection unit 302 may recognize at least one of the shape, color, and brightness of the retroreflective material of the highly visible safety vest by using a learned model LM based on a teacher dataset including a large number of images in which a person (operator) wearing a highly visible safety vest is shown.

[0179] Further, in this example, the imaging device 40 (camera 40X) may acquire an imaging image of the surroundings of the excavator 100 while visible light is being irradiated from the irradiation device 70. In this case, the irradiation device 70 may constantly irradiate visible light while the camera 40X is operating (power is on) under the control of the controller 30 (irradiation control unit 305), or may irradiate visible light in synchronization with the imaging timing of the camera 40X. As a result, when there is a person (worker) wearing highly visible safety clothing within the imaging range of the imaging device 40, the retroreflective material or fluorescent color of the highly visible safety clothing will be more clearly reflected in the imaging image of the camera 40X. Therefore, the object detection unit 302 can more easily detect (recognize) the highly visible safety clothing from the input image (the imaging image of the camera 40X or the corrected imaging image by the image correction unit 306), and as a result, can detect a person as a monitoring object with higher accuracy.

[0180] Further, in this example, the imaging device 40 (camera 40X) may also acquire an imaging image of the surroundings of the excavator 100 while infrared light is being irradiated from the irradiation device 70. In this case, the irradiation device 70 may constantly irradiate infrared light while the camera 40X is operating (power is on) under the control of the controller 30 (irradiation control unit 305), or may intermittently (periodically) irradiate infrared light in synchronization with the imaging timing of the camera 40X. As a result, when there is a person (worker) wearing highly visible safety clothing within the imaging range of the imaging device 40, the outline of the retroreflective material of the highly visible safety clothing will be more clearly reflected in the imaging image of the camera 40X. Therefore, the object detection unit 302 can more easily detect (recognize) the highly visible safety clothing from the input image, and as a result, can detect a person as a monitoring object with higher accuracy. Also, in the case of infrared light, since it cannot be directly perceived by the human eye like visible light, the impact on people around the excavator 100 can be suppressed.

[0181] Also, in this example, the irradiation device 70 may switch between a state of constantly (continuously) irradiating infrared light and a state of intermittently irradiating infrared light in synchronization with the imaging timing of the camera 40X under the control of the controller 30. For example, the irradiation control unit 305 determines whether there is an influence of infrared light on the peripheral image displayed on the display device 50A or the display device 208. Then, the irradiation control unit 305 may switch the irradiation state of the infrared light from the irradiation device 70 according to the determination result. The influence of infrared light means, for example, that when the camera 40X captures near-infrared light reflected from a subject (e.g., a highly visible safety suit), the peripheral image of the display device 50A becomes reddish. The irradiation control unit 305 may determine the presence or absence of the influence of infrared light by image analysis. Also, the irradiation control unit 305 may determine that there is an influence of infrared light by receiving a predetermined input from the user (operator or monitor) through the input device 52 or the communication device 60. Thereby, when the operator or monitor recognizes that the peripheral image of the display device 50A or the display device 208 is in a reddish state, they can notify the controller 30 of that state by making a predetermined input through the input device 52 or the input device 207.

[0182] Specifically, the irradiation control unit 305 usually, that is, when there is no influence of infrared light on the peripheral image displayed on the display device 50A or the display device 208, continuously irradiates infrared light from the irradiation device 70. On the other hand, when the peripheral image displayed on the display device 50A or the display device 208 is in a state where there is an influence of infrared light, the irradiation control unit 305 intermittently irradiates infrared light in synchronization with the imaging timing of the camera 40X. Thereby, it is possible to achieve a balance between suppressing the control load of the irradiation device 70 by the controller 30 and suppressing the influence of infrared light on the peripheral image such as the display device 50A.

[0183] <Second Example> FIG. 16 is a diagram showing an example of an imaging image of the imaging device 40 (camera 40X). FIG. 17 is a diagram showing an example of an image after correction (corrected imaging image) for the imaging device 40 (camera 40X) by the image correction unit 306.

[0184] In this example, the object detection unit 302 detects a monitoring object using, as an input image, a corrected image (corrected captured image) of the captured image of the imaging device 40 by the image correction unit 306.

[0185] For example, when viewed from the operator in the cab 10, it is difficult to directly visually recognize the rear, as well as the left and right sides, of the upper swing body 3. Also, for example, a remotely operating operator cannot directly visually recognize the surroundings including the front of the cab 10. Therefore, it is necessary to cover a 360-degree range of the surroundings when the excavator 100 is viewed from above with four cameras 40F, 40B, 40L, and 40R. As a result, for example, as shown in FIG. 2, the four cameras 40F, 40B, 40L, and 40R may be wide-angle cameras with a very large horizontal angle of view (field of view) set, and wide-angle distortion (volume distortion) may occur in the captured images thereof.

[0186] Also, for example, as shown in FIGS. 1 and 2, the camera 40X is installed on the upper part of the upper swing body 3. Therefore, the camera 40X needs to detect monitoring objects including monitoring objects near the ground from a relatively high position, and the optical axis of the camera 40X is set obliquely downward. As a result, distortion due to perspective may occur in the captured image of the camera 40X.

[0187] For example, as shown in FIG. 16, workers W1 to W3 are respectively shown in the central part, the left end part, and the right end part of the captured image of the camera 40X. The worker W1 in the central part of the captured image of the camera 40X is shown such that the vertical axis with respect to the subject substantially coincides with the vertical axis of the captured image. The term "substantially" is intended to allow for manufacturing errors of the excavator 100 and installation errors of the camera 40X. On the other hand, in the captured image of the camera 40X, the vertical axes of the workers W2 and W3 are inclined to the left and right sides with respect to the vertical axis, respectively. This is because the captured image of the camera 40X is distorted such that the vertical axes of the subjects at the left and right ends are inclined toward the end sides with respect to the vertical axis.

[0188] On the other hand, in this example, the image correction unit 306 corrects the captured image of the camera 40X so that the difference between the vertical axis of the captured image of the camera 40X and the vertical axis for the subject becomes small, and outputs the corrected captured image.

[0189] For example, as shown in FIG. 17, the image correction unit 306 generates a corrected captured image by inclining only the latter image regions of the central portion and both left and right end portions of the captured image of the camera 40X inward by a predetermined amount respectively. Thereby, for the operators W2 and W3 in the corrected captured image, the difference between the vertical axis in the image and the vertical axis for the subject becomes relatively small.

[0190] Also, the image correction unit 306 may correct the distortion of the captured image of the camera 40X itself using a known method such as projective transformation. Thereby, the image correction unit 306 can reduce the difference between the vertical axis in the corrected captured image and the vertical axis for the subject.

[0191] In this example, the object detection unit 302 is configured to be able to detect only the former of the monitoring objects with a relatively small difference between the vertical axis in the input image and the vertical axis for the subject and the monitoring objects with a relatively large difference. The monitoring objects with a relatively large difference between the vertical axis in the image and the vertical axis for the subject are, for example, the monitoring objects reflected in both left and right end portions of the captured image of the camera 40X like the operators W2 and W3 in FIG. 16. Also, the monitoring objects with a relatively small difference between the vertical axis in the image and the vertical axis for the subject are, for example, the monitoring objects reflected in both left and right end portions of the captured image of the camera 40X like the operator W1 in FIG. 16. Also, the monitoring objects with a relatively small difference between the vertical axis in the image and the vertical axis for the subject are, for example, the monitoring objects reflected in the corrected captured image of the camera 40X like the operators W1 to W3 in FIG. 17. Thereby, the object detection unit 302 can detect the monitoring objects using the corrected captured image as the input image.

[0192] Specifically, the object detection unit 302 may detect such a monitored object from the input image by applying shape detection, pattern recognition, etc. to the monitored object for which the difference between the vertical axis in the input image and the vertical axis for the subject is relatively small. Further, the object detection unit 302 may use a learned model LM based on a teacher data set including only the former of the image in which a monitored object with a relatively small difference between the vertical axis in the input image and the vertical axis for the subject is reflected and the image in which a monitored object with a relatively large difference is reflected. Thereby, since the object detection unit 302 only needs to recognize only the monitored object with a relatively small difference between the vertical axis in the image (corrected captured image) and the vertical axis for the subject from the input image (corrected captured image), the monitored object can be detected with higher accuracy. Further, when the learned model LM is used, since it only needs to learn the features of only the monitored object with a relatively small difference between the vertical axis in the image (corrected captured image) and the vertical axis for the subject from the image (corrected captured image), the efficiency of machine learning can be improved.

[0193] Also, when the learned model LM is used, as described above, for example, images in which a monitoring object is shown, captured by a camera of the same model as the camera 40X, are used for the teacher dataset. Also, as described above, for example, as shown in FIGS. 8 to 13, images in which a monitoring object is shown, captured by a camera of the same model as the camera 40X, installed in substantially the same position and in substantially the same posture as the excavator 100 of the same model, may be used for the teacher dataset. In this case, the captured images of cameras of the same type as the camera 40X are corrected by the same function as the image correction unit 306, and the corrected images may be included in the teacher dataset. Thereby, the learned model LM can detect the monitoring object in consideration of the way the monitoring object appears in the captured image of the camera 40X, and is machine-learned so as to be able to detect only monitoring objects in which the difference between the vertical axis in the image and the vertical axis for the subject is relatively small. Therefore, the object detection unit 302 can detect the monitoring object with higher accuracy. Also, in this case, the teacher dataset may include corrected images corresponding to, for example, captured images in which the monitoring object is shown in the central portion in the left-right direction and captured images in which the monitoring object is shown at at least one end in the left-right direction. Thereby, the learned model LM is machine-learned so as to be able to detect both the monitoring object without correction in the central portion in the left and right of the input image (corrected captured image) and the monitoring object with correction at the end in the left-right direction in the input image. Therefore, the object detection unit 302 can detect the monitoring object with higher accuracy.

[0194] <Example 3> FIGS. 18 to 21 are diagrams showing the first to fourth examples of captured images in which only a part of the whole (full body) of the monitoring object (person) is shown.

[0195] For example, as shown in FIGS. 18 to 21, in the captured image of camera 40X, only a part including the lower part (lower body) of the whole body of a person (worker W) as a monitoring object located at both ends of the imaging range in the left-right direction may be shown. Specifically, in FIG. 18 (the first example), the lower body and a part of the upper body (waist part) of worker W are shown at a position relatively close to the upper swivel body 3 at the right end of the captured image of camera 40X. In FIG. 19 (the second example), the lower body and a part of the upper body (chest, waist, arms, and hands excluding the head) of worker W are shown at a position relatively close to the upper swivel body 3 at the left end of the captured image of camera 40X. In FIG. 20 (the third example), a part of the lower body (lower legs and feet) of worker W is shown at a position relatively close to the upper swivel body 3 at the right end of the captured image of camera 40X. In FIG. 21 (the fourth example), a part of the lower body (lower legs and feet) of worker W is shown at a position relatively far from the upper swivel body 3 at the left end of the captured image of camera 40X. This is because camera 40X has an optical axis obliquely downward from the upper surface of the upper swivel body 3 toward the ground, and the upper body of worker W as a monitoring object is out of the three-dimensional imaging range of camera 40X.

[0196] Also, when there is an obstacle between the monitoring object and camera 40X, the lower part (lower body) of the monitoring object (person) may be hidden by the obstacle, and only a part including the upper part (upper body) of the whole of the monitoring object may be shown in the captured image of camera 40X.

[0197] In contrast, in this example, the object detection unit 302 detects (recognizes) the monitoring object (person) by detecting (recognizing) only a predetermined part of the whole body of the monitoring object (person) from the input image. The predetermined part is, for example, the lower part (lower body) or the upper part (upper body) of the monitoring object (person) as described above. Thereby, even when only a part including a predetermined part of the whole of the monitoring object is shown in the input image, the object detection unit 302 can detect the monitoring object. Therefore, the object detection unit 302 can detect the monitoring object with higher accuracy.

[0198] Specifically, the object detection unit 302 may detect a predetermined part of the monitored object from the input image by applying shape detection, pattern recognition, etc. to the predetermined part of the monitored object. Further, the object detection unit 302 may detect a predetermined part of the monitored object using a learned model LM based on a teacher dataset including an image in which only a part including a predetermined part of the whole of the monitored object is shown.

[0199] Also, the object detection unit 302 may be configured to be able to detect (recognize) both the whole of the monitored object and a predetermined part of the whole of the monitored object. Thereby, the object detection unit 302 can determine that the monitored object has been detected, for example, when either the whole of the monitored object or a predetermined part of the monitored object is detected. Therefore, the object detection unit 302 can detect the monitored object with higher accuracy.

[0200] For example, the object detection unit 302 uses a learned model LM based on a teacher dataset including an image in which the whole of the monitored object is shown and an image in which only a part including a predetermined part of the whole of the monitored object is shown, to detect both the whole of the monitored object and a predetermined part of the monitored object. In this case, the object detection unit 302 may output a label corresponding to the detected monitored object from among a plurality of labels including a label representing the whole of the monitored object and a label representing a predetermined part of the whole of the monitored object. Then, when the label of the detected monitored object is output as either a label representing the whole of the monitored object or a label representing a predetermined part of the whole of the monitored object from the learned model LM, the object detection unit 302 may determine that the monitored object has been detected.

[0201] Also, when the learned model LM is used, for the teacher dataset, for example, an image captured by a camera of the same model as the camera 40X, in which only a part including the lower part of the entire monitored object is shown at at least one of the left end and the right end, is used. Also, for the teacher dataset, for example, an image captured by a camera of the same model as the camera 40X, which is installed in substantially the same position and in substantially the same posture as the shovel 100 of the same model, and in which only a part including the lower part of the entire monitored object is shown at at least one of the left end and the right end, may be used. Thereby, the learned model LM is machine-learned so as to be able to detect the lower part (for example, the lower body of a person) of the entire monitored object in consideration of the way the monitored object appears in the captured image of the camera 40X. Therefore, the object detection unit 302 can detect a predetermined part of the entire monitored object with higher accuracy.

[0202] Also, the object detection unit 302 may detect the entire monitored object based on the image at the center part in the left-right direction of the input image, and may detect only the lower part (for example, the lower body of a person) of the entire monitored object based on the images at both ends in the left-right direction of the input image. This is because, as described above, in the input image, there may be a case where only the lower part (for example, the lower body of a person) of the entire monitored object is shown at both ends in the left-right direction.

[0203] In addition, when the object detection unit 302 detects (recognizes) a predetermined part of the monitored object in both parts of the overlapping area of the imaging ranges based on the respective captured images of two cameras 40X whose imaging ranges partially overlap with each other, it may be configured to detect the monitored object in that area. The two cameras 40X whose imaging ranges partially overlap with each other correspond to, for example, any one combination of cameras 40F and 40R, cameras 40R and 40B, cameras 40B and 40L, and cameras 40L and 40F. Thereby, even if the object detection unit 302 erroneously detects a predetermined part of the monitored object based on one camera 40X in a part of the overlapping imaging range of the two cameras 40X, and does not detect a predetermined part of the monitored object based on the other camera 40X, it can be determined as undetected. Therefore, the object detection unit 302 can suppress false detection when detecting a predetermined part of the entire monitored object.

[0204] [Specific Example of Method for Estimating Actual Position of Monitored Object] Next, with reference to FIGS. 22 to 24, a specific example of a method for estimating the actual position of the detected monitored object will be described.

[0205] FIGS. 22 to 24 are diagrams showing an example, another example, and still another example of the detection state of the monitored object (person) by the object detection unit 302. Specifically, FIGS. 22 to 24 are diagrams showing a detection frame FR representing the detection range of the monitored object in the input image when the monitored object (operator W) is detected from the input image by the object detection unit 302.

[0206] The images in FIGS. 22 to 24 may be displayed on, for example, the display device 50A or the remote operation display device when the monitored object (operator W) is detected by the object detection unit 302. Thereby, the operator or the monitor can grasp the detection status of the monitored object (operator W) around the excavator 100 while viewing the surrounding image (the captured image of the camera 40X).

[0207] In the situations of FIGS. 22 and 23, the operator W is standing upright in the same place relatively close to the upper slewing body 3, facing the camera 40X frontally. Also, in the situation of FIG. 24, the operator W is standing upright in a place relatively far from the upper slewing body 3, facing the camera 40X frontally.

[0208] For example, in the situation of FIG. 22, the object detection unit 302 detects (recognizes) the entire operator W shown in the input image (for example, the captured image of the camera 40X) as a monitoring object (person). Therefore, the detection frame FR is represented as a rectangular shape surrounding the operator W.

[0209] On the other hand, for example, in the situation of FIG. 23, the object detection unit 302 detects (recognizes) only the upper part (the upper body part) in the vertical direction of the entire operator W shown in the input image as a monitoring object (person). Therefore, the detection frame FR is represented as a rectangular shape surrounding only the upper part of the operator W including the head where the helmet is worn and the torso where the highly visible safety suit is worn.

[0210] When the object detection unit 302 detects (recognizes) a person by detecting (recognizing) items worn by the person such as a helmet or a highly visible safety suit, there is a possibility that only a part of the area corresponding to the items worn by the person in the input image is determined as the detection range of the monitoring object. Also, depending on the background of the monitoring object and the weather at that time, the features such as the shape of the monitoring object in the input image become unclear, and the object detection unit 302 may be able to recognize only a part of the entire monitoring object as the monitoring object. Therefore, the object detection unit 302 may detect (recognize) only a part in the vertical direction of the entire monitoring object on the input image.

[0211] For example, in the situation of FIG. 23, as described above, when the actual position of the operator W is estimated based on the size of the detected (recognized) monitoring object on the input image (the vertical size of the detection frame FR), there is a possibility that it may be determined that the operator exists at a position farther from the upper slewing body 3 than the actual situation. Specifically, even though the operator W exists at the same position as in the situation of FIG. 22, in the situation of FIG. 23, there is a possibility that the actual position estimated in the situation of FIG. 23 is farther from the upper slewing body 3 than the actual position estimated in the situation of FIG. 22.

[0212] Also, for example, in the situation of FIG. 23 and the situation of FIG. 24, although the distance between the operator W and the upper slewing body 3 is different, the detection position of the person (operator W) on the input image, specifically, the lower end position of the detection range (detection frame FR) is substantially the same. Therefore, as described above, when the actual position of the monitoring object is estimated by projective transformation or the like based on the detection position (reference point) of the monitoring object on the input image, there is a possibility that the actual position relatively far from the upper slewing body 3 is estimated, similar to the operator W detected in the situation of FIG. 24.

[0213] In contrast, in this example, the position estimation unit 303 estimates the actual position of the monitoring object based on information regarding the vertical detection position of the monitoring object on the input image and information regarding the size and shape of the detection range of the monitoring object on the input image. The information regarding the vertical detection position of the monitoring object on the input image is the coordinate position representing the detection range of the monitoring object on the input image, and includes, for example, at least one coordinate of the lower end, upper end, and centroid of the detection range of the monitoring object on the input image. The information regarding the size of the detection range of the monitoring object on the input image includes, for example, information regarding the height (vertical dimension) of the detection range, information regarding the width (horizontal dimension), information regarding the aspect ratio, information regarding the diagonal line, information regarding the area, and the like. Further, the information regarding the shape of the detection range of the monitoring object on the input image includes, for example, information regarding the aspect ratio and the like.

[0214] Specifically, the position estimation unit 303 may determine whether there is consistency in the shape and size of the detection range of the monitored object on the input image with respect to the vertical detection position of the monitored object on the input image. Assuming that the entire monitored object in the input image has been detected, the shape and size range of the assumed detection range of the monitored object are determined by the vertical detection position of the monitored object on the input image. Therefore, the position estimation unit 303 can determine whether the entire monitored object has been detected based on whether the shape and size of the detection range of the monitored object on the input image fall within this assumed range with respect to the vertical detection position of the monitored object on the input image.

[0215] The assumed range of the shape and size of the detection range of the monitored object on the input image may be uniformly defined according to the vertical detection position of the monitored object on the input image. Further, the assumed range of the shape and size of the detection range of the monitored object on the input image may take other conditions into consideration in addition to the vertical detection position of the monitored object on the input image. For example, the assumed range of the shape and size of the detection range of the monitored object on the input image may be variable according to the posture and orientation of the detected monitored object in addition to the vertical detection position of the monitored object on the input image. This is because the appearance of the monitored object on the input image varies depending on the posture and orientation of the monitored object, and as a result, the shape and size of the detection range of the monitored object on the input image change. The orientation and posture of the monitored object can be obtained, for example, by machine learning the learned model LM so that the monitored object can be detected for different orientations and postures of the monitored object, that is, so that the monitored object can be detected for each of a plurality of labels corresponding to different orientations and postures. Also, the assumed range of the shape and size of the detection range of the monitored object on the input image (captured image) may be defined in consideration of, for example, not only the vertical detection position of the monitored object on the input image (captured image) but also the distortion caused by the lens at that detection position.

[0216] When there is consistency between the shape and size of the detection range of the monitored object in the input image and the detection position in the vertical direction of the monitored object in the input image, the position estimation unit 303 determines that the entire monitored object has been detected. In this case, the position estimation unit 303 may estimate the actual position of the monitored object based on the detection position and the detection range of the monitored object in the input image as described above.

[0217] On the other hand, when there is no consistency between the shape and size of the detection range of the monitored object in the input image and the detection position in the vertical direction of the monitored object in the input image, the position estimation unit 303 determines that only a part of the entire monitored object has been detected. In this case, the position estimation unit 303 may correct the detection position or the detection range of the monitored object in the input image, and estimate the actual position of the monitored object based on the corrected detection position and detection range.

[0218] For example, when the width of the detection range of the monitored object in the input image deviates in the larger direction from the assumed range with respect to the detection position in the vertical direction of the monitored object in the input image, the position estimation unit 303 determines that only the upper part of the entire monitored object in the input image has been detected (see FIG. 23).

[0219] Further, for example, when the aspect ratio of the detection range of the monitored object on the input image deviates in the direction where the horizontal dimension is larger than the assumed range with respect to the detection position in the vertical direction on the input image of the monitored object, the position estimation unit 303 determines that only a part of the entire monitored object in the vertical direction in the input image is detected. Further, when only a predetermined part of the entire monitored object or only the clothing of the monitored object is detected by the object detection unit 302, the position estimation unit 303 may determine which part of the upper and lower sides of the entire monitored object in the input image is detected only, considering the predetermined part or the wearing part of the clothing. For example, when a helmet or highly visible safety clothing is detected (recognized) by the object detection unit 302, the position estimation unit 303 determines that only the upper part including the wearing location of the helmet or highly visible safety clothing of the entire monitored object (person) in the input image is detected (see FIG. 23). Further, when the position estimation unit 303 determines that only a part of the entire monitored object in the vertical direction in the input image is detected, it may uniformly consider that only the upper part of the entire monitored object in the vertical direction is detected, giving priority to safety.

[0220] In this case, the position estimation unit 303 may perform correction to extend the detection range on the input image of the monitored object downward so as to have a shape (aspect ratio) assumed for the monitored object, based on information regarding the height and width of the detection range on the input image of the monitored object, for example. The shape (aspect ratio) assumed for the monitored object is defined according to, for example, the detection position on the input image of the monitored object. Also, the shape (aspect ratio) assumed for the monitored object may be defined according to, for example, in addition to the detection position on the input image of the monitored object, the orientation and posture of the monitored object, distortion due to the lens at the detection position, and the like. Further, the position estimation unit 303 may perform correction to move the detection position of the monitored object downward or extend the detection range of the monitored object downward so as to match the width of the detection range on the input image of the monitored object, based on information regarding the width of the detection range on the input image of the monitored object, for example. At this time, the position estimation unit 303 may correct the detection position and detection range of the monitored object in consideration of, for example, the orientation and posture of the monitored object, distortion due to the lens at the detection position, and the like. Then, the position estimation unit 303 may estimate the actual position of the monitored object based on the detection range and detection position of the monitored object after correction.

[0221] As described above, in this example, the position estimation unit 303 grasps a state in which only a part (particularly, an upper part in the vertical direction) of the entire monitored object on the input image is detected by the object detection unit 302, and estimates the actual position of the monitored object in consideration of that state. Therefore, the position estimation unit 303 can suppress a situation in which, for example, the actual position of the monitored object is estimated to be a position relatively farther from the upper revolving body 3 than the actual position, and can estimate the actual position of the monitored object more appropriately. Thus, the controller 30 (safety control unit 304) can operate the safety function more appropriately, and as a result, the safety of the excavator 100 can be improved.

[0222] [Another Example of the Functional Configuration of the Excavator] Next, with reference to FIG. 25, another example of the functional configuration of the excavator 100 will be described. Hereinafter, the description will be centered on the parts different from the above-described example (FIG. 6), and the same or corresponding descriptions as those in the above-described example may be omitted.

[0223] FIG. 25 is a functional block diagram showing another example of the functional configuration of the excavator 100 according to the present embodiment.

[0224] As shown in FIG. 25, the controller 30 includes a display processing unit 301, an object detection unit 302, a position estimation unit 303, a safety control unit 304, and an irradiation control unit 305, similar to the case of the above-described example, and may include an image correction unit 306. Further, unlike the above-described example, the controller 30 includes a detection determination unit 307.

[0225] The imaging device 40 includes an object detection unit 402. Further, the imaging device 40 may include an image correction unit 404.

[0226] The object detection unit 402 (an example of a detection unit) detects a monitoring object based on the captured image of the imaging device 40, similar to the object detection unit 302. The object detection unit 402 includes object detection units 402F, 402B, 402L, and 402R. Hereinafter, the object detection units 402F, 402B, 402L, and 402R may be collectively or individually referred to as "object detection unit 402X".

[0227] The object detection unit 402F is mounted on the camera 40F and detects a monitoring object based on the captured image of the camera 40F, similar to the object detection unit 302. Specifically, the object detection unit 402F may recognize a monitoring object from the captured image of the camera 40F and specify the position (area) where the monitoring object is shown. Further, the object detection unit 402F may recognize a monitoring object from the image (corrected captured image) after the captured image of the camera 40F (i.e., the captured image generated by the captured image generation function) is corrected by the image correction unit 404F and specify the position (area) where the monitoring object is shown. The function of the object detection unit 402F is realized by any hardware or any combination of hardware and software, etc. For example, the function of the object detection unit 402F is realized by, for example, a program installed in the auxiliary storage device of the image processing engine 42 of the camera 40F being loaded into the memory device and executed on the CPU. Hereinafter, the functions of the object detection units 402B, 402L, and 402R are the same.

[0228] The object detection unit 402B is mounted on the camera 40B and, similar to the object detection unit 302, detects a monitoring object based on the captured image of the camera 40B. Specifically, the object detection unit 402B may recognize the monitoring object from the captured image of the camera 40B and specify the position (area) where the monitoring object is shown. Further, the object detection unit 402B may recognize the monitoring object from the image (corrected captured image) after the captured image of the camera 40B (i.e., the captured image generated by the captured image generation function) is corrected by the image correction unit 404B and specify the position (area) where the monitoring object is shown.

[0229] The object detection unit 402L is mounted on the camera 40L and, similar to the object detection unit 302, detects a monitoring object based on the captured image of the camera 40L. Specifically, the object detection unit 402L may recognize the monitoring object from the captured image of the camera 40L and specify the position (area) where the monitoring object is shown. Further, the object detection unit 402L may recognize the monitoring object from the image (corrected captured image) after the captured image of the camera 40L (i.e., the captured image generated by the captured image generation function) is corrected by the image correction unit 404L and specify the position (area) where the monitoring object is shown.

[0230] The object detection unit 402R is mounted on the camera 40R and, similar to the object detection unit 302, detects a monitoring object based on the captured image of the camera 40R. Specifically, the object detection unit 402R may recognize the monitoring object from the captured image of the camera 40R and specify the position (area) where the monitoring object is shown. Further, the object detection unit 402R may recognize the monitoring object from the image (corrected captured image) after the captured image of the camera 40R (i.e., the captured image generated by the captured image generation function) is corrected by the image correction unit 404R and specify the position (area) where the monitoring object is shown.

[0231] Similar to the image correction unit 306, the image correction unit 404 performs predetermined correction on the output (captured image) of the imaging device 40 (camera 40X) and outputs the corrected captured image to the object detection unit 402. The image correction unit 404 includes 404F, 404B, 404L, and 404R.

[0232] The image correction unit 404F is mounted on the camera 40F and, similar to the above-described image correction unit 306, performs predetermined correction on the captured image of the camera 40F and outputs the corrected captured image to the object detection unit 402F.

[0233] In addition, the camera 40F may output the corrected captured image corrected by the image correction unit 404F to the controller 30 in addition to the captured image generated by the captured image generation function. The same may apply to the cameras 40B, 40L, and 40R below. In this case, the image correction unit 306 of the controller 30 may be omitted, and the object detection unit 302 may detect a monitoring object based on the corrected captured image input from the camera 40X.

[0234] The image correction unit 404B is mounted on the camera 40B and, similar to the above-described image correction unit 306, performs predetermined correction on the captured image of the camera 40B and outputs the corrected captured image to the object detection unit 402B.

[0235] The image correction unit 404L is mounted on the camera 40L and, similar to the above-described image correction unit 306, performs predetermined correction on the captured image of the camera 40L and outputs the corrected captured image to the object detection unit 402L.

[0236] The image correction unit 404R is mounted on the camera 40R and, similar to the above-described image correction unit 306, performs predetermined correction on the captured image of the camera 40R and outputs the corrected captured image to the object detection unit 402R.

[0237] The object detection unit 402X detects a monitored object from the input image using the same algorithm as each other. On the other hand, the object detection unit 402X detects a monitored object from the input image using an algorithm different from that of the object detection unit 302. Thereby, for example, even if either one of the object detection units 302 and 402X cannot detect the monitored object, the possibility that the other one detects the monitored object can be increased. As a result, it is possible to suppress the occurrence of a situation where the monitored object around the excavator 100 cannot be detected. Further, for example, even if either one of the object detection units 302 and 402X detects a monitored object that does not exist, it is possible to prevent the monitored object from being detected by the fact that the other one does not detect the monitored object.

[0238] For example, either one of the object detection units 302 and 402X may detect the monitored object by using only the former of the image processing technology and the machine learning technology, and the other one may use both the image processing technology and the machine learning technology, that is, the learned model LM.

[0239] Further, for example, either one of the object detection units 302 and 402X may detect the monitored object from the input image by using the SVM that has been machine-learned with the image feature amount of the monitored object as the learned model LM, and the other one may use the learned model LM by deep learning.

[0240] In addition, the object detection units 302 and 402X may detect a monitoring object from an input image, for example, using pre-trained models LM by different neural networks (DNNs). For example, either one of the object detection units 302 and 402X may use a classification model based on CNN as the pre-trained model LM, and the other may use a regression model based on CNN as the pre-trained model LM to detect a monitoring object from the input image. Specifically, either one of the object detection units 302 and 402X may use a pre-trained model LM by Faster R-CNN, and the other may use a pre-trained model LM by YOLO or SSD to detect a monitoring object from the input image. Also, for example, the object detection units 302 and 402X may use pre-trained models LM based on CNN, and at least one of the network structure, the number of network layers, the structure of the feature map, and the characteristics of the candidate regions of each pre-trained model LM may be different from each other. The characteristics of the candidate regions may include the aspect ratio and the difference between fixed or variable candidate regions. Specifically, either one of the object detection units 302 and 402X may use a pre-trained model LM by YOLO, and the other may use a pre-trained model LM by SSD to detect a monitoring object from the input image. Also, for example, the object detection units 302 and 402X may use the same pre-trained model LM by SSD, and at least one of the parameters such as the network structure, the number of network layers, the structure of the feature map, and the characteristics of the candidate regions of each pre-trained model LM may be different from each other.

[0241] In addition, for example, the object detection units 302 and 402X may detect a monitoring object from the input image using pre-trained models LM that are machine-learned with different teacher datasets.

[0242] For example, the object detection units 302 and 402X may detect a monitored object from an input image by using a learned model LM that has been machine-learned with a teacher dataset of images in which at least one of the postures and orientations of the monitored objects is different from each other. For example, the learned model LM of the object detection units 302 and 402X may be machine-learned with a teacher dataset of images showing a person in an upright state, and may also be machine-learned with a teacher dataset of images showing a person in a semi-bent or crouched state. Also, for example, the learned model of the object detection units 302 and 402X may be machine-learned with a teacher dataset of images showing a person facing forward or backward toward the camera, and may also be machine-learned with a teacher dataset of images showing a person facing sideways toward the camera. Thereby, even in a situation where the object detection units 302 and 402X cannot appropriately detect the monitored object due to the influence of the orientation or posture of the monitored object shown in the input image, the other one can increase the possibility of appropriately detecting the monitored object from the input image. Therefore, the object detection units 302 and 402X can more reliably detect the monitored object as a whole.

[0243] Also, for example, the object detection units 302 and 402X may detect the monitored object from the input image by detecting (recognizing) the whole of the monitored object (person) including the shape of the monitored object (person) with one of them and detecting (recognizing) the clothing of the monitored object (person) with the other one. For example, the object detection units 302 and 402X may detect a person from the input image by detecting the whole person from the input image with one of them and detecting (recognizing) a helmet or safety vest worn by the person with the other one. Thereby, the object detection units 302 and 402X can more reliably detect the monitored object (person) with the other one even in a situation where one of them cannot appropriately detect (recognize) the whole person for some reason.

[0244] Further, for example, the object detection units 302 and 402X may detect a monitoring object from an input image by detecting different articles of clothing of the monitoring object (person) from each other. For example, the object detection units 302 and 402X may detect a person from the input image by one of them detecting (recognizing) a helmet worn by a person from the input image and the other detecting (recognizing) a safety vest worn by the person from the input image. Thereby, even when, for example, an operator does not wear one article of clothing and one of the object detection units 302 and 402X cannot recognize one article of clothing, the other can recognize another article of clothing, and thus the monitoring object (person) can be detected more reliably.

[0245] Further, for example, the object detection units 302 and 402X may detect a monitoring object from an input image by using a learned model LM that is machine-learned with a teacher data set of images in which the types of backgrounds of the monitoring object are different from each other. For example, the learned model LM of the object detection units 302 and 402X may be machine-learned with a teacher data set of images in which one of them has an image of earthen ground in the background and the other may be machine-learned with a teacher data set of images in which there is an image of ground such as asphalt in the background. Thereby, even in a situation where one of the object detection units 302 and 402X cannot appropriately detect the monitoring object due to the influence of the background of the input image, the other can increase the possibility of appropriately detecting the monitoring object from the input image. Therefore, the object detection units 302 and 402X can more reliably detect the monitoring object as a whole.

[0246] Further, for example, the object detection units 302 and 402X may detect a monitored object from an input image by using a learned model LM that has been machine-learned with teacher data sets of images having different acquisition (imaging) time zones from each other. For example, the learned models of the object detection units 302 and 402X may be machine-learned with a teacher data set of images captured during the morning or daytime by one of them, and machine-learned with a teacher data set of images captured during the evening or nighttime by the other. Thereby, the controller 30 can, for example, selectively use the object detection units 302 and 402X according to the time zone during the operation of the excavator 100, or switch the method of making a final determination regarding the detection of the monitored object based on the detection results of the object detection units 302 and 402X.

[0247] Also, when the excavator 100 is used at night, for example, the object detection units 302 and 402X may detect a monitored object from an input image by using a learned model LM that has been machine-learned with teacher data sets of images having different nighttime lighting conditions from each other. For example, the learned model LM of the object detection units 302 and 402X may be machine-learned with a teacher data set of images acquired at a relatively high illuminance by one of them, and machine-learned with a teacher data set of images acquired at a relatively low illuminance by the other. Further, for example, the learned model LM of the object detection units 302 and 402X may be machine-learned with a teacher data set of images acquired under lighting conditions of different colors from each other. Thereby, even in a situation where one of them cannot appropriately detect the monitored object due to the influence of the lighting condition, the object detection units 302 and 402X can increase the possibility that the other can appropriately detect the monitored object. Therefore, the object detection units 302 and 402X can more reliably detect the monitored object as a whole.

[0248] Further, for example, the object detection units 302 and 402X may detect a monitoring object from an input image by using a learned model LM that has been machine-learned with a teacher dataset of images acquired (captured) under different weather conditions, for example. For example, for the learned model LM of the object detection units 302 and 402X, one of them may be machine-learned with a teacher dataset of images acquired in a sunny or cloudy state, and the other may be machine-learned with a teacher dataset of images acquired in a rainy or snowy state. Thereby, the object detection units 302 and 402X can increase the possibility that the other can appropriately detect the monitoring object even in a situation where one of them cannot appropriately detect the monitoring object due to the influence of the weather. Therefore, the object detection units 302 and 402X can more reliably detect the monitoring object as a whole.

[0249] Further, for example, the object detection units 302 and 402X may detect a monitoring object from an input image by using a learned model LM that has been machine-learned with a teacher dataset of images acquired with different positional relationships between a light source and an imaging range, for example. For example, the learned model LM of the object detection units 302 and 402X may be machine-learned with a teacher dataset of images corresponding to front light, semi-front light, or side light for one of them, and may be machine-learned with a teacher dataset of images corresponding to backlight for the other. Thereby, the object detection units 302 and 402X can increase the possibility that the other can appropriately detect the monitoring object even in a situation where one of them cannot appropriately detect the monitoring object due to the influence of the positional relationship between the light source and the imaging range. Therefore, the object detection units 302 and 402X can more reliably detect the monitoring object as a whole.

[0250] The detection determination unit 307 detects a monitoring object based on the detection results of the object detection unit 302 and the object detection unit 402X. Specifically, the detection determination unit 307 makes a final determination regarding the detection of the monitoring object based on the detection results of the object detection unit 302 and the detection results of the object detection unit 402X. Details of the method for detecting the monitoring object by the detection determination unit 307 will be described later.

[0251] When the detection determination unit 307 detects a monitored object, the position estimation unit 303 estimates the actual position of the detected monitored object. When the detection determination unit 307 detects a plurality of monitored objects, the position estimation unit 303 estimates the actual position for each of the plurality of monitored objects.

[0252] For example, when a monitored object is detected by either one of the object detection units 302 and 402X, the position of the monitored object is estimated by arbitrarily applying the above method based on the detection range and the like on the input image specified by either one of them.

[0253] Also, for example, when the same monitored object is detected by both of the object detection units 302 and 402X, the actual position of the monitored object may be estimated for each of the detection results of the object detection units 302 and 402X, and the actual position of the monitored object may be estimated from each of the estimation results. For example, the actual position of the detected monitored object may be estimated by taking the average of the estimation results of the actual position of the monitored object for each of the detection results of the object detection units 302 and 402X.

[0254] When the detection determination unit 307 detects a monitored object within a predetermined range around the excavator 100, the safety control unit 304 activates a safety function, for example. Specifically, the safety control unit 304 may activate the safety function when the actual position (estimated value) of the monitored object estimated by the position estimation unit 303 is within a predetermined range around the excavator 100.

[0255] Note that the function of either one of the object detection units 302 and 402 may be transferred to the display device 50A. In addition to the object detection units 302 and 402, one or more object detection units for detecting a monitored object from the input image may be provided in the excavator 100. The additional object detection unit may be provided in the display device 50A, may be provided in the controller 30, or may be provided in a controller different from the controller 30. In this case, three or more object detection units including the additional ones may detect a monitored object from the input image using different algorithms from each other.

[0256] [Specific Example of Object Detection Method] Next, a method for detecting a monitored object by the detection determination unit 307, that is, a method for making a final determination regarding the detection of the monitored object will be described.

[0257] When the monitored object is detected by at least one of, for example, the object detection unit 302 and the object detection unit 402X, the detection determination unit 307 may determine that the monitored object exists around the excavator 100 and detect the monitored object. Thereby, the controller 30 can detect the monitored object if it is detected by the other one even if it is not detected by either one of the object detection units 302 and 402X. Therefore, in a situation where a detection failure of the monitored object is likely to occur or where there is a high need to prevent a detection failure of the monitored object, the controller 30 can more reliably detect the existing monitored object, and as a result, the safety can be further improved along with the operation of the safety function of the excavator 100.

[0258] When the monitored object is detected by both the object detection unit 302 and the object detection unit 402X, the detection determination unit 307 may determine that the monitored object exists around the excavator 100 and detect the monitored object. Thereby, the controller 30 can prevent the detection of the monitored object if it is not detected by the other one even if the monitored object that does not actually exist is erroneously detected by either one of the object detection units 302 and 402X. Therefore, in a situation where an erroneous detection of the monitored object is likely to occur or where there is a high need to prevent a decrease in work efficiency due to an erroneous detection of the monitored object, the controller 30 can suppress the erroneous detection of the monitored object.

[0259] As described above, when three or more object detection units are provided in the excavator 100, the detection determination unit 307 may detect a monitored object, for example, when the monitored object is detected by a predetermined number (an integer of 2 or more) or more of the object detection units among all the object detection units. The predetermined number may be fixed or variable. For example, the predetermined number may be varied according to a predetermined input from a user (operator or monitor) received through the input device 52 or the communication device 60.

[0260] Further, the detection determination unit 307 may switch between the above two methods according to a predetermined input from a user (operator or monitor) received through the input device 52 or the communication device 60.

[0261] Further, the detection determination unit 307 may detect a monitored object based on the object detection unit 302 and the object detection unit 402X, for example, using the importance defined by the detection results of the object detection unit 302 and the object detection unit 402X respectively. Specifically, the detection determination unit 307 detects the monitored object when the monitored object is detected by the one with a relatively higher importance of the detection results among the object detection unit 302 and the object detection unit 402X.

[0262] The importance level may be set (variable) according to, for example, a predetermined input from a user (operator or monitor) received through the input device 52 or the communication device 60. Thereby, the user can compare, for example, the surrounding images displayed on the display device 50A or the remote operation display device with the detection status of the monitored object by the actual object detection units 302 and 402X, and can set a higher importance level for the detection result of one side that is considered to be more accurate. Further, the importance level may be automatically set (variable) according to the situation around the excavator 100. For example, when there is a difference in the detection accuracy of the monitored object between the object detection units 302 and 402X according to the environmental state where the excavator 100 is placed, the importance level of the detection result of the one with higher detection accuracy may be set higher than the importance level of the other detection result. The environmental state where the excavator 100 is placed may include the state of the objects around the excavator 100 (for example, the type of earth and sand on the ground, asphalt, etc.) reflected as the background in the captured image of the camera 40X. Further, the environmental state where the excavator 100 is placed may include, for example, the time zone when the work is being performed, the lighting state during night work, the weather state, the positional relationship between the light source and the imaging range of the camera 40X, and the like. Thereby, the controller 30 can relatively increase the importance level of any one of the object detection units 302 and 402X that is assumed to have higher detection accuracy according to the situation where the excavator 100 is placed.

[0263] In addition, as described above, when three or more object detection units are provided in the excavator 100, the detection determination unit 307 may detect the monitored object so that the detection result of each object detection unit is more likely to be reflected in the result of the final determination as the importance level of the detection result of each object detection unit becomes relatively higher. For example, when the monitored object is detected by some of the object detection units among all the object detection units, the detection determination unit 307 may detect the monitored object when the sum of the importance levels of the respective detection results of some of the object detection units is greater than the sum of the importance levels of the respective detection results of the other remaining object detection units. Further, for example, when the monitored object is detected by some of the object detection units among all the object detection units, the detection determination unit 307 may detect the monitored object when the sum of the importance levels of the respective detection results of some of the object detection units is equal to or greater than a predetermined threshold value.

[0264] [Modification and Change] Although the embodiments have been described in detail above, the present disclosure is not limited to such specific embodiments, and various modifications and changes are possible within the scope of the gist described in the claims.

[0265] For example, in the above-described embodiment, the method for detecting a monitored object based on the captured image of the imaging device 40 mounted on the excavator 100 has been described. However, the same method may be applied to a method for detecting a monitored object based on the captured image of an imaging device mounted on other work machines. Other work machines include, for example, a lift magnet machine in which a lifting magnet is attached to the tip of the attachment AT instead of the bucket 6, a bulldozer, a forestry machine (e.g., a harvester), a road machine (e.g., an asphalt finisher), and the like.

Description of Reference Numerals

[0266] 1 Lower Traveling Body 1C Crawler 1ML Travel Hydraulic Motor 1MR Travel Hydraulic Motor 2 Swing Mechanism 2M Swing Hydraulic Motor 3 Upper Swing Structure 4 Boom 5 Arm 6 Bucket 7 Boom Cylinder 8 Arm Cylinder 9 Bucket Cylinder 10 Cabin 11 Engine 13 Regulator 14 Main Pump 15 Pilot Pump 17 Control Valve 25 Pilot Line 25A Pilot Line 25B Pilot Line 26 Operating Device 27 Pilot Line 27A Pilot Line 27B Pilot Line 29 Operating Pressure Sensor 30 Controller 30A Auxiliary Storage Device 30B Memory Device 30C CPU 30D Interface Device 31 Hydraulic Control Valve 32 Shuttle Valve 33 Hydraulic Control Valve 40 Imaging Device 40B Camera 40F Camera 40L Camera 40R Camera 50 Output Device 50A Display Device 50B Sound Output Device 52 Input Device 60 Communication Device 70 Irradiation Device 100 Excavator 200 Management Device 201 External Interface 201A Recording Medium 202 Auxiliary Storage Device 203 Memory Device 204 CPU 205 High-Speed Arithmetic Unit 206 Communication Interface 207 Input Device 208 Display Device 301 Display Processing Unit 302 Object Detection Unit (Detection Unit) 303 Position Estimation Unit (Estimation Unit) 304 Safety Control Unit (Control Unit) 305 Irradiation Control Unit 306 Image Correction Unit 307 Detection Judgment Unit 402,402B,402F,402L,402R Object Detection Unit (Detection Unit) 404,404B,404F,404L,404R Image Correction Unit AT Attachment HA Hydraulic Actuator HMT Helmet LM Learned Model NW Communication Line RV Highly Visible Safety Suit S1 Boom Angle Sensor S2 Arm Angle Sensor S3 Bucket Angle Sensor S4 Aircraft Attitude Sensor S5 Turning Angle Sensor SYS Excavator Management System W, W1~W3 Operators

Claims

1. A lower traveling body, an upper slewing body rotatably mounted on the lower traveling body, an imaging device mounted on the upper slewing body and imaging the periphery of the upper slewing body, a detection unit that detects a predetermined object around the excavator from an input image based on the captured image of the imaging device, and based on information regarding a detection position in the vertical direction of the predetermined object on the input image and information regarding a detection range of the predetermined object on the input image, whether the relationship between the detection position in the vertical direction of the predetermined object on the input image and the detection range of the predetermined object on the input image has consistency when it is assumed that the entire predetermined object is shown in the input image, an estimation unit that estimates the position of the predetermined object around the excavator, an excavator.

2. The information regarding the detection position in the vertical direction of the predetermined object on the input image is a coordinate position representing the detection range of the predetermined object on the input image, including at least one of an upper end, a lower end, and a centroid of the detection range of the predetermined object on the input image. The excavator according to Claim 1.

3. The information regarding the detection range of the predetermined object on the input image includes at least one of information regarding the width, height, aspect ratio, diagonal length, and area of the detection range of the predetermined object on the input image. The excavator according to Claim 1 or 2.

4. The detection unit recognizes at least one of the orientation and the posture of the predetermined object, and the estimation unit estimates the position of the predetermined object around the excavator by determining whether the relationship between the detection position in the vertical direction of the predetermined object on the input image and the detection range of the predetermined object on the input image has the consistency, taking into account at least one of the orientation and the posture of the predetermined object. The excavator according to any one of Claims 1 to 3.

5. The estimation unit estimates the position of the predetermined object around the excavator by determining whether the relationship between the detection position in the vertical direction of the predetermined object on the input image and the detection range of the predetermined object on the input image has the consistency, taking into account distortion caused by a lens in the input image. The excavator according to any one of Claims 1 to 4.

6. The detection unit detects the predetermined object by detecting a predetermined part of the whole of the predetermined object or a wearable of the predetermined object from the input image. The estimation unit estimates the position of the predetermined object around the shovel based on the predetermined part within the detection range of the predetermined object on the input image or the position of the wearable. The shovel according to any one of claims 1 to 5.

7. When the size of the detection range of the predetermined object on the input image is within a predetermined range defined according to the vertical detection position of the predetermined object on the input image, the estimation unit estimates the position of the predetermined object around the shovel based on information regarding at least one of the detection position and the detection range of the predetermined object on the input image. When the size of the detection range of the predetermined object on the input image is outside the predetermined range, the estimation unit corrects at least one of the detection position and the detection range of the predetermined object on the input image based on information regarding the detection range of the predetermined object on the input image, and estimates the position of the predetermined object around the shovel based on at least one of the corrected detection position and the detection range of the predetermined object on the input image. The shovel according to any one of claims 1 to 6.

8. When the predetermined object is detected within a predetermined range of the shovel by the detection unit, it includes a control unit that decelerates or stops the operation of the shovel. The shovel according to any one of claims 1 to 7.

Images