Excavator

Abstract

To provide a technique for more approximately detecting an object to be detected from a picked-up image around a shovel.SOLUTION: A shovel 100 includes an undercarriage 1, a super structure 3 turnably mounted on the undercarriage 1, an image pick-up device 40 mounted on the super structure 3 for picking up an image around the super structure 3, an object detection part 302 for recognizing the former only out of predetermined objects showing that a difference between the vertical axis of an input image and a vertical axis for a subject thereon is in a relatively small state and showing that it is in a relatively big state, respectively, to detect a monitored object from the input image, and an image correction part 306 for correcting the picked-up image so as to reduce the difference between the vertical axis of the image picked up by the image pick-up device 40 and the vertical axis for the subject on the picked-up image. The object detection part 302 detects the monitored object from the image corrected by the image correction part 306, as the input image.SELECTED DRAWING: Figure 17

Description

[Technical Field]

[0001] The present disclosure relates to a shovel. [Background technology]

[0002] Conventionally, a technology has been known in which a predetermined object (e.g., a person) within a predetermined range around a shovel (upper rotating body) is detected from an image captured by an imaging device mounted on the upper rotating body, and a safety function (e.g., alarm output) is activated (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-181508 Summary of the Invention [Problem to be solved by the invention]

[0004] However, for example, because the range of visibility behind the upper rotating body is very limited from the perspective of an excavator operator, the viewing angle that the imaging device must cover may be set relatively wide. Furthermore, for example, because the imaging device is installed on the upper rotating body, it may be installed at a relatively high position from the ground, with the lens of the imaging device facing diagonally downward toward the ground. This may result in distortion in the image captured by the imaging device, and for example, the subject may appear tilted outward at both ends of the image. Therefore, it may be impossible to properly detect (recognize) the object to be detected from the captured image, which may result in the safety function not operating properly.

[0005] In view of the above problem, an object of the present invention is to provide a technology that can more appropriately detect a detection target object from a captured image of the periphery of a shovel. [Means for solving the problem]

[0006] In order to achieve the above object, in one embodiment of the present disclosure, a lower running body; an upper rotating body rotatably mounted on the lower traveling body; an imaging device mounted on the upper rotating body and configured to capture an image of the periphery of the upper rotating body; a detection unit that detects a predetermined object from the input image by recognizing only the former of the predetermined objects in which the difference between the up-down axis of the input image and the vertical axis of the subject is relatively small and the former in which the difference between the up-down axis of the input image and the vertical axis of the subject is relatively large; a correction unit that corrects the captured image so that a difference between an up-down axis of the captured image of the imaging device and a vertical axis of a subject appearing in the captured image becomes small, The correction unit does not correct the first image region of the captured image, in which a difference between an up-down axis of the captured image and a vertical axis for the subject is relatively small, and the correction unit corrects only the second image region as a whole in a direction in which the difference between the up-down axis of the captured image and the vertical axis for the subject is relatively large, out of the first image region of the captured image, in which a difference between an up-down axis of the captured image and a vertical axis for the subject is relatively small, One and correcting the captured image in a manner that tilts the captured image as follows: the detection unit detects the predetermined object from the image corrected by the correction unit as the input image. Shovels are provided. [Effects of the Invention]

[0007] According to the above-described embodiment, it is possible to more appropriately detect the object to be detected from the captured image around the shovel. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a side view showing an example of a shovel. [Figure 2] FIG. 1 is a top view showing an example of a shovel. [Figure 3] FIG. 1 is a diagram illustrating an example of an excavator management system. [Figure 4]FIG. 2 is a diagram illustrating an example of a hardware configuration of a shovel. [Figure 5] FIG. 2 illustrates an example of a hardware configuration of a management apparatus. [Figure 6] FIG. 2 is a functional block diagram showing an example of a functional configuration of a shovel. [Figure 7] FIG. 10 is a diagram illustrating a specific example of an object detection method. [Figure 8] FIG. 10 is a diagram showing a first example of an image of training data. [Figure 9] FIG. 10 is a diagram showing a second example of an image of training data. [Figure 10] FIG. 10 is a diagram showing a third example of an image of training data. [Figure 11] FIG. 10 is a diagram showing a fourth example of an image of training data. [Figure 12] FIG. 10 is a diagram showing a fifth example of an image of training data. [Figure 13] FIG. 10 is a diagram showing a sixth example of an image of training data. [Figure 14] FIG. 10 is a diagram illustrating an example of a worker. [Figure 15] FIG. 10 is a diagram illustrating another example of a worker. [Figure 16] FIG. 2 is a diagram illustrating an example of an image captured by an imaging device. [Figure 17] FIG. 10 is a diagram showing an example of an image (corrected captured image) after correction of an image captured by an imaging device. [Figure 18] FIG. 1 is a diagram showing a first example of a captured image showing only a part of the entire (whole body) of a monitored object (person). [Figure 19] FIG. 10 is a diagram showing a second example of a captured image showing only a part of the entire (whole body) of a monitored object (person). [Figure 20] FIG. 10 is a diagram showing a third example of a captured image showing only a part of the entire (whole body) of a monitored object (person). [Figure 21] FIG. 10 is a diagram showing a fourth example of a captured image showing only a part of the entire (whole body) of a monitored object (person). [Figure 22] FIG. 10 is a diagram showing an example of a state in which a monitored object (person) is detected. [Figure 23]10A and 10B are diagrams showing other examples of the detection state of a monitored object (person). [Figure 24] FIG. 10 is a diagram showing yet another example of the detection state of a monitored object (person). [Figure 25] FIG. 10 is a functional block diagram showing another example of the functional configuration of the shovel. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment will be described with reference to the drawings.

[0010] [Outline of the Excavator] The outline of the shovel will be described with reference to FIGS. 1 to 3. FIG.

[0011] 1 and 2 are a side view and a top view showing an example of an excavator 100 according to this embodiment. Fig. 3 is a diagram showing an example of an excavator management system SYS.

[0012] As shown in Figures 1 and 2, the excavator 100 includes a lower traveling body 1, an upper rotating body 3 mounted on the lower traveling body 1 so as to be rotatable via a rotating mechanism 2, an attachment AT for performing various tasks, and a cabin 10. Hereinafter, the front of the excavator 100 (upper rotating body 3) corresponds to the direction in which the attachment to the upper rotating body 3 extends when the excavator 100 is viewed in a plan view (top view) from directly above along the rotation axis of the upper rotating body 3.

[0013] The lower traveling body 1 includes, for example, a pair of left and right crawlers 1C. The lower traveling body 1 allows the excavator 100 to travel by hydraulically driving each of the crawlers 1C by a left traveling hydraulic motor 1ML and a right traveling hydraulic motor 1MR.

[0014] The upper rotating body 3 rotates relative to the lower traveling body 1 as a result of the rotating mechanism 2 being hydraulically driven by the rotating hydraulic motor 2M.

[0015] The attachment AT includes a boom 4, an arm 5, and a bucket 6 as driven elements.

[0016] The boom 4 is attached to the front center of the upper rotating body 3 so as to be able to tilt up and down, and an arm 5 is attached to the tip of the boom 4 so as to be able to rotate up and down, and a bucket 6 is attached to the tip of the arm 5 so as to be able to rotate up and down.

[0017] The bucket 6 is an example of an end attachment and is used, for example, for excavation work.

[0018] Furthermore, instead of the bucket 6, another end attachment may be attached to the tip of the arm 5 depending on the type of work, etc. The other end attachment may be, for example, another type of bucket, such as a large bucket, a slope bucket, or a dredging bucket. The other end attachment may also be a type of end attachment other than a bucket, such as an agitator, a breaker, or a grapple. Furthermore, a spare attachment, such as a quick coupling or a tiltrotator, may be interposed between the arm 5 and the end attachment.

[0019] A hook for crane work may also be attached to the bucket 6. The base end of the hook is rotatably connected to a bucket pin that connects the arm 5 and the bucket 6. As a result, when work other than crane work (lifting work), such as excavation work, is performed, the hook is stored in the space formed between the two bucket links.

[0020] The boom 4, the arm 5, and the bucket 6 are hydraulically driven by a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9, which serve as hydraulic actuators, respectively.

[0021] Note that the shovel 100 may have some or all of the various hydraulic actuators replaced with electric actuators, that is, the shovel 100 may be a hybrid shovel or an electric shovel.

[0022] The cabin 10 is a control room in which an operator sits, and is mounted on the front left side of the upper rotating body 3, for example.

[0023] In addition, if an operator does not board the shovel 100 to operate it, and the shovel 100 operates solely by remote control or fully automatic operation function, as described below, the cabin 10 may be omitted.

[0024] 3, the shovel 100 may be included in an shovel management system SYS together with the management device 200, and may be able to communicate with the management device 200 via a predetermined communication line NW. This allows the shovel 100 to transmit (upload) various types of information to the management device 200 and receive various types of signals (for example, information signals and control signals) from the management device 200.

[0025] The communication line NW includes, for example, a wide area network (WAN). The wide area network may include, for example, a mobile communication network terminated at a base station. The wide area network may also include, for example, a satellite communication network that uses a communication satellite above the shovel 100. The wide area network may also include, for example, the Internet network. The communication line NW may also include, for example, a local area network (LAN) of a facility or the like in which the management device 200 is installed. The local network may be a wireless line, a wired line, or a line that includes both. The communication line NW may also include, for example, a short-range communication line based on a predetermined wireless communication method such as WiFi or Bluetooth (registered trademark).

[0026] The shovel management system SYS performs management of the shovel 100 and support for the shovel 100 in the management device 200.

[0027] For example, the shovel management system SYS may manage (monitor) the operating state and operational status of the shovel 100 in the management device 200 based on various information uploaded from the shovel 100. Furthermore, as will be described later, the shovel management system SYS may support remote operation of the shovel 100 in the management device 200. Furthermore, as will be described later, when the shovel 100 performs work by fully automatic operation, the shovel management system SYS may support, for example, remote monitoring of work performed by the fully automatic operation of the shovel 100 in the management device 200. In this case, the shovel management system SYS may support, in the management device 200, operation of the shovel 100 by intervention of a monitor for the fully automatic operation function.

[0028] The excavator management system SYS may include one or more excavators 100. The excavator management system SYS may include one or more management devices 200. That is, the multiple management devices 200 may execute processing related to the excavator management system SYS in a distributed manner. For example, each of the multiple management devices 200 may communicate with some of the excavators 100 that it is responsible for out of all the excavators 100 included in the excavator management system SYS, and execute processing targeted at some of the excavators 100.

[0029] The management device 200 may be, for example, an on-premise server or a cloud server installed in a management center or the like outside the work site where the shovel 100 performs work. The management device 200 may also be, for example, an edge server installed within the work site where the shovel 100 performs work, or in a location relatively close to the work site. The management device 200 may also be a stationary terminal device or a portable terminal device (mobile terminal) installed in a management office or the like within the work site of the shovel 100. The stationary terminal device may include, for example, a desktop PC (Personal Computer). The portable terminal device may include, for example, a smartphone, a tablet terminal, a laptop PC, etc.

[0030] The excavator 100 may operate an actuator (e.g., a hydraulic actuator) in response to operation by an operator in the cabin 10 to drive driven elements such as the lower traveling body 1, the upper rotating body 3, and the attachment AT.

[0031] The shovel 100 may also be configured to be remotely operable from a predetermined external device (for example, the management device 200). When the shovel 100 is remotely operated, the interior of the cabin 10 may be unmanned. The following description will be given on the assumption that the operation by the operator includes both operation of the operating device 26 by the operator in the cabin 10 and remote operation by an external operator.

[0032] Specifically, the shovel 100 is operated by input from a user (operator) regarding the actuator of the shovel 100, which is performed by the management device 200. In this case, the shovel 100 transmits data of an image (hereinafter referred to as a "peripheral image") representing the surroundings of the shovel 100, such as an image captured by the imaging device 40 described below or a processed image (e.g., a viewpoint-converted image) based on the captured image, to an external device. The peripheral image is then displayed on a display device for remote operation provided in the management device 200 (hereinafter referred to as a "remote operation display device"). Moreover, various information images (information screens) displayed on the display device 50A (see FIG. 6) inside the cabin 10 of the shovel 100 may also be displayed on the remote operation display device of the management device 200. This allows the operator to remotely operate the shovel 100 while checking the display contents, such as the peripheral image representing the surroundings of the shovel 100 and various information images, displayed on the remote operation display device. The shovel 100 can operate the actuators and drive driven elements such as the lower traveling body 1, the upper rotating body 3, and the attachment AT in accordance with a remote control signal representing the content of the remote control received from the management device 200.

[0033] Furthermore, the excavator 100 may automatically operate the actuators regardless of the operation by the operator. This allows the excavator 100 to realize a function of automatically operating at least some of the driven elements such as the lower traveling body 1, the upper rotating body 3, and the attachment AT, i.e., a so-called "automatic driving function" or "machine control (MC) function."

[0034] The automatic driving function may include a function for automatically operating driven elements (actuators) other than the driven element (actuator) to be operated in response to an operator's operation, i.e., a so-called "semi-automatic driving function" or "operation-assisted MC function." The automatic driving function may also include a function for automatically operating at least some of the multiple driven elements (hydraulic actuators) without operator operation, i.e., a so-called "fully automatic driving function" or "fully automatic MC function." In the excavator 100, when the fully automatic driving function is enabled, the interior of the cabin 10 may be unmanned. The semi-automatic driving function, the fully automatic driving function, etc. may also include a mode in which the operation content of the driven elements (actuators) to be operated automatically is automatically determined according to predetermined rules. The semi-automatic driving function, the fully automatic driving function, etc. may also include a so-called "autonomous driving function" in which the excavator 100 autonomously makes various decisions and autonomously determines the operation content of the driven elements (hydraulic actuators) to be operated automatically based on the results of the decisions.

[0035] [Excavator hardware configuration] Next, the hardware configuration of the shovel 100 will be described with reference to FIG.

[0036] FIG. 4 is a diagram illustrating an example of a hardware configuration of the shovel 100.

[0037] In Figure 4, the paths through which mechanical power is transmitted are indicated by double lines, the paths through which high-pressure hydraulic oil that drives the hydraulic actuator flows are indicated by solid lines, the paths through which pilot pressure is transmitted are indicated by dashed lines, and the paths through which electrical signals are transmitted are indicated by dotted lines.

[0038] The shovel 100 includes various components, such as a hydraulic drive system for hydraulically driving the driven elements, an operation system for operating the driven elements, a user interface system for exchanging information with the user, a communication system for communicating with the outside world, and a control system for various controls.

[0039] <Hydraulic drive system> 4, the hydraulic drive system of the excavator 100 according to this embodiment includes the hydraulic actuator HA that hydraulically drives each of the driven elements such as the lower traveling body 1 (left and right crawlers 1C), the upper rotating body 3, and the attachment AT, as described above. The hydraulic drive system of the excavator 100 according to this embodiment also includes the engine 11, a regulator 13, a main pump 14, and a control valve 17.

[0040] As shown in FIG. 6, the hydraulic actuator HA includes traveling hydraulic motors 1ML, 1MR, a swing hydraulic motor 2M, a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, and the like.

[0041] The engine 11 is the prime mover of the excavator 100 and the main power source in the hydraulic drive system. The engine 11 is, for example, a diesel engine that uses light oil as fuel. The engine 11 is mounted, for example, on the rear of the upper rotating body 3. The engine 11 rotates at a constant speed at a preset target speed under direct or indirect control by a controller 30 (described later), and drives the main pump 14 and the pilot pump 15.

[0042] It should be noted that instead of or in addition to the engine 11, the excavator 100 may be equipped with another prime mover (for example, an electric motor).

[0043] The regulator 13 controls (adjusts) the discharge amount of the main pump 14 under the control of the controller 30. For example, the regulator 13 adjusts the angle of the swash plate of the main pump 14 (hereinafter referred to as the "tilting angle") in response to a control command from the controller 30.

[0044] The main pump 14 supplies hydraulic oil to the control valve 17 through a high-pressure hydraulic line. The main pump 14 is mounted, for example, on the rear of the upper rotating body 3, similar to the engine 11. As described above, the main pump 14 is driven by the engine 11. The main pump 14 is, for example, a variable displacement hydraulic pump, and as described above, under the control of the controller 30, the tilt angle of the swash plate is adjusted by the regulator 13, thereby adjusting the stroke length of the piston and controlling the discharge flow rate (discharge pressure).

[0045] The control valve 17 is a hydraulic control device that controls the hydraulic actuators HA in response to an operator's operation of the operating device 26, the details of remote operation, or operation commands related to the automatic operation function output from the controller 30. The operation commands corresponding to the automatic operation function may be generated by the controller 30 or by another control device (computing device) that performs control related to the automatic operation function. The control valve 17 is mounted, for example, in the center of the upper rotating body 3. As described above, the control valve 17 is connected to the main pump 14 via a high-pressure hydraulic line, and selectively supplies hydraulic oil supplied from the main pump 14 to each hydraulic actuator in response to an operator's operation or an operation command corresponding to the automatic operation function. Specifically, the control valve 17 includes a plurality of control valves (also referred to as "directional control valves") that control the flow rate and flow direction of hydraulic oil supplied from the main pump 14 to each hydraulic actuator HA.

[0046] <Operation system> As shown in FIG. 4, the operating system of the excavator 100 according to this embodiment includes a pilot pump 15, an operating device 26, a hydraulic control valve 31, a shuttle valve 32, and a hydraulic control valve 33.

[0047] The pilot pump 15 supplies pilot pressure to various hydraulic devices via a pilot line 25. The pilot pump 15 is mounted, for example, on the rear of the upper rotating body 3, similar to the engine 11. The pilot pump 15 is, for example, a fixed displacement hydraulic pump, and is driven by the engine 11 as described above.

[0048] The pilot pump 15 may be omitted. In this case, the relatively high-pressure hydraulic oil discharged from the main pump 14 is reduced in pressure by a predetermined pressure reducing valve, and the resulting relatively low-pressure hydraulic oil is supplied to various hydraulic devices as pilot pressure.

[0049] The operating device 26 is provided near the cockpit of the cabin 10 and is used by the operator to operate the various driven elements. In other words, the operating device 26 is used by the operator to operate the hydraulic actuators HA that drive the respective driven elements. The operating device 26 includes pedal devices and lever devices for operating the respective driven elements (hydraulic actuators HA).

[0050] For example, as shown in Fig. 4, the operating device 26 is of a hydraulic pilot type. Specifically, the operating device 26 uses hydraulic oil supplied from the pilot pump 15 through a pilot line 25 and a pilot line 25A branching from the pilot line 25, and outputs a pilot pressure corresponding to the operation to a secondary pilot line 27A. The pilot line 27A is connected to one inlet port of a shuttle valve 32, and is connected to the control valve 17 via a pilot line 27 connected to an outlet port of the shuttle valve 32. This allows pilot pressure corresponding to the operation of various driven elements (hydraulic actuators) in the operating device 26 to be input to the control valve 17 via the shuttle valve 32. Therefore, the control valve 17 can drive each hydraulic actuator HA according to the operation of the operating device 26 by an operator or the like.

[0051] The operating device 26 may be electric. In this case, the pilot line 27A, the shuttle valve 32, and the hydraulic control valve 33 are omitted. Specifically, the operating device 26 outputs an electric signal (hereinafter, "operation signal") corresponding to the operation content, and the operation signal is input to the controller 30. The controller 30 then outputs a control command corresponding to the operation signal, i.e., a control signal corresponding to the operation content of the operating device 26, to the hydraulic control valve 31. As a result, a pilot pressure corresponding to the operation content of the operating device 26 is input from the hydraulic control valve 31 to the control valve 17, and the control valve 17 can drive each hydraulic actuator HA according to the operation content of the operating device 26. Furthermore, the control valves (directional control valves) built into the control valve 17 and driving each hydraulic actuator may be electromagnetic solenoid-type. In this case, the operation signal output from the operating device 26 may be directly input to the control valve 17, i.e., to the electromagnetic solenoid-type control valve. Furthermore, when the excavator 100 is remotely operated or operates using an automatic operation function, the operating device 26 may be omitted.

[0052] The hydraulic control valve 31 is provided for each driven element (hydraulic actuator HA) to be operated by the operating device 26. The hydraulic control valve 31 may be provided, for example, in a pilot line 25B between the pilot pump 15 and the control valve 17, and configured to change its flow path area (i.e., the cross-sectional area through which hydraulic oil can flow). This allows the hydraulic control valve 31 to output a predetermined pilot pressure to a secondary pilot line 27B using hydraulic oil from the pilot pump 15 supplied through the pilot line 25B. Therefore, as shown in FIG. 4, the hydraulic control valve 31 can indirectly apply a predetermined pilot pressure to the control valve 17 in response to a control signal from the controller 30 through a shuttle valve 32 between the pilot line 27B and the pilot line 27. Furthermore, as shown in FIG. 5, the hydraulic control valve 31 can directly apply a predetermined pilot pressure to the control valve 17 in response to a control signal from the controller 30 through the pilot line 27B and the pilot line 27. Therefore, the controller 30 can supply pilot pressure corresponding to the operation of the electric operating device 26 from the hydraulic control valve 31 to the control valve 17, thereby realizing operation of the excavator 100 based on the operation of the operator.

[0053] Furthermore, the controller 30 may, for example, control the hydraulic control valve 31 to realize an automatic driving function. Specifically, the controller 30 outputs a control signal corresponding to an operation command related to the automatic driving function to the hydraulic control valve 31, regardless of whether the operating device 26 is operated or not. As a result, the controller 30 causes the hydraulic control valve 31 to supply a pilot pressure corresponding to the operation command related to the automatic driving function to the control valve 17, thereby realizing the operation of the excavator 100 based on the automatic driving function.

[0054] Furthermore, the controller 30 may, for example, control the hydraulic control valve 31 to realize remote operation of the shovel 100. Specifically, the controller 30 outputs a control signal corresponding to the content of the remote operation specified in the remote operation signal received from the management device 200 to the hydraulic control valve 31 via the communication device 60. As a result, the controller 30 causes the hydraulic control valve 31 to supply a pilot pressure corresponding to the content of the remote operation to the control valve 17, thereby realizing the operation of the shovel 100 based on the remote operation by the operator.

[0055] The shuttle valve 32 has two inlet ports and one outlet port, and outputs hydraulic oil having a higher pilot pressure of the two pilot pressures input to the two inlet ports to the outlet port. A shuttle valve 32 is provided for each driven element (hydraulic actuator HA) operated by the operating device 26. A shuttle valve 32 is also provided for each movement direction of the driven element (e.g., the raising and lowering directions of the boom 4). One of the two inlet ports of the shuttle valve 32 is connected to a pilot line 27A on the secondary side of the operating device 26 (specifically, the lever device or pedal device included in the operating device 26), and the other is connected to a pilot line 27B on the secondary side of the hydraulic control valve 31. The outlet port of the shuttle valve 32 is connected to the pilot port of a corresponding control valve of the control valve 17 via the pilot line 27. The corresponding control valve is a control valve that drives a hydraulic actuator that is operated by the lever device or pedal device connected to one inlet port of the shuttle valve 32. Therefore, each of these shuttle valves 32 can apply the higher of the pilot pressure in pilot line 27A on the secondary side of the operating device 26 and the pilot pressure in pilot line 27B on the secondary side of the hydraulic control valve 31 to the pilot port of the corresponding control valve. In other words, by outputting a pilot pressure higher than the pilot pressure on the secondary side of the operating device 26 from the hydraulic control valve 31, the controller 30 can control the corresponding control valve regardless of the operation of the operating device 26 by the operator. Therefore, the controller 30 can control the operation of the driven elements (undercarriage 1, upper revolving body 3, and attachment AT) regardless of the operating state of the operating device 26 by the operator, thereby achieving an automatic driving function.

[0056] The hydraulic control valve 33 is provided in a pilot line 27A connecting the operating device 26 and the shuttle valve 32. The hydraulic control valve 33 is configured, for example, to be able to change its flow path area. The hydraulic control valve 33 operates in response to a control signal input from the controller 30. As a result, the controller 30 can forcibly reduce the pilot pressure output from the operating device 26 when the operating device 26 is operated by an operator. Therefore, even when the operating device 26 is being operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator corresponding to the operation of the operating device 26. Furthermore, for example, even when the operating device 26 is being operated, the controller 30 can reduce the pilot pressure output from the operating device 26 to make it lower than the pilot pressure output from the hydraulic control valve 31. Therefore, by controlling the hydraulic control valves 31 and 33, the controller 30 can reliably apply a desired pilot pressure to the pilot port of the control valve in the control valve 17, for example, regardless of the operation of the operating device 26. Therefore, the controller 30 can more appropriately realize the automatic operation function and remote control function of the excavator 100 by controlling the hydraulic control valve 33 in addition to the hydraulic control valve 31, for example.

[0057] <User Interface> As shown in FIG. 4, the user interface system of the shovel 100 according to this embodiment includes an operation device 26, an output device 50, and an input device 52.

[0058] The output device 50 outputs various types of information to a user of the shovel 100 (for example, an operator of the cabin 10, a work vehicle around the shovel 100, etc.).

[0059] For example, the output device 50 includes a lighting device, a display device 50A (see FIG. 6), etc. that output various types of information visually. The lighting device is, for example, a warning light, etc. The display device 50A is, for example, a liquid crystal display, an organic EL (Electroluminescence) display, etc. The lighting device and the display device 50A may be provided, for example, inside the cabin 10, and output various types of information visually to an operator or the like inside the cabin 10. Furthermore, the lighting device and the display device 50A may be provided, for example, on the side of the upper rotating body 3, etc., and output various types of information visually to workers or the like around the excavator 100.

[0060] Furthermore, for example, the output device 50 includes a sound output device 50B (see FIG. 6) that outputs various types of information auditorily. The sound output device 50B includes, for example, a buzzer, a speaker, etc. The sound output device 50B may be provided, for example, inside or outside the cabin 10, and may output various types of information auditorily to an operator inside the cabin 10 or to people (workers, etc.) around the excavator 100.

[0061] Furthermore, for example, the output device 50 may include a device that outputs various types of information in a tactile manner, such as by vibrating the cockpit.

[0062] The input device 52 receives various inputs from the user of the excavator 100, and signals corresponding to the received inputs are taken into the controller 30. The input device 52 is provided, for example, inside the cabin 10, and receives inputs from an operator or the like inside the cabin 10. The input device 52 may also be provided, for example, on the side of the upper rotating body 3, and receive inputs from workers or the like around the excavator 100.

[0063] For example, the input device 52 includes an operation input device that accepts operation inputs. The operation input device may include a touch panel mounted on the display device, a touch pad installed around the display device, a button switch, a lever, a toggle, a knob switch provided on the operation device 26 (lever device), and the like.

[0064] Furthermore, for example, the input device 52 may include an audio input device that accepts audio input from the user. The audio input device includes, for example, a microphone.

[0065] Furthermore, for example, the input device 52 may include a gesture input device that accepts gesture inputs from the user. The gesture input device includes, for example, an imaging device that captures an image of a gesture made by the user.

[0066] Furthermore, for example, the input device 52 may include a biometric input device that accepts biometric input from the user. The biometric input includes input of biometric information such as the user's fingerprint or iris.

[0067] <Communications> As shown in FIG. 4, the communication system of the shovel 100 according to this embodiment includes a communication device 60.

[0068] The communication device 60 is connected to a communication line NW and communicates with a device (for example, a management device 200) provided separately from the shovel 100. The device provided separately from the shovel 100 may include a device external to the shovel 100, as well as a portable terminal device brought into the cabin 10 by the user of the shovel 100. The communication device 60 may be, for example, a 4G (4 th Generation) and 5G (5 th The communication device 60 may include a mobile communication module that complies with standards such as the IEEE 802.11 Generation. The communication device 60 may also include, for example, a satellite communication module. The communication device 60 may also include, for example, a Wi-Fi communication module or a Bluetooth (registered trademark) communication module.

[0069] <Control system> 4, the control system of the shovel 100 according to this embodiment includes a controller 30. The control system of the shovel 100 according to this embodiment also includes an operating pressure sensor 29, an imaging device 40, an illumination device 70, a boom angle sensor S1, an arm angle sensor S2, a bucket angle sensor S3, a machine body attitude sensor S4, and a swing angle sensor S5.

[0070] The controller 30 performs various controls related to the shovel 100 .

[0071] The functions of the controller 30 may be realized by any hardware or any combination of hardware and software, etc. For example, as shown in Fig. 4, the controller 30 includes an auxiliary storage device 30A, a memory device 30B, a CPU (Central Processing Unit) 30C, and an interface device 30D, which are connected by a bus B1.

[0072] The auxiliary storage device 30A is a non-volatile storage means, and stores the programs to be installed as well as necessary files, data, etc. The auxiliary storage device 30A is, for example, a flash memory.

[0073] For example, when an instruction to start a program is received, the memory device 30B loads the program from the auxiliary storage device 30A so that it can be read by the CPU 30C. The memory device 30B is, for example, an SRAM (Static Random Access Memory).

[0074] The CPU 30C executes, for example, a program loaded into the memory device 30B, and realizes various functions of the controller 30 according to instructions from the program.

[0075] The interface device 30D is used, for example, as an interface for connecting to a communication line inside the shovel 100. The interface device 30D may include a plurality of different types of interface devices in accordance with the types of communication lines to be connected.

[0076] The programs that realize the various functions of the controller 30 are provided by, for example, a portable recording medium. In this case, the interface device 30D functions as an interface for reading data from the recording medium and writing data to the recording medium. The recording medium is, for example, a dedicated tool that is connected to a connector installed inside the cabin 10 by a detachable cable. The recording medium may also be a general-purpose recording medium such as an SD memory card or a USB (Universal Serial Bus) memory. The programs may also be downloaded from another computer (for example, the management device 200) external to the excavator 100 via a predetermined communication line and installed in the auxiliary storage device 30A.

[0077] Note that some of the functions of the controller 30 may be realized by another controller (control device). That is, the functions of the controller 30 may be realized in a distributed manner by a plurality of controllers. For example, a function related to image processing of images captured by the imaging device 40, a function related to detection of objects in the vicinity of the shovel 100, and a function related to ensuring the safety of the shovel 100 may be realized by different controllers. The function related to image processing of images captured by the imaging device 40 includes, for example, functions of a display processing unit 301 and an image correction unit 306, which will be described later. The function related to detection of objects in the vicinity of the shovel 100 includes, for example, functions of an object detection unit 302 and a detection determination unit 307, which will be described later. The function related to ensuring the safety of the shovel 100 includes, for example, functions of a safety control unit 304, which will be described later. Furthermore, the function related to detection of objects in the vicinity of the shovel 100 may be implemented in a distributed manner between a controller that realizes functions related to image processing of images captured by the imaging device 40 and a controller that realizes functions related to ensuring the safety of the shovel 100. Specifically, among the functions related to the detection of objects in the vicinity of the shovel 100, functions highly related to image processing, such as the function of recognizing objects from images captured by the imaging device 40 (the function of the object detection unit 302), may be installed in the former controller. On the other hand, among the functions related to the detection of objects in the vicinity of the shovel 100, functions highly related to ensuring the safety of the shovel 100, such as the function of finally determining whether or not a recognized object has been detected (the function of the detection determination unit 307), may be installed in the latter controller.

[0078] The operating pressure sensor 29 detects the pilot pressure on the secondary side (pilot line 27A) of the hydraulic pilot type operating device 26, i.e., the pilot pressure corresponding to the operating state of each driven element (hydraulic actuator) in the operating device 26. A detection signal of the pilot pressure by the operating pressure sensor 29 corresponding to the operating state of each driven element (hydraulic actuator HA) in the operating device 26 is taken into the controller 30.

[0079] If the operating device 26 is an electrical type, the operating pressure sensor 29 is omitted because the controller 30 can grasp the operating state of each driven element through the operating device 26 based on the operating signal received from the operating device 26.

[0080] The imaging device 40 acquires images for interpolating blind spots and places that are difficult to see from the operator's perspective around the shovel 100. The output (captured image) of the imaging device 40 is input to the controller 30.

[0081] The imaging device 40 is, for example, a monocular camera, a stereo camera, a depth camera, etc. Furthermore, the imaging device 40 may acquire three-dimensional data (for example, point cloud data or surface data) representing the positions and outer shapes of objects around the shovel 100 within a predetermined imaging range (angle of view) based on the captured image.

[0082] For example, as shown in FIGS. 1 and 2 , the imaging device 40 includes a camera 40F that captures images in front of the upper rotating body 3, a camera 40B that captures images behind the upper rotating body 3, a camera 40L that captures images to the left of the upper rotating body 3, and a camera 40R that captures images to the right of the upper rotating body 3. This allows the operator to view images captured by the cameras 40B, 40L, and 40R and peripheral images such as processing images generated based on the captured images through the display device 50A or the remote control display device, and check the conditions to the left, right, and rear of the upper rotating body 3. Furthermore, by viewing images captured by the camera 40F and peripheral images such as processing images generated based on the captured images through the remote control display device, the operator can remotely operate the excavator 100 while checking the operation of the attachment AT including the bucket 6. Hereinafter, the cameras 40F, 40B, 40L, and 40R may be collectively or individually referred to as "camera 40X."

[0083] Each imaging device 40 (camera 40X) includes an imaging element 41 and an image processing engine 42. The camera 40X generates a captured image using the image processing engine 42 based on the output (electrical signal) of the imaging element 41, and outputs the generated image (captured image). The image processing engine 42 is a computer dedicated to image processing, including, for example, a CPU, a memory device, and an auxiliary storage device, and realizes various types of image processing by executing programs installed in the auxiliary storage device on the CPU. Hereinafter, the function of the image processing engine 42 to generate a captured image will be referred to as the captured image generation function, and the captured image of the imaging device 40 refers to an image generated by the captured image generation function.

[0084] The two-dot chain lines in FIG. 2 represent the angles of view (image pickup ranges) of the cameras 40F, 40B, 40L, and 40R as viewed from above.

[0085] The illumination device 70 illuminates the imaging range of the imaging device 40 with predetermined light under the control of the controller 30. The predetermined light is, for example, visible light. The predetermined light may also be, for example, infrared light. As long as the illumination device 70 can illuminate the entire imaging range of the imaging device 40, there may be one or more illumination devices 70. For example, an illumination device 70 is provided for each camera 40X and is installed on the upper rotating body 3 so as to be close to the cameras 40X.

[0086] The irradiation device 70 may be omitted.

[0087] The boom angle sensor S1 acquires detection information related to the attitude angle of the boom 4 (hereinafter referred to as "boom angle") relative to a predetermined reference (for example, a horizontal plane or a state at either end of the movable angle range of the boom 4). The boom angle sensor S1 may include, for example, a rotary encoder, an acceleration sensor, an angular velocity sensor, a six-axis sensor, an IMU (Inertial Measurement Unit), etc. The boom angle sensor S1 may also include a cylinder sensor capable of detecting the extension / retraction position of the boom cylinder 7.

[0088] The arm angle sensor S2 acquires detection information relating to the attitude angle of the arm 5 (hereinafter referred to as "arm angle") relative to a predetermined reference (for example, a straight line connecting the connection points at both ends of the boom 4 or a state at either end of the movable angle range of the arm 5). The arm angle sensor S2 may include, for example, a rotary encoder, an acceleration sensor, an angular velocity sensor, a six-axis sensor, an IMU, etc. The arm angle sensor S2 may also include a cylinder sensor capable of detecting the extension / retraction position of the arm cylinder 8.

[0089] The bucket angle sensor S3 acquires detection information related to the attitude angle of the bucket 6 (hereinafter referred to as "bucket angle") relative to a predetermined reference (for example, a straight line connecting the connection points at both ends of the arm 5 or a state at either end of the movable angle range of the bucket 6). The bucket angle sensor S3 may include, for example, a rotary encoder, an acceleration sensor, an angular velocity sensor, a six-axis sensor, an IMU, etc. The bucket angle sensor S3 may also include a cylinder sensor capable of detecting the extension / retraction position of the bucket cylinder 9.

[0090] The machine body attitude sensor S4 acquires detection information relating to the attitude state of the machine body of the shovel 100, including the undercarriage 1 and the upper rotating body 3. The machine body attitude sensor S4 is mounted, for example, on the upper rotating body 3, and acquires detection information relating to the tilt angle of the upper rotating body 3 with respect to the horizontal plane and the attitude angle around the rotation axis (i.e., the orientation of the upper rotating body 3 with the ground as the reference). The machine body attitude sensor S4 may include, for example, an acceleration sensor (tilt sensor), an angular velocity sensor, a six-axis sensor, an IMU, etc.

[0091] The swing angle sensor S5 acquires detection information relating to the swing angle of the upper swing body 3 (i.e., the orientation of the upper swing body 3) relative to the lower traveling body 1. The swing angle sensor S5 includes, for example, a potentiometer, a rotary encoder, a resolver, etc.

[0092] Furthermore, for example, the shovel 100 may include a positioning device capable of measuring the absolute position of the shovel 100 itself. The positioning device is, for example, a Global Navigation Satellite System (GNSS) sensor. This can improve the accuracy of estimating the attitude state of the shovel 100.

[0093] Furthermore, for example, the shovel 100 may include, in addition to the imaging device 40, a distance sensor that detects the distance to an object in the vicinity of the shovel 100. The distance sensor includes, for example, a LIDAR (Light Detecting and Ranging), a millimeter-wave radar, an ultrasonic sensor, an infrared sensor, a distance image sensor, etc. This allows the controller 30 to detect objects in the vicinity of the shovel 100 using, for example, the output of the distance sensor in addition to the output of the imaging device 40.

[0094] Note that some or all of the boom angle sensor S1, arm angle sensor S2, bucket angle sensor S3, machine body attitude sensor S4, and swing angle sensor S5 may be omitted. For example, if a remote control function or an automatic driving function is not employed, it may not be necessary to estimate the attitude state of the attachment AT or the machine body (upper rotating structure 3) of the shovel 100. Also, it may be possible to estimate the attitude state of the shovel 100 from information about the periphery of the shovel 100 acquired by, for example, the imaging device 40 or a distance sensor described below. Specifically, the information about the periphery of the shovel 100 acquired by the imaging device 40 or a distance sensor described below may include information about the position and shape of surrounding objects and attachments as seen from the machine body (upper rotating structure 3). In this case, the controller 30 may estimate the attitude state of the attachment AT or the machine body (upper rotating structure 3) from that information, depending on the required accuracy.

[0095] [Hardware configuration of management device] Next, the hardware configuration of the management device 200 will be described with reference to FIG.

[0096] FIG. 5 is a diagram illustrating an example of the hardware configuration of the management device 200 according to this embodiment.

[0097] The functions of the management device 200 are realized by any hardware or any combination of hardware and software, etc. For example, as shown in Fig. 5, the management device 200 includes an external interface 201, an auxiliary storage device 202, a memory device 203, a CPU 204, a communication interface 206, an input device 207, and a display device 208, which are connected via a bus B2.

[0098] The external interface 201 functions as an interface for reading data from the recording medium 201A and writing data to the recording medium 201A. Examples of the recording medium 201A include a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disc), a BD (Blu-ray (registered trademark) Disc), an SD memory card, a USB memory, etc. This allows the management device 200 to read various data used in processing through the recording medium 201A, store the data in the auxiliary storage device 202, and install programs that realize various functions.

[0099] The management device 200 may acquire various data and programs from external devices via the communication interface 206.

[0100] The auxiliary storage device 202 stores various installed programs as well as files and data required for various processes. The auxiliary storage device 202 includes, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc.

[0101] When an instruction to start a program is received, the memory device 203 reads and stores the program from the auxiliary storage device 202. The memory device 203 includes, for example, a dynamic random access memory (DRAM) or a static random access memory (SRAM).

[0102] The CPU 204 executes various programs loaded from the auxiliary storage device 202 to the memory device 203, and realizes various functions related to the management device 200 in accordance with the programs.

[0103] The high-speed arithmetic unit 205 performs arithmetic processing at a relatively high speed in cooperation with the CPU 204. The high-speed arithmetic unit 205 includes, for example, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

[0104] The high speed calculation device 205 may be omitted depending on the required calculation processing speed.

[0105] The communication interface 206 is used as an interface for connecting to an external device so that communication is possible. This allows the management device 200 to communicate with an external device such as the shovel 100 through the communication interface 206. The communication interface 206 may have multiple types of communication interfaces depending on the communication method between the connected device and the like.

[0106] The input device 207 receives various inputs from a user. For example, the input device 207 includes an input device (remote control device) for an operator to perform remote control.

[0107] The input device 207 includes, for example, an operation input device that accepts mechanical operation input from a user. The operation input device includes, for example, a button, a toggle, a lever, etc. The operation input device also includes, for example, a touch panel mounted on the display device 208, a touch pad provided separately from the display device 208, etc.

[0108] The input device 207 also includes, for example, a voice input device capable of receiving voice input from the user. The voice input device includes, for example, a microphone capable of collecting the user's voice.

[0109] The input device 207 includes, for example, a gesture input device capable of receiving a gesture input from a user. The gesture input device includes, for example, a camera capable of capturing an image of a user's gesture.

[0110] The input device 207 includes, for example, a biometric input device capable of accepting biometric input from a user. The biometric input device includes, for example, a camera capable of acquiring image data containing information about a user's fingerprint or iris.

[0111] The display device 208 displays an information screen or an operation screen for the user. For example, the display device 208 includes the above-mentioned remote control display device. The display device 208 is, for example, a liquid crystal display or an organic EL (Electroluminescence) display.

[0112] [Example of excavator functional configuration] Next, an example of the functional configuration of the shovel 100 will be described with reference to FIG.

[0113] FIG. 6 is a functional block diagram showing an example of the functional configuration of the shovel 100 according to this embodiment.

[0114] 6, the controller 30 includes, as functional units, a display processing unit 301, an object detection unit 302, a position estimation unit 303, a safety control unit 304, and an illumination control unit 305. The controller 30 may also include, as functional units, an image correction unit 306. The functions of the display processing unit 301, the object detection unit 302, the position estimation unit 303, the safety control unit 304, and the illumination control unit 305 are realized, for example, by loading a program installed in the auxiliary storage device 30A into the memory device 30B and executing it on the CPU 30C.

[0115] The display processing unit 301 displays a surrounding image on the display device 50A inside the cabin 10 based on the captured image input from the imaging device 40.

[0116] The display processing unit 301 may display a peripheral image corresponding to the imaging range of any one of cameras 40F, 40B, 40L, and 40R on the display device 50A, or may display the imaging ranges of any two or more of cameras 40F, 40B, 40L, and 40R on the display device 50A. For example, the display processing unit 301 displays a peripheral image including the imaging ranges of at least cameras 40B and 40R out of cameras 40F, 40B, 40L, and 40R on the display device 50A. This is because the areas behind and to the right of the upper rotating body 3, which correspond to the imaging ranges of cameras 40B and 40R, are likely to be blind spots when viewed from the operator in the cabin 10.

[0117] The object detection unit 302 (an example of a detection unit) detects a predetermined object (hereinafter, "monitored object") among the monitored objects around the shovel 100 based on the output (captured image) of the imaging device 40. Specifically, the object detection unit 302 may recognize the monitored object from the captured image of the imaging device 40 and identify the position (area) in which the monitored object is captured. The object detection unit 302 may also recognize the monitored object from an image (hereinafter, "corrected captured image") after the image captured by the imaging device 40 (camera 40X) has been corrected by the image correction unit 306 and identify the position (area) in which the monitored object is captured. In other words, detecting a monitored object may mean recognizing the monitored object captured in the captured image of the imaging device 40 and identifying the position (area) in the input image in which the monitored object is included. Hereinafter, the image input to the object detection unit 302 (the captured image itself output from the imaging device 40 or the corrected captured image output from the image correction unit 306) may be referred to as the "input image" for convenience.

[0118] The monitored objects may include people such as workers working around the shovel 100 and work site supervisors. The monitored objects may also include any object (obstacle) other than people at the work site. Obstacles other than people at the work site may include, for example, specific topography such as holes, ditches, and piles of earth and sand, as well as fixed obstacles (i.e., obstacles that do not move under their own power) such as road cones, fences, utility poles, temporarily stored materials, and temporary offices at the work site. The obstacles other than people at the work site may also include, for example, movable obstacles such as other work machines and work vehicles. The number of types of monitored objects to be detected by the object detection unit 302 may be one or more. The following description will focus on the case where the monitored object is a person.

[0119] The object detection method by object detection unit 302 will be described in detail later.

[0120] The function of the object detection unit 302 may be switched between ON (enabled) and OFF (disabled) in response to a predetermined input by an operator or the like via the input device 52. Alternatively, the function of the object detection unit 302 may be transferred to the camera 40X, and the detection result may be transmitted to the controller 30. Alternatively, the function of the object detection unit 302 may be transferred to the display device 50A, and the detection result may be transferred to the controller 30. In this case, the image captured by the camera 40X is directly input to the display device 50A. Therefore, in this case, the function of the display processing unit 301 may also be transferred to the display device 50A. Alternatively, the function of the object detection unit 302 may be transferred to the imaging device 40. In this case, the function of the object detection unit 302 is divided into object detection units that detect a monitored object based on the images captured by the cameras 40F, 40B, 40L, and 40R, and the object detection units are built into each of the cameras 40F, 40B, 40L, and 40R (see FIG. 25 described later).

[0121] When the object detection unit 302 detects a monitored object from the image captured by the imaging device 40, the position estimation unit 303 estimates the actual position of the detected monitored object, i.e., the position of the monitored object in the vicinity of the shovel 100 (hereinafter referred to as the "actual position").

[0122] Specifically, the position estimation unit 303 estimates the actual position of the monitored object based on the detected position (detection area) in the captured image of the monitored object identified by the object detection unit 302. Furthermore, when multiple monitored objects are detected by the object detection unit 302, the position estimation unit 303 estimates the actual position of each of the multiple monitored objects.

[0123] For example, when the camera 40X is a monocular camera, the position estimation unit 303 identifies a reference point (detection position) in the detection area of ​​the monitored object in the input image, and estimates the direction as seen from the shovel 100 (upper rotating body 3) based on the horizontal position of the reference point in the input image. The position estimation unit 303 also estimates the distance from the shovel 100 to the monitored object (specifically, the distance in the direction along the work plane on which the shovel 100 is located) based on the size (e.g., the size in the vertical direction) of the detection area of ​​the monitored object in the input image. This is because there is a correlation in which the size of a recognized monitored object in the input image becomes smaller as the monitored object becomes farther away from the shovel 100 (upper rotating body 3). Specifically, since there is a range of expected sizes for monitored objects (e.g., a range of expected human heights), the correlation between the position of the monitored object as seen from the shovel 100 within the expected size range and the size of the monitored object in the input image can be specified in advance. Therefore, the object detection unit 302 can estimate the actual position of the recognized monitored object based on, for example, a map or a conversion formula that is stored in advance in the auxiliary storage device 30A or the like and that indicates the correlation between the size of the monitored object on the input image and the distance as seen from the upper rotating body 3. Therefore, the position estimation unit 303 can estimate the position of the recognized monitored object by estimating the direction and distance of the monitored object as seen from the excavator 100 (upper rotating body 3).

[0124] Furthermore, for example, when the camera 40X is a monocular camera, the position estimation unit 303 may estimate the actual position of the monitored object by projective transformation of the input image onto the plane, assuming that the monitored object exists on the same plane as the lower traveling structure 1. Specifically, the position estimation unit 303 may identify a reference point (detection position) that represents the contact point of the monitored object with the ground, and calculate the actual position of the monitored object with respect to the reference point by projective transformation that is defined according to the installation position and installation angle of the camera 40X with respect to the upper rotating structure 3. In this case, a certain portion (for example, a certain pixel) that constitutes the input image is associated one-to-one with a certain position on the same plane as the excavator 100 (lower traveling structure 1).

[0125] Furthermore, for example, if the camera 40X is a stereo camera, the actual position of the monitored object is estimated based on the difference (parallax) between the reference positions in the detection areas of the monitored object for each of the two captured images.

[0126] The safety control unit 304 (an example of a control unit) performs control related to the functional safety of the shovel 100.

[0127] The safety control unit 304 activates the safety function, for example, when the object detection unit 302 detects a monitored object within a predetermined range around the shovel 100. Specifically, the safety control unit 304 may activate the safety function when the actual position (estimated value) of the monitored object estimated by the position estimation unit 303 is within a predetermined range around the shovel 100.

[0128] The safety functions may include, for example, a notification function that outputs an alarm or the like to at least one of the inside of the cabin 10, the outside of the cabin 10, and a remote operator or monitor of the shovel 100, thereby notifying them of the detection of a monitored object. This makes it possible to alert the operator inside the cabin 10, workers around the shovel 100, and the operator or monitor who remotely operates or remotely monitors the shovel 100 that a monitored object is present in the monitored area around the shovel 100. Hereinafter, the notification function to the inside of the cabin 10 (operator, etc.) may be referred to as an "internal notification function," the notification function to the outside of the shovel 100 (workers, etc.) may be referred to as an "external notification function," and the notification function to the operator or monitor who remotely operates or remotely monitors the shovel 100 may be referred to as a "remote notification function," respectively, to distinguish between them.

[0129] The safety function may also include, for example, an operation limiting function that limits the operation of the shovel 100 in response to an operation command corresponding to the operation of the operating device 26, remote operation, or the automatic driving function. This forcibly limits the operation of the shovel 100, thereby reducing the possibility of the shovel 100 approaching or coming into contact with surrounding objects. The operation limiting function may also include an operation deceleration function that slows down the operation speed of the shovel 100 in response to an operation command corresponding to the operation of the operating device 26, remote operation, or the automatic driving function, compared to normal. The operation limiting function may also include an operation stop function that stops the operation of the shovel 100 and maintains the stopped state, regardless of an operation command corresponding to the operation of the operating device 26, remote operation, or the automatic driving function.

[0130] The safety control unit 304 activates the alarm function, for example, when the object detection unit 302 detects a monitored object within a predetermined range (hereinafter referred to as the "alert range") around the shovel 100. The alert range is, for example, a range in which the distance D from a predetermined part of the shovel 100 is equal to or less than a threshold value Dth1. The predetermined part of the shovel 100 is, for example, the upper rotating body 3. Furthermore, the predetermined part of the shovel 100 may be, for example, the bucket 6 or hook at the tip of the attachment AT. The threshold value Dth1 may be constant regardless of the direction as viewed from the predetermined part of the shovel 100, or may change depending on the direction as viewed from the predetermined part of the shovel 100.

[0131] The safety control unit 304 controls the sound output device 50B, for example, to activate an internal notification function or an external notification function using sound (i.e., an auditory method) to at least one of the inside and the outside of the cabin 10. At this time, the safety control unit 304 may vary the pitch, sound pressure, tone color, the sound cycle when the sound is periodically sounded, the content of the sound, and the like of the output sound according to various conditions.

[0132] The safety control unit 304 also activates an internal notification function using, for example, a visual method. Specifically, the safety control unit 304 may control the display device 50A inside the cabin 10 via the display processing unit 301 to display an image indicating that a monitored object has been detected, along with a peripheral image, on the display device 50A. The safety control unit 304 may also highlight, via the display processing unit 301, the monitored object shown in the peripheral image displayed on the display device 50A inside the cabin 10, or the position on the peripheral image corresponding to the detected monitored object. More specifically, the safety control unit 304 may superimpose a frame surrounding the detected monitored object on the peripheral image displayed on the display device 50A inside the cabin 10, or superimpose a marker on a position on the peripheral image corresponding to the actual location of the detected monitored object. This allows the display device 50A to realize a visual notification function for the operator. The safety control unit 304 may also use a warning light, lighting device, or the like inside the cabin 10 to notify the operator, etc., inside the cabin 10 that a monitored object has been detected.

[0133] The safety control unit 304 may also activate an external alarm function by a visual method, for example, by controlling an output device 50 (for example, a lighting device such as a headlight or a display device 50A) provided on the side of the house of the upper rotating body 3, etc. The safety control unit 304 may also activate an internal alarm function by a tactile method, for example, by controlling a vibration generating device that vibrates the operator's seat in which the operator sits. This allows the controller 30 to make the operator, workers and supervisors around the shovel 100 aware that a monitored object (for example, a person such as a worker) is present in a relatively close location around the shovel 100. Therefore, the controller 30 can urge the operator to check the safety status around the shovel 100, or urge workers and others in the monitored area to evacuate from the monitored area.

[0134] Furthermore, the safety control unit 304 may activate the remote alarm function by, for example, transmitting a command signal indicating activation of the alarm function to the management device 200 via the communication device 60. In this case, when the management device 200 receives a command signal from the shovel 100 via the communication interface 206, it may output an alarm by a visual or audio method via the display device 208 or the like. This allows an operator or a supervisor who remotely operates or remotely monitors the shovel 100 via the management device 200 to know that a monitored object has entered the alarm range around the shovel 100.

[0135] The remote notification function of the safety control unit 304 may be transferred to the management device 200. In this case, the management device 200 receives information from the shovel 100 regarding the detection status of the monitored object by the object detection unit 302 and the estimation result of the actual position of the monitored object by the position estimation unit 303. Then, based on the received information, the management device 200 determines whether or not a monitored object has entered the notification range, and activates the external notification function if a monitored object is present within the notification range.

[0136] Furthermore, the safety control unit 304 may change the notification mode (that is, the way of notification) depending on the positional relationship between the monitored object detected within the notification range and the upper rotating body 3.

[0137] For example, when a monitored object detected within the notification range by the object detection unit 302 is located relatively far from a predetermined portion of the shovel 100, the safety control unit 304 may output a relatively low-urgency alarm (hereinafter, "alarm at a warning level") that calls attention to the monitored object. Hereinafter, for convenience, a range within the notification range in which the distance to the predetermined portion of the shovel 100 is relatively far, i.e., a range corresponding to an alert-level alarm, may be referred to as an "alert range." On the other hand, when a monitored object detected within the notification range by the object detection unit 302 is located relatively close to the predetermined portion of the shovel 100, the safety control unit 304 may output a relatively high-urgency alarm (hereinafter, "alarm at an alert level") that notifies the user that the monitored object is approaching the predetermined portion of the shovel 100 and that the risk is increasing. Hereinafter, a range within the notification range in which the distance to the predetermined portion of the shovel 100 is relatively close, i.e., a range corresponding to an alert-level alarm, may be referred to as an "alert range."

[0138] In this case, the safety control unit 304 may change the pitch, sound pressure, tone, sound cycle, etc. of the sound output from the sound output device 50B between the caution level alarm and the alert level alarm. Furthermore, the safety control unit 304 may change the color, shape, size, presence or absence of blinking, blinking cycle, etc. of the image indicating that a monitored object has been detected and the image (e.g., a frame, marker, etc.) that highlights the monitored object or the position of the monitored object on the peripheral image displayed on the display device 50A between the caution level alarm and the alert level alarm. In this way, the controller 30 allows the operator, etc. to grasp the level of urgency, in other words, the proximity of the monitored object to a predetermined portion of the excavator 100, based on the difference in the alert sound (alarm sound) output from the sound output device 50B and the alert image displayed on the display device 50A.

[0139] After the alarm function starts operating, the safety control unit 304 may stop the alarm function if the monitored object that was detected by the object detection unit 302 is no longer detected within the alarm range. Furthermore, after the alarm function starts operating, the safety control unit 304 may stop the alarm function if a predetermined input for canceling the operation of the alarm function is received via the input device 52.

[0140] Furthermore, the safety control unit 304 activates the operation restriction function, for example, when the object detection unit 302 detects a monitored object within a predetermined range around the shovel 100 (hereinafter referred to as the "operation restriction range"). The operation restriction range is set, for example, to be the same as the notification range described above. Furthermore, the operation restriction range may be set, for example, to a range whose outer edge is relatively closer to a predetermined part of the shovel 100 than the notification range. In this way, the safety control unit 304 can, for example, first activate the notification function when a monitored object enters the notification range from the outside, and then further activate the operation restriction function when the monitored object enters the inner operation restriction range. Therefore, the controller 30 can activate the notification function and the operation restriction function in stages in accordance with the movement of the monitored object inward within the monitoring area.

[0141] Specifically, the safety control unit 304 may activate the operation restriction function when a monitored object is detected within an operation restriction range where the distance D from a predetermined portion of the shovel 100 is within a threshold Dth2 (≦Dth1). The threshold Dth2 may be constant regardless of the direction as viewed from the predetermined portion of the shovel 100, or may change depending on the direction as viewed from the predetermined portion of the shovel 100.

[0142] The operation limit range also includes at least one of an operation deceleration range in which the operation speed of the shovel 100 is slower than normal in response to an operation command corresponding to the operation of the operating device 26, remote operation, or the automatic operation function, and an operation stop range in which the operation of the shovel 100 is stopped and maintained in a stopped state regardless of an operation command corresponding to the operation of the operating device 26, remote operation, or the automatic operation function. For example, when the operation limit range includes both the operation deceleration range and the operation stop range, the operation stop range is a range within the operation limit range that is close to a predetermined part of the shovel 100. The operation deceleration range is a range within the operation limit range that is set outside the operation stop range.

[0143] The safety control unit 304 activates an operation limiting function that limits the operation of the shovel 100 by controlling the hydraulic control valve 31. In this case, the safety control unit 304 may limit the operation of all driven elements (i.e., corresponding hydraulic actuators), or may limit the operation of some of the driven elements (hydraulic actuators). This allows the controller 30 to slow down or stop the operation of the shovel 100 when a monitored object is present around the shovel 100. Therefore, the controller 30 can prevent the occurrence of contact between the monitored object around the shovel 100 and the shovel 100 or the suspended load. Furthermore, the safety control unit 304 may activate the operation limiting function (operation stop function) by controlling an electromagnetic switching valve (not shown) in the pilot line 25 to cut off the pilot line 25.

[0144] Furthermore, after the operation of the operation restriction function has started, the safety control unit 304 may stop the operation restriction function when the monitored object that had been detected by the object detection unit 302 is no longer detected within the operation restriction range. Furthermore, after the operation of the operation restriction function has started, the safety control unit 304 may stop the operation restriction function when a predetermined input for canceling the operation of the operation restriction function is received via the input device 52. The content of the input to the input device 52 for canceling the operation of the alarm function and the content of the input for canceling the operation of the operation restriction function may be the same as or different from each other.

[0145] Furthermore, the function of the safety control section 304 may be switched between ON (enabled) and OFF (disabled) in response to a predetermined input by an operator or the like via the input device 52.

[0146] The irradiation control unit 305 controls the irradiation device 70 .

[0147] As described above, when the irradiation device 70 is omitted, the irradiation control unit 305 is also omitted as a matter of course.

[0148] The image correction unit 306 (an example of a correction unit) performs a predetermined correction on the output (captured image) of the imaging device 40 (camera 40X), and outputs the corrected captured image to the object detection unit 302.

[0149] [Object detection method overview] Next, an overview of an object detection method performed by the object detection unit will be described with reference to FIGS.

[0150] Fig. 7 is a diagram illustrating a specific example of an object detection method. Figs. 8 to 13 are diagrams illustrating first to sixth examples (images 800 to 1300) of images of training data. Specifically, Figs. 8 to 13 are images captured by a camera of the same type as camera 40X installed in the same position on an excavator of the same model as excavator 100, and the side of upper revolving body 3 is captured on the near side (lower end) of the captured image. Figs. 14 and 15 are diagrams illustrating an example and another example of worker W.

[0151] The object detection unit 302 detects a monitored object from an image captured by the camera 40X by simply applying image processing techniques such as shape detection and pattern recognition (template matching).

[0152] As shown in FIG. 7 , the object detection unit 302 detects a monitored object from an image captured by the camera 40X by applying machine learning in addition to image processing techniques. Specifically, the object detection unit 302 uses a trained model LM that has undergone machine learning to learn the features of the monitored object captured in the input image to output a rectangular frame (hereinafter, a “detection frame”) representing an area in which the monitored object is captured and a label representing the type of monitored object from the input image (the image captured by the camera 40X or a corrected image thereof). The labels include, for example, a label representing the absence of a monitored object and a label set for each type of monitored object. Only one label may be set for each type of monitored object, or multiple labels may be set, as described below. Furthermore, when multiple types of monitored objects exist, a single trained model LM may be configured to be able to output labels for all types of monitored objects, i.e., to be able to detect all types of monitored objects. Alternatively, multiple trained models LM may be provided that can detect only some of all types of monitored objects. For example, a trained model LM exists for each type of monitored object, and the labels for each trained model LM may consist only of labels indicating the presence of a certain type of monitored object and labels indicating the absence of that type of monitored object.

[0153] The trained model LM is generated by applying supervised learning to a base learning model. Specifically, the trained model LM is generated by having the base learning model perform machine learning on a collection of training data (training dataset) that is a combination of images as input and correct answers (detection frames and labels) as output. The trained model LM may also be generated (updated) by additionally training a new training dataset on an existing training model LM. Naturally, the input images included in the training dataset include both images that include (show) a monitored object and images that do not include (show) a monitored object.

[0154] The trained model LM is generated, for example, by an external device such as the management device 200, and is written to the auxiliary storage device 30A from a predetermined recording medium via the interface device 30D when the shovel 100 is manufactured. The trained model LM may also be downloaded to the shovel 100 from an external device such as the management device 200 via a predetermined communication line and registered in the auxiliary storage device 30A of the controller 30. This allows the object detection unit 302 to detect a monitored object using the trained model LM registered in the auxiliary storage device 30A.

[0155] The trained model LM may also be updated by installing update data from a predetermined recording medium into the auxiliary storage device 30A via the interface device 30D. The trained model LM may also be updated by downloading update data from an external device such as the management device 200 to the shovel 100 via a predetermined communication line and installing it into the auxiliary storage device 30A. This allows the object detection unit 302 to detect a monitored object using the latest updated trained model LM.

[0156] For example, the object detection unit 302 detects a monitored object from an input image using a support vector machine (SVM) that has undergone machine learning to learn the tendency of image features of the monitored object appearing in the image. In this case, the trained model LM includes a processing unit that extracts image features from an image captured by the camera 40X as a pre-processing step. The image features are, for example, histogram of oriented gradients (HOG) features.

[0157] Furthermore, for example, the object detection unit 302 detects a monitored object from an input image using a trained model LM based on machine learning using a deep neural network (DNN), i.e., deep learning. Specifically, the object detection unit 302 may detect a monitored object from an input image using a trained model LM based on deep learning using a convolutional neural network (CNN). The CNN is configured by connecting multiple combinations of convolution layers that perform convolution processing and pooling layers that perform pooling processing using activation functions, and a final fully connected layer makes a final decision based on feature amounts (feature maps). The activation function is, for example, a rectified linear unit (ReLU). This allows the trained model LM to handle input images as they are, without requiring pre-processing.

[0158] For example, the object detection unit 302 uses a CNN-based trained model LM to generate regions of candidate monitored objects from an input image and classify these candidates into labels, thereby detecting the monitored object. That is, the CNN-based trained model LM may be, for example, a classification model that treats the detection of a monitored object from an input image as a classification problem, generates regions of candidate monitored objects from an image captured by the camera 40X, and classifies these candidates into labels. The classification model is, for example, R (Region-based)-CNN or its derivatives (Fast R-CNN, Faster R-CNN, etc.).

[0159] Furthermore, for example, the object detection unit 302 detects a monitored object by simultaneously recognizing the monitored object and identifying its position (area) from an input image using a trained model LM based on CNN. That is, the trained model LM based on CNN may be, for example, a regression model that treats the detection of a monitored object from an image captured by the camera 40X as a regression problem and simultaneously recognizes the monitored object and identifies its position (area) from the input image. Examples of the regression model include YOLO (You Only Look Once) and SSD (Single Shot Detector).

[0160] The learned model LM is generated, for example, by machine learning a base learned model or an existing learned model LM so that it can detect monitored objects with different poses.

[0161] For example, as shown in Fig. 8, image 800 of the training data shows worker W standing upright and facing the camera. As shown in Fig. 9, image 900 of the training data shows worker W crouching and facing the camera. As shown in Fig. 10, image 1000 of the training data shows worker W crouching and facing the camera. The trained model LM is machine-trained using a training dataset that includes images, such as images 800 to 1000, in which monitored objects are in different poses, and is thereby able to detect monitored objects (people in this example) in different poses from input images.

[0162] Furthermore, the learned model LM may be generated, for example, by machine learning a base learned model or an existing learned model LM so that it can detect monitored objects in different orientations.

[0163] For example, as shown in Fig. 11, image 1100 of the training data shows worker W standing upright and facing sideways (right) relative to the camera. As shown in Fig. 12, image 1200 of the training data shows worker W crouching with his back to the camera. As shown in Fig. 13, image 1300 of the training data shows worker W crouching and facing sideways (left) relative to the camera. The trained model LM is machine-trained using a training dataset including images in which monitored objects are viewed in different orientations, such as images 800 and 1100, images 900 and 1200, and images 1000 and 1300, and is thereby able to detect monitored objects in different orientations from input images.

[0164] Furthermore, the trained model LM may be generated by, for example, machine learning a base training model or an existing trained model LM so that it can detect monitored objects whose orientations and / or postures are different from each other. Specifically, the trained model LM is trained by machine learning using a training dataset including images in which monitored objects whose orientations and postures are different from each other, such as images 800 to 1300, so that it can detect monitored objects whose orientations and postures are different from each other from input images.

[0165] Furthermore, when the monitored object is a person, the trained model LM may be generated by machine learning so as to be able to detect (recognize) a person or item worn by a person that is relatively frequently worn by workers around the shovel 100. This allows the controller 30 to appropriately recognize the characteristics of the item, detect the person or the item worn by the person, and grasp the presence of the person on the shovel 100, even in the case of a captured image in which it is difficult to distinguish between work clothes or the like and the background.

[0166] For example, as shown in Fig. 14, the trained model LM may be generated by machine learning to detect (recognize) a person wearing a helmet (worker W) and the helmet HMT worn by the person. Specifically, the trained model LM is trained by machine learning using a training dataset that includes a large number of images in which the worker W wearing the helmet HMT is captured, thereby enabling the person wearing the helmet and the helmet worn by the person to be detected from the input image.

[0167] Furthermore, for example, as shown in FIG. 15, the trained model LM may be generated by machine learning to detect (recognize) a person (worker W) wearing high-visibility safety clothing RV and the high-visibility safety clothing RV worn by the person. The high-visibility safety clothing RV is configured to relatively increase the visibility of the wearer from those around the wearer. For example, a retroreflective material is attached to the high-visibility safety clothing RV. Furthermore, the high-visibility safety clothing RV is made of fluorescent fabric, for example, fluorescent yellow or fluorescent green. Specifically, the trained model LM can detect a person wearing high-visibility safety clothing and the high-visibility safety clothing worn by the person from an input image by undergoing machine learning using a training dataset including a large number of images of a worker W wearing high-visibility safety clothing RV.

[0168] Furthermore, for example, as shown in FIG. 15, the trained model LM may be trained to detect (recognize) a person (worker W) wearing both a helmet HMT and high-visibility safety clothing RV, and both the high-visibility safety clothing RV and the helmet HMT. In this case, the trained model LM may be capable of outputting both a label indicating the presence of a helmet and a label indicating the presence of high-visibility safety clothing. Specifically, the trained model LM is trained on a training dataset including a large number of images of a worker W wearing a helmet HMT and high-visibility safety clothing RV, and is thereby able to detect a person wearing a helmet and high-visibility safety clothing, and the helmet and high-visibility safety clothing worn by the person, from an input image.

[0169] The teacher dataset may also include, for example, images of a monitored object captured by a camera of the same model as the camera 40X. The teacher dataset may also include, for example, images of a monitored object captured by a camera of the same model as the camera 40X, which is installed in substantially the same position and with substantially the same attitude on an excavator of the same model as the shovel 100, as shown in FIGS. 8 to 13. As a result, the trained model LM is machine-trained to be able to detect a monitored object from an image captured by the camera 40X, taking into account how the monitored object appears in the image captured by the camera 40X. This allows the object detection unit 302 to detect a monitored object with higher accuracy.

[0170] The training dataset also includes, for example, images in which a monitored object is captured in each of multiple different types of backgrounds. For example, the multiple types of backgrounds include dirt ground, asphalt ground, forests, houses, urban buildings, etc. As a result, the trained model LM is machine-trained to be able to detect a monitored object from an input image, taking into account how the monitored object appears depending on the background. This allows the object detection unit 302 to detect the monitored object with higher accuracy.

[0171] The training data set also includes, for example, images of a monitored object captured in each of a plurality of different time periods. The plurality of time periods includes, for example, a morning time period (e.g., 6:00 AM to 10:00 AM), a daytime time period (e.g., 10:00 AM to 3:00 PM), an evening time period (e.g., 4:00 PM to 7:00 PM), and a night time period (e.g., 7:00 PM to 6:00 AM). As a result, the trained model LM is machine-trained to be able to detect a monitored object from an input image, taking into account how the monitored object appears depending on the time period. This allows the object detection unit 302 to detect the monitored object with higher accuracy.

[0172] The training dataset also includes images of the monitored object captured under different lighting conditions, for example, during the night. The lighting conditions include, for example, the illuminance level and color of the lighting. As a result, the trained model LM is trained to detect the monitored object from the input image, taking into account how the monitored object appears under different lighting conditions. This allows the object detection unit 302 to detect the monitored object with higher accuracy.

[0173] The training dataset also includes, for example, images of monitored objects captured under a plurality of different weather conditions. The plurality of weather conditions may include, for example, sunny, cloudy, rainy, and snowy. As a result, the trained model LM is machine-trained to be able to detect monitored objects from input images, taking into account how monitored objects appear depending on the weather. This allows the object detection unit 302 to detect monitored objects with higher accuracy.

[0174] The training dataset also includes, for example, multiple images showing a monitored object with different positional relationships between the light source and the imaging range. The light source is, for example, the sun or nighttime lighting. The multiple images include images with positional relationships between the light source and the imaging range corresponding to front-lit, semi-front-lit, side-lit, backlit, etc. The multiple images also include, for example, images with the same front-lit, side-lit, backlit, etc. conditions but with different heights (elevation angles) of the sun from the ground. As a result, the training dataset is machine-trained to detect a monitored object from an input image, taking into account how the monitored object appears depending on the position of the light source. This allows the object detection unit 302 to detect a monitored object with higher accuracy.

[0175] [Example of object detection method] Next, a specific example of a method for detecting a monitored object by the object detection unit 302 will be described with reference to FIGS.

[0176] <Example 1> In this example, the object detection unit 302 detects (recognizes) high-visibility safety clothing from the input image, thereby detecting people (workers) around the shovel 100. This makes it easier for the object detection unit 302 to detect people even from an input image in which it is difficult to distinguish between the work clothing of people around the shovel 100 and the background, and as a result, it is possible to more accurately detect people as monitored objects.

[0177] Specifically, the object detection unit 302 may detect (recognize) a person wearing high visibility safety clothing from the input image, may detect (recognize) the high visibility safety clothing itself worn by a person, or may detect (recognize) both. In this way, when the object detection unit 302 detects a person wearing high visibility safety clothing or high visibility safety clothing (worn by a person), it can determine that it has detected a person as a monitored object.

[0178] For example, the object detection unit 302 detects a person around the shovel 100 by recognizing at least one of the shape, color (e.g., fluorescent yellow or fluorescent green), and luminance of the retroreflective material of the high-visibility safety suit. Specifically, the object detection unit 302 may recognize at least one of the shape, color, and luminance of the retroreflective material of the high-visibility safety suit by applying shape detection, pattern recognition, or the like to the shape, color, and luminance of the retroreflective material of the high-visibility safety suit. Furthermore, as described above, the object detection unit 302 may recognize at least one of the shape, color, and luminance of the retroreflective material of the high-visibility safety suit by using a trained model LM based on a teacher dataset that includes a large number of images of people (workers) wearing high-visibility safety suit.

[0179] Furthermore, in this example, the imaging device 40 (camera 40X) may acquire an image of the periphery of the shovel 100 while being irradiated with visible light from the irradiation device 70. In this case, the irradiation device 70 may constantly irradiate visible light while the camera 40X is operating (powered on) under the control of the controller 30 (irradiation control unit 305), or may irradiate visible light in synchronization with the timing of imaging by the camera 40X. As a result, when a person (worker) wearing high-visibility safety clothing is present within the imaging range of the imaging device 40, the retroreflective material and fluorescent color of the high-visibility safety clothing appear more clearly in the image captured by the camera 40X. This makes it easier for the object detection unit 302 to detect (recognize) the high-visibility safety clothing from the input image (the image captured by the camera 40X or the image corrected by the image correction unit 306), thereby enabling more accurate detection of the person as a monitored object.

[0180] Furthermore, in this example, the imaging device 40 (camera 40X) may acquire an image of the periphery of the shovel 100 while infrared light is being emitted from the irradiation device 70. In this case, the irradiation device 70 may continuously emit infrared light while the camera 40X is operating (powered on) under the control of the controller 30 (illumination control unit 305), or may intermittently (periodically) emit infrared light in synchronization with the imaging timing of the camera 40X. As a result, when a person (worker) wearing high-visibility safety clothing is present within the imaging range of the imaging device 40, the outline of the retroreflective material of the high-visibility safety clothing appears more clearly in the image captured by the camera 40X. This makes it easier for the object detection unit 302 to detect (recognize) the high-visibility safety clothing from the input image, thereby enabling more accurate detection of the person as a monitored object. Furthermore, because infrared light cannot be directly detected by the human eye as in the case of visible light, the impact of the shovel 100 on people around the shovel 100 can be reduced.

[0181] In this example, the irradiation device 70 may switch between a state in which it constantly (continuously) irradiates infrared light and a state in which it intermittently irradiates infrared light in synchronization with the image capture timing of the camera 40X under the control of the controller 30. For example, the irradiation control unit 305 determines whether or not the peripheral image displayed on the display device 50A or the display device 208 is affected by infrared light. The irradiation control unit 305 may then switch the irradiation state of infrared light from the irradiation device 70 based on the determination result. The influence of infrared light refers to, for example, the camera 40X capturing near-infrared light reflected from a subject (e.g., high-visibility safety clothing) and the peripheral image on the display device 50A appearing reddish. The irradiation control unit 305 may determine the presence or absence of the influence of infrared light by image analysis. Alternatively, the irradiation control unit 305 may determine the presence or absence of the influence of infrared light by receiving a predetermined input from a user (an operator or a supervisor) via the input device 52 or the communication device 60. As a result, when an operator or observer recognizes that the peripheral image of display device 50A or display device 208 is reddish, the operator or observer can notify controller 30 of the state by making a predetermined input through input device 52 or input device 207.

[0182] Specifically, the irradiation control unit 305 normally causes the irradiation device 70 to continuously irradiate infrared light, that is, in a state where the infrared light is not affecting the peripheral images displayed on the display device 50A or the display device 208. On the other hand, when the infrared light is affecting the peripheral images displayed on the display device 50A or the display device 208, the irradiation control unit 305 causes the infrared light to be intermittently irradiated in synchronization with the image capturing timing of the camera 40X. This makes it possible to achieve a balance between reducing the control load of the irradiation device 70 imposed by the controller 30 and reducing the influence of the infrared light on the peripheral images of the display device 50A, etc.

[0183] <Example 2> Fig. 16 is a diagram showing an example of an image captured by the imaging device 40 (camera 40X). Fig. 17 is a diagram showing an example of an image (corrected captured image) after correction by the image correction unit 306 for the imaging device 40 (camera 40X).

[0184] In this example, the object detection unit 302 detects a monitored object using an input image that is a corrected image (corrected captured image) obtained by the image correction unit 306 in response to an image captured by the image capture device 40.

[0185] For example, it is difficult for the operator in the cabin 10 to directly see what is behind the upper rotating body 3 and what is to the left and right. Also, for example, a remote-controlled operator cannot directly see the surroundings, including the front of the cabin 10. Therefore, it becomes necessary to cover a 360-degree range around the excavator 100 when viewed from above with the four cameras 40F, 40B, 40L, and 40R. As a result, for example, as shown in FIG. 2, the four cameras 40F, 40B, 40L, and 40R may be wide-angle cameras with a very large angle of view (field of view) in the left-right direction, and wide-angle distortion (volume anamorphosis) may occur in the captured image.

[0186] 1 and 2, the camera 40X is installed on the upper part of the upper rotating body 3. Therefore, the camera 40X needs to detect monitored objects, including those near the ground, from a relatively high position, and the optical axis of the camera 40X is set to point diagonally downward. As a result, perspective distortion may occur in the image captured by the camera 40X.

[0187] For example, as shown in FIG. 16, workers W1 to W3 are captured at the center, left end, and right end of the image captured by camera 40X. Worker W1, who is captured at the center of the image captured by camera 40X, is captured so that the vertical axis of the subject approximately coincides with the up-down axis of the captured image. The word "approximately" is intended to allow for errors in manufacturing the shovel 100 and errors in installing the camera 40X. Meanwhile, workers W2 and W3 in the image captured by camera 40X have their vertical axes tilted to the left and right, respectively, with respect to the up-down axis. This is because the image captured by camera 40X is distorted so that the vertical axes of the subjects at the left and right ends are tilted toward the ends with respect to the up-down axis.

[0188] In contrast to this, in this example, the image correction unit 306 corrects the image captured by the camera 40X so that the difference between the up-down axis of the image captured by the camera 40X and the vertical axis for the subject becomes smaller, and outputs the corrected captured image.

[0189] 17, the image correcting unit 306 generates a corrected captured image by tilting only the central portion and the image areas of both the left and right ends of the image captured by the camera 40X inward by a predetermined amount, so that the difference between the up-down axis of the image of the workers W2 and W3 and the vertical axis of the subject becomes relatively small in the corrected captured image.

[0190] Furthermore, the image correction unit 306 may correct the distortion of the image captured by the camera 40X using a known method such as projective transformation, thereby enabling the image correction unit 306 to reduce the difference between the up-down axis in the corrected captured image and the vertical axis for the subject.

[0191] In this example, the object detection unit 302 is configured to detect only a monitored object in which the difference between the up-down axis in the input image and the vertical axis of the subject is relatively small, and a monitored object in which the difference between the up-down axis in the input image and the vertical axis of the subject is relatively large. A monitored object in which the difference between the up-down axis in the input image and the vertical axis of the subject is relatively large is a monitored object that appears at both the left and right ends of the image captured by the camera 40X, such as workers W2 and W3 in FIG. 16. A monitored object in which the difference between the up-down axis in the input image and the vertical axis of the subject is relatively small is a monitored object that appears at both the left and right ends of the image captured by the camera 40X, such as worker W1 in FIG. 16. A monitored object in which the difference between the up-down axis in the input image and the vertical axis of the subject is relatively small is a monitored object that appears in the corrected image captured by the camera 40X, such as workers W1 to W3 in FIG. 17. This allows the object detection unit 302 to detect a monitored object using the corrected image as an input image.

[0192] Specifically, the object detection unit 302 may detect a monitoring object in which the difference between the up-down axis in the input image and the vertical axis of the subject is relatively small by applying shape detection, pattern recognition, or the like. Alternatively, the object detection unit 302 may use a trained model LM based on a training dataset that includes only images of monitoring objects in which the difference between the up-down axis in the input image and the vertical axis of the subject is relatively small, and images of monitoring objects in which relatively large monitoring objects are captured. This allows the object detection unit 302 to recognize only monitoring objects in which the difference between the up-down axis in the image and the vertical axis of the subject is relatively small from the input image (corrected captured image), thereby enabling more accurate detection of monitoring objects. Furthermore, when the trained model LM is used, it is only necessary to learn the features of only monitoring objects in which the difference between the up-down axis in the image and the vertical axis of the subject is relatively small from the image (corrected captured image), thereby improving the efficiency of machine learning.

[0193] Furthermore, when the trained model LM is used, as described above, the teacher dataset may include images of a monitored object captured by, for example, a camera of the same model as the camera 40X. Furthermore, as described above, the teacher dataset may include images of a monitored object captured by a camera of the same model as the camera 40X, which is installed in substantially the same position and with substantially the same attitude on a shovel of the same model as the shovel 100, as shown in, for example, FIGS. 8 to 13. In this case, the images captured by the same camera as the camera 40X may be corrected using the same function as the image correction unit 306, and the corrected images may be included in the teacher dataset. As a result, the trained model LM is machine-trained to detect a monitored object while taking into account how the monitored object appears in the image captured by the camera 40X and to detect only monitored objects for which the difference between the up-down axis in the image and the vertical axis relative to the subject is relatively small. Therefore, the object detection unit 302 can detect monitored objects with higher accuracy. In this case, the teacher dataset may include, for example, corrected images corresponding to both a captured image in which a monitored object is captured in the horizontal center and a captured image in which a monitored object is captured at at least one of the horizontal edges. This allows the trained model LM to be machine-trained to detect both a monitored object in the horizontal center of the input image (corrected captured image) without correction and a monitored object in the horizontal edge of the input image with correction. This allows the object detection unit 302 to detect monitored objects with higher accuracy.

[0194] <Example 3> 18 to 21 are diagrams showing first to fourth examples of captured images in which only a part of the entire (whole body) of a monitored object (person) is shown.

[0195] For example, as shown in FIGS. 18 to 21, the image captured by camera 40X may capture only a portion of the entire body (including the lower part of the body) of a person (worker W) as a monitored object located at both ends of the horizontal imaging range. Specifically, in FIG. 18 (first example), the lower and part of the upper body (waist) of worker W are captured at a position relatively close to the upper revolving structure 3 at the right end of the image captured by camera 40X. In addition, in FIG. 19 (second example), the lower and part of the upper body (chest, waist, arms, and hands excluding the head) of worker W are captured at a position relatively close to the upper revolving structure 3 at the left end of the image captured by camera 40X. In addition, in FIG. 20 (third example), part of the lower body (lower legs and feet) of worker W is captured at a position relatively close to the upper revolving structure 3 at the right end of the image captured by camera 40X. 21 (fourth example), part of the lower body (lower legs and feet) of worker W is captured at a position relatively distant from the upper revolving structure 3 at the left end of the image captured by camera 40X. This is because camera 40X has an optical axis that is directed diagonally downward from the top surface of the upper revolving structure 3 toward the ground, and the upper body of worker W (person) as a monitored object falls outside the three-dimensional capturing range of camera 40X.

[0196] Furthermore, if there is an obstacle between the monitored object and camera 40X, the lower part (lower body) of the monitored object (person) may be hidden by the obstacle, and the image captured by camera 40X may only show a part of the monitored object, including the upper part (upper body).

[0197] In contrast to this, in this example, the object detection unit 302 detects (recognizes) a monitored object (person) by detecting (recognizing) only a predetermined part of the entire monitored object (person) from the input image. The predetermined part is, for example, the lower part (lower body) or upper part (upper body) of the monitored object (person), as described above. This allows the object detection unit 302 to detect the monitored object even when only a part including a predetermined part of the entire monitored object is captured in the input image. Therefore, the object detection unit 302 can detect the monitored object with higher accuracy.

[0198] Specifically, the object detection unit 302 may detect the predetermined part of the monitored object from the input image by applying shape detection, pattern recognition, etc. Alternatively, the object detection unit 302 may detect the predetermined part of the monitored object using a trained model LM based on a teacher dataset including an image showing only a portion of the entire monitored object, including the predetermined part.

[0199] Furthermore, the object detection unit 302 may be configured to be able to detect (recognize) both the entire monitored object and a predetermined part of the entire monitored object. This allows the object detection unit 302 to determine that a monitored object has been detected when, for example, the object detection unit 302 detects either the entire monitored object or a predetermined part of the monitored object. This allows the object detection unit 302 to detect the monitored object with higher accuracy.

[0200] For example, the object detection unit 302 detects both the entire monitored object and the predetermined portion of the monitored object using a trained model LM based on a teacher dataset including images showing the entire monitored object and images showing only a portion, including a predetermined portion, of the entire monitored object. In this case, the object detection unit 302 may output a label corresponding to the detected monitored object from among multiple labels including a label representing the entire monitored object and a label representing the predetermined portion of the entire monitored object. Then, the object detection unit 302 may determine that the monitored object has been detected when the trained model LM outputs either a label representing the entire monitored object or a label representing the predetermined portion of the entire monitored object as the label of the detected monitored object.

[0201] Furthermore, when the trained model LM is used, the teacher dataset may include, for example, images captured by a camera of the same model as the camera 40X, in which only a portion of the entire monitored object, including the lower part, is captured on at least one of the left and right ends. Alternatively, the teacher dataset may include, for example, images captured by a camera of the same model as the camera 40X, installed in approximately the same position and with approximately the same attitude on an excavator of the same model as the shovel 100, in which only a portion of the entire monitored object, including the lower part, is captured on at least one of the left and right ends. This allows the trained model LM to be machine-trained to detect the lower part of the entire monitored object (e.g., the lower half of a person's body) taking into account how the monitored object is captured in the image captured by the camera 40X. This allows the object detection unit 302 to more accurately detect a predetermined portion of the entire monitored object.

[0202] Alternatively, the object detection unit 302 may detect the entire monitored object based on an image at the center in the horizontal direction of the input image, and detect only the lower part of the entire monitored object (for example, the lower half of a person's body) based on images at both ends in the horizontal direction of the input image. This is because, as described above, there may be cases where only the lower part of the entire monitored object (for example, the lower half of a person's body) is captured at both ends in the horizontal direction of the input image.

[0203] The object detection unit 302 may also detect a monitored object in a region where the image capture ranges of two cameras 40X partially overlap, based on the captured images of the two cameras. The two cameras 40X partially overlapping each other's image capture ranges may correspond to, for example, any one of the combinations of cameras 40F and 40R, cameras 40R and 40B, cameras 40B and 40L, and cameras 40L and 40F. Thus, even if the object detection unit 302 erroneously detects a monitored object in a region where the image capture ranges of the two cameras 40X partially overlap based on one camera 40X, the object detection unit 302 can determine that the monitored object has not been detected because the other camera 40X did not detect the monitored object. Therefore, the object detection unit 302 can suppress erroneous detections when detecting a monitored object in a region where the image capture ranges of the two cameras 40X partially overlap.

[0204] [Example of a method for estimating the actual position of a monitored object] Next, a specific example of a method for estimating the actual position of a detected monitored object will be described with reference to FIGS.

[0205] 22 to 24 are diagrams showing an example, another example, and yet another example of a state of detection of a monitored object (person) by the object detection unit 302. Specifically, Fig. 22 to Fig. 24 are diagrams showing a detection frame FR representing the detection range of a monitored object (worker W) in an input image when the monitored object (worker W) is detected from the input image by the object detection unit 302.

[0206] 22 to 24 may be displayed on the display device 50A or the remote control display device, for example, when a monitored object (worker W) is detected by the object detection unit 302. This allows the operator or supervisor to understand the detection status of the monitored object (worker W) around the shovel 100 while viewing the peripheral image (image captured by the camera 40X).

[0207] 22 and 23, the worker W is standing upright and facing the camera 40X in the same location relatively close to the upper rotating structure 3. In the situation in Fig. 24, the worker W is standing upright and facing the camera 40X in a location relatively far from the upper rotating structure 3.

[0208] 22, the object detection unit 302 detects (recognizes) the entire worker W shown in the input image (e.g., the image captured by the camera 40X) as a monitored object (person). Therefore, the detection frame FR is expressed as a rectangle that surrounds the worker W.

[0209] 23, the object detection unit 302 detects (recognizes) only the upper part (upper body) of the entire worker W shown in the input image as a monitored object (person). Therefore, the detection frame FR is expressed as a rectangle that surrounds only the upper part of the worker W, including the head where the helmet is worn and the torso where the high-visibility safety clothing is worn.

[0210] For example, when detecting (recognizing) a person by detecting (recognizing) an item worn by the person, such as a helmet or high-visibility safety clothing, the object detection unit 302 may determine only a partial area in the input image that corresponds to the item worn by the person as the detection range of the monitored object. Furthermore, depending on the background of the monitored object and the weather at the time, the features of the monitored object in the input image, such as its shape, may become unclear, and the object detection unit 302 may only be able to recognize a portion of the entire monitored object as the monitored object. Therefore, the object detection unit 302 may detect (recognize) only a portion of the entire monitored object in the vertical direction in the input image.

[0211] For example, in the situation of Fig. 23, if the actual position of the worker W is estimated based on the size of the monitored object detected (recognized) on the input image (the size of the detection frame FR in the vertical direction) as described above, it may be determined that the worker is present at a position farther from the upper rotating structure 3 than the actual position. Specifically, even though the worker W is present at the same position as in the situation of Fig. 22, in the situation of Fig. 23, the actual position estimated may be farther from the upper rotating structure 3 than the actual position estimated in the situation of Fig. 22.

[0212] 23 and 24, the distance between the worker W and the upper revolving structure 3 is different, but the detected position of the person (worker W) on the input image, specifically, the lower end position of the detection range (detection frame FR), is substantially the same. Therefore, as described above, when the actual position of the monitored object is estimated by projective transformation or the like based on the detected position (reference point) of the monitored object on the input image, there is a possibility that the actual position that is relatively distant from the upper revolving structure 3 will be estimated, similar to the worker W detected in the situation of FIG. 24.

[0213] In contrast, in this example, the position estimation unit 303 estimates the actual position of the monitored object based on information about the vertical detection position of the monitored object on the input image and information about the size and shape of the monitored object's detection range on the input image. The information about the vertical detection position of the monitored object on the input image is a coordinate position that represents the monitored object's detection range on the input image, and includes, for example, at least one of the coordinates of the lower end, upper end, and centroid of the monitored object's detection range on the input image. The information about the size of the monitored object's detection range on the input image includes, for example, information about the height (vertical dimension) of the detection range, information about the width (horizontal dimension), information about the aspect ratio, information about the diagonal, information about the area, etc. Furthermore, the information about the shape of the monitored object's detection range on the input image includes, for example, information about the aspect ratio, etc.

[0214] Specifically, the position estimation unit 303 may determine whether or not the shape and size of the detection range of the monitored object on the input image are consistent with the vertical detection position of the monitored object on the input image. Assuming that the entire monitored object is detected in the input image, the range of the shape and size of the expected detection range of the monitored object is determined by the vertical detection position of the monitored object on the input image. Therefore, the position estimation unit 303 can determine whether or not the entire monitored object is detected based on whether or not the shape and size of the detection range of the monitored object on the input image are consistent with the vertical detection position of the monitored object on the input image, i.e., whether they fall within this expected range.

[0215] The expected range of the shape and size of the detection range of the monitored object in the input image may be uniformly defined depending on the detected vertical position of the monitored object in the input image. Furthermore, the expected range of the shape and size of the detection range of the monitored object in the input image may take into account other conditions in addition to the detected vertical position of the monitored object in the input image. For example, the expected range of the shape and size of the detection range of the monitored object in the input image may vary depending on the posture and orientation of the detected monitored object in addition to the detected vertical position of the monitored object in the input image. This is because the appearance of the monitored object in the input image differs depending on the posture and orientation of the monitored object, and as a result, the shape and size of the detection range of the monitored object in the input image change. The orientation and orientation of the monitored object can be acquired, for example, by machine learning the trained model LM so that it can detect the monitored object for each different orientation and orientation of the monitored object, i.e., so that it can detect the monitored object for each of multiple labels corresponding to different orientations and orientations. Furthermore, the expected range of the shape and size of the detection range on the input image (captured image) of the monitored object may be determined, for example, not only by taking into account the vertical detection position on the input image (captured image) of the monitored object, but also by taking into account the distortion caused by the lens at that detection position.

[0216] The position estimation unit 303 determines that the entire monitored object has been detected if the shape and size of the monitored object's detection range on the input image are consistent with the detected position of the monitored object in the vertical direction on the input image. In this case, the position estimation unit 303 may estimate the actual position of the monitored object as described above, based on the detected position and detection range of the monitored object on the input image.

[0217] On the other hand, if there is no consistency between the shape or size of the detection range of the monitored object on the input image and the vertical detected position of the monitored object on the input image, the position estimation unit 303 determines that only a part of the entire monitored object has been detected. In this case, the position estimation unit 303 may correct the detected position or detection range of the monitored object on the input image, and estimate the actual position of the monitored object based on the corrected detected position or detection range.

[0218] For example, if the width of the detection range of the monitored object on the input image deviates in a large direction from the range expected for the vertical detection position of the monitored object on the input image, the position estimation unit 303 determines that only a portion of the upper side of the entire monitored object on the input image has been detected (see FIG. 23).

[0219] Furthermore, for example, if the aspect ratio of the detection range of the monitored object in the input image deviates from the range expected for the vertical detection position of the monitored object in the input image in the direction in which the horizontal dimension increases, the position estimation unit 303 determines that only a portion of the entire monitored object in the vertical direction has been detected in the input image. Furthermore, if the object detection unit 302 detects only a predetermined portion of the entire monitored object or only an item worn by the monitored object, the position estimation unit 303 may determine whether only a portion of the entire monitored object in the vertical direction has been detected, either an upper portion or a lower portion, in the vertical direction, of the entire monitored object in the input image, taking into account the predetermined portion or the location where the item is worn. For example, if the object detection unit 302 detects (recognizes) a helmet or high-visibility safety clothing, the position estimation unit 303 determines that only an upper portion of the entire monitored object (person) in the input image, including the location where the helmet or high-visibility safety clothing is worn, has been detected (see FIG. 23). Furthermore, when the position estimation unit 303 determines that only a portion of the entire vertical direction of the monitored object in the input image has been detected, it may uniformly assume that only a portion of the upper side of the entire vertical direction of the monitored object has been detected, in order to prioritize safety.

[0220] In this case, the position estimation unit 303 may, for example, correct the monitored object detection range on the input image by extending it downward based on information about the height and width of the monitored object detection range on the input image so that the monitored object has the expected shape (aspect ratio). The expected shape (aspect ratio) of the monitored object is determined, for example, according to the detected position of the monitored object on the input image. The expected shape (aspect ratio) of the monitored object may also be determined, for example, according to the orientation and posture of the monitored object, distortion due to the lens at the detected position, and the like, in addition to the detected position of the monitored object on the input image. Furthermore, the position estimation unit 303 may, for example, correct the monitored object detection position by moving it downward or extending the monitored object detection range downward based on information about the width of the monitored object detection range on the input image so that the width of the monitored object matches the width of the monitored object detection range on the input image. In this case, the position estimation unit 303 may correct the monitored object detection position and detection range by taking into account, for example, the orientation and posture of the monitored object, distortion due to the lens at the detected position, and the like. Then, the position estimation unit 303 may estimate the actual position of the monitored object based on the corrected detection range and detected position of the monitored object.

[0221] As described above, in this example, the position estimation unit 303 is able to grasp a state in which the object detection unit 302 detects only a portion of the entire monitored object on the input image (particularly, an upper portion in the up-down direction), and estimate the actual position of the monitored object by taking this state into consideration. Therefore, the position estimation unit 303 can, for example, prevent a situation in which the actual position of the monitored object is estimated to be a position relatively farther from the upper rotating body 3 than it actually is, and can more appropriately estimate the actual position of the monitored object. Therefore, the controller 30 (safety control unit 304) can more appropriately activate the safety function, and as a result, the safety of the excavator 100 can be improved.

[0222] [Another example of excavator function configuration] Next, another example of the functional configuration of the shovel 100 will be described with reference to Fig. 25. The following description will focus on parts that differ from the above example (Fig. 6), and explanations that are the same as or correspond to the above example may be omitted.

[0223] FIG. 25 is a functional block diagram showing another example of the functional configuration of the shovel 100 according to this embodiment.

[0224] 25 , similar to the example described above, the controller 30 includes a display processing unit 301, an object detection unit 302, a position estimation unit 303, a safety control unit 304, and an illumination control unit 305, and may also include an image correction unit 306. Unlike the example described above, the controller 30 also includes a detection determination unit 307.

[0225] The imaging device 40 includes an object detection unit 402. The imaging device 40 may also include an image correction unit 404.

[0226] Similar to object detection unit 302, object detection unit 402 (an example of a detection unit) detects a monitored object based on an image captured by imaging device 40. Object detection unit 402 includes object detection units 402F, 402B, 402L, and 402R. Hereinafter, object detection units 402F, 402B, 402L, and 402R may be collectively referred to as "object detection unit 402X," or any one of them may be individually referred to as "object detection unit 402X."

[0227] The object detection unit 402F is mounted on the camera 40F and, similar to the object detection unit 302, detects a monitored object based on an image captured by the camera 40F. Specifically, the object detection unit 402F may recognize a monitored object from an image captured by the camera 40F and identify the position (area) in which the monitored object is captured. The object detection unit 402F may also recognize a monitored object from an image (corrected captured image) obtained after the image correction unit 404F corrects the image captured by the camera 40F (i.e., the captured image generated by the captured image generation function) and identify the position (area) in which the monitored object is captured. The function of the object detection unit 402F is realized by any hardware or a combination of any hardware and software. For example, the function of the object detection unit 402F is realized by, for example, loading a program installed in an auxiliary storage device of the image processing engine 42 of the camera 40F into a memory device and executing it on the CPU. The same applies to the functions of the object detection units 402B, 402L, and 402R.

[0228] The object detection unit 402B is mounted on the camera 40B, and similarly to the object detection unit 302, detects a monitored object based on an image captured by the camera 40B. Specifically, the object detection unit 402B may recognize a monitored object from an image captured by the camera 40B and identify the position (area) in which the monitored object is captured. The object detection unit 402B may also recognize a monitored object from an image (corrected captured image) obtained after the image captured by the camera 40B (i.e., the captured image generated by the captured image generation function) has been corrected by the image correction unit 404B, and identify the position (area) in which the monitored object is captured.

[0229] The object detection unit 402L is mounted on the camera 40L, and similar to the object detection unit 302, detects a monitored object based on an image captured by the camera 40L. Specifically, the object detection unit 402L may recognize a monitored object from an image captured by the camera 40L and identify a position (area) in which the monitored object is captured. The object detection unit 402L may also recognize a monitored object from an image (corrected captured image) obtained after the image captured by the camera 40L (i.e., the captured image generated by the captured image generation function) has been corrected by the image correction unit 404L, and identify a position (area) in which the monitored object is captured.

[0230] The object detection unit 402R is mounted on the camera 40R, and similar to the object detection unit 302, detects a monitored object based on an image captured by the camera 40R. Specifically, the object detection unit 402R may recognize a monitored object from an image captured by the camera 40R and identify a position (area) in which the monitored object is captured. The object detection unit 402R may also recognize a monitored object from an image (corrected captured image) obtained after the image captured by the camera 40R (i.e., the captured image generated by the captured image generation function) has been corrected by the image correction unit 404R, and identify a position (area) in which the monitored object is captured.

[0231] Similar to image correction unit 306, image correction unit 404 (an example of a correction unit) performs predetermined correction on the output (captured image) of imaging device 40 (camera 40X) and outputs the corrected captured image to object detection unit 402. Image correction units 404F, 404B, 404L, and 404R are included.

[0232] The image correction unit 404F is mounted on the camera 40F, and similarly to the image correction unit 306 described above, performs predetermined corrections on the image captured by the camera 40F, and outputs the corrected captured image to the object detection unit 402F.

[0233] In addition to the captured image generated by the captured image generation function, the camera 40F may output a corrected captured image corrected by the image correction unit 404F to the controller 30. The same may be true for the cameras 40B, 40L, and 40R. In this case, the image correction unit 306 of the controller 30 may be omitted, and the object detection unit 302 may detect a monitored object based on the corrected captured image input from the camera 40X.

[0234] Image correction section 404B is mounted on camera 40B, and similar to image correction section 306 described above, performs predetermined corrections on the image captured by camera 40B, and outputs the corrected captured image to object detection section 402B.

[0235] The image correction unit 404L is mounted on the camera 40L, and similarly to the image correction unit 306 described above, performs predetermined corrections on the image captured by the camera 40L, and outputs the corrected captured image to the object detection unit 402L.

[0236] The image correction unit 404R is mounted on the camera 40R, and similar to the image correction unit 306 described above, performs predetermined corrections on the image captured by the camera 40R, and outputs the corrected captured image to the object detection unit 402R.

[0237] The object detection units 402X use the same algorithm to detect a monitored object from an input image. On the other hand, the object detection unit 402X uses an algorithm different from that of the object detection unit 302 to detect a monitored object from an input image. This increases the likelihood that, for example, even if one of the object detection units 302, 402X cannot detect a monitored object, the other will detect the monitored object. As a result, it is possible to prevent a situation in which a monitored object in the vicinity of the excavator 100 cannot be detected. Furthermore, for example, even if one of the object detection units 302, 402X detects a monitored object that does not exist, it is also possible to prevent the other from detecting the monitored object because it has not detected the monitored object.

[0238] For example, one of the object detection units 302, 402X may use only the former of image processing technology and machine learning technology, and the other may use both image processing technology and machine learning technology, i.e., a learned model LM, to detect a monitored object.

[0239] Furthermore, for example, one of the object detection units 302, 402X may use an SVM that has machine-learned the image features of the monitored object as a trained model LM, and the other may use a trained model LM based on deep learning to detect the monitored object from the input image.

[0240] Furthermore, the object detection units 302 and 402X may detect a monitored object from an input image using, for example, trained models LM based on different networks (DNNs). For example, one of the object detection units 302 and 402X may use a CNN-based classification model as the trained model LM, and the other may use a CNN-based regression model as the trained model LM to detect a monitored object from an input image. Specifically, one of the object detection units 302 and 402X may use a trained model LM based on Faster R-CNN, and the other may use a trained model LM based on YOLO or SSD to detect a monitored object from an input image. Furthermore, for example, the object detection units 302 and 402X may use a CNN-based trained model LM, and the trained models LM may differ from each other in at least one of the network structure, the number of network layers, the feature map structure, and the characteristics of the candidate region. The characteristics of the candidate region may include differences such as the aspect ratio and whether the candidate region is fixed or variable. Specifically, one of the object detection units 302 and 402X may use a trained model LM based on YOLO, and the other may use a trained model LM based on SSD to detect a monitored object from an input image. Furthermore, for example, the object detection units 302 and 402X may use the same trained model LM based on SSD, and the trained models LM may differ from each other in at least one parameter, such as the network structure, the number of network layers, the structure of the feature map, or the characteristics of the candidate region.

[0241] Furthermore, for example, the object detection units 302 and 402X may detect a monitoring object from an input image using a trained model LM that has been machine-trained using different teacher data sets.

[0242] For example, the object detection units 302 and 402X may detect a monitored object from an input image using a trained model LM that has been machine-trained using a training dataset based on images in which at least one of the posture and orientation of the monitored object differs from the other. For example, one of the trained models LM of the object detection units 302 and 402X may be machine-trained using a training dataset based on images depicting a person standing upright, while the other may be machine-trained using a training dataset based on images depicting a person crouching or squatting. Furthermore, for example, one of the trained models of the object detection units 302 and 402X may be machine-trained using a training dataset based on images depicting a person facing forward or backward toward the camera, while the other may be machine-trained using a training dataset based on images depicting a person facing sideways toward the camera. This allows the object detection units 302 and 402X to increase the likelihood that the other will properly detect the monitored object from the input image, even when one of the units is unable to properly detect the monitored object due to the orientation or posture of the monitored object in the input image. Therefore, the object detection units 302 and 402X can detect the monitored object more reliably overall.

[0243] Furthermore, for example, one of the object detection units 302 and 402X may detect (recognize) the entire monitored object (person) including its shape, while the other may detect (recognize) an item worn by the monitored object (person), thereby detecting a monitored object from an input image. For example, one of the object detection units 302 and 402X may detect the entire monitored person from the input image, while the other may detect (recognize) a helmet or safety vest worn by the person, thereby detecting a monitored object (person) from an input image. In this way, even if, for some reason, one of the object detection units 302 and 402X cannot properly detect (recognize) the entire monitored person, the other can more reliably detect the monitored object (person).

[0244] Furthermore, for example, the object detection units 302 and 402X may each detect a different item worn by the monitored object (person) to detect the monitored object from the input image. For example, one of the object detection units 302 and 402X may detect (recognize) a helmet worn by the person from the input image, and the other may detect (recognize) a safety vest worn by the person from the input image, thereby detecting the person from the input image. In this way, even if one of the object detection units 302 and 402X cannot recognize one item worn by the person because, for example, the worker is not wearing the one item, the other object detection unit 302 and 402X can more reliably detect the monitored object (person) because the other object detection unit recognizes the other item worn.

[0245] Furthermore, for example, the object detection units 302 and 402X may each detect a monitored object from an input image using a trained model LM that has been machine-trained using a training dataset containing images with different types of backgrounds for the monitored object. For example, one of the trained models LM for the object detection units 302 and 402X may be machine-trained using a training dataset containing images with a sandy ground in the background, while the other may be machine-trained using a training dataset containing images with a ground such as asphalt in the background. This increases the likelihood that one of the object detection units 302 and 402X will properly detect the monitored object from the input image, even in a situation where the other unit is unable to properly detect the monitored object due to the background of the input image. As a result, the object detection units 302 and 402X can more reliably detect monitored objects overall.

[0246] Furthermore, for example, the object detection units 302, 402X may each detect a monitored object from an input image using a trained model LM that has been machine-learned using a teacher data set based on images captured (imaged) in different time periods. For example, one of the trained models of the object detection units 302, 402X may be machine-learned using a teacher data set based on images captured in the morning or afternoon, and the other may be machine-learned using a teacher data set based on images captured in the evening or night. This allows the controller 30 to selectively use the object detection units 302, 402X depending on, for example, the time period during which the excavator 100 is working, or to switch the method of making a final decision regarding the detection of a monitored object based on the detection results of the object detection units 302, 402X.

[0247] Furthermore, when the shovel 100 is used at night, for example, the object detection units 302 and 402X may each detect a monitored object from an input image using a trained model LM that has been machine-trained using a teacher data set of images captured under different nighttime lighting conditions. For example, one of the trained models LM of the object detection units 302 and 402X may be machine-trained using a teacher data set of images captured under relatively high illuminance, while the other may be machine-trained using a teacher data set of images captured under relatively low illuminance. Furthermore, for example, the trained models LM of the object detection units 302 and 402X may be machine-trained using teacher data sets of images captured under lighting conditions of different colors. This increases the likelihood that one of the object detection units 302 and 402X will properly detect the monitored object even in a situation where the other unit cannot properly detect the monitored object due to lighting conditions. Therefore, the object detection units 302 and 402X can more reliably detect monitored objects overall.

[0248] Furthermore, for example, the object detection units 302 and 402X may detect a monitored object from an input image using a trained model LM that has been machine-trained using a training dataset of images acquired (captured) under different weather conditions. For example, the trained model LM of one of the object detection units 302 and 402X may be machine-trained using a training dataset of images acquired under sunny or cloudy conditions, and the other may be machine-trained using a training dataset of images acquired under rainy or snowy conditions. This increases the likelihood that one of the object detection units 302 and 402X will properly detect the monitored object even in a situation where one of the object detection units 302 and 402X cannot properly detect the monitored object due to, for example, weather conditions. Therefore, the object detection units 302 and 402X can more reliably detect monitored objects overall.

[0249] Furthermore, for example, the object detection units 302 and 402X may detect a monitored object from an input image using a trained model LM that has been machine-trained using a training dataset of images acquired under different positional relationships between the light source and the imaging range. For example, one of the trained models LM of the object detection units 302 and 402X may be machine-trained using a training dataset of images corresponding to front lighting, partial front lighting, or side lighting, and the other may be machine-trained using a training dataset of images corresponding to back lighting. This increases the likelihood that one of the object detection units 302 and 402X will properly detect the monitored object even in a situation where one of the units cannot properly detect the monitored object due to the influence of the positional relationship between the light source and the imaging range. Therefore, the object detection units 302 and 402X can more reliably detect monitored objects overall.

[0250] The detection and determination unit 307 detects a monitored object based on the detection result of the object detection unit 302 and the object detection unit 402X. Specifically, the detection and determination unit 307 makes a final determination regarding the detection of a monitored object based on the detection result of the object detection unit 302 and the detection result of the object detection unit 402X. The method of detecting a monitored object by the detection and determination unit 307 will be described in detail later.

[0251] The position estimation unit 303 estimates the actual position of the monitored object when the detection and determination unit 307 detects the monitored object. When the detection and determination unit 307 detects multiple monitored objects, the position estimation unit 303 estimates the actual position of each of the multiple monitored objects.

[0252] For example, if a monitored object is detected by either of the object detection units 302 and 402X, the position of the monitored object is estimated by arbitrarily applying the above-mentioned method based on the detection range on the input image specified by either of the object detection units.

[0253] Furthermore, for example, when the same monitored object is detected by both the object detection units 302 and 402X, the actual position of the monitored object may be estimated for each of the detection results of the object detection units 302 and 402X, and the actual position of the monitored object may be estimated from each of the estimation results. For example, the actual position of the detected monitored object may be estimated by averaging the estimation results of the actual position of the monitored object for each of the detection results of the object detection units 302 and 402X.

[0254] The safety control unit 304 activates the safety function when, for example, the detection determination unit 307 detects a monitored object within a predetermined range around the shovel 100. Specifically, the safety control unit 304 may activate the safety function when the actual position (estimated value) of the monitored object estimated by the position estimation unit 303 is within a predetermined range around the shovel 100.

[0255] The function of either one of the object detection units 302, 402 may be transferred to the display device 50A. Furthermore, in addition to the object detection units 302, 402, one or more object detection units that detect a monitored object from an input image may be provided in the shovel 100. The additional object detection unit may be provided in the display device 50A, in the controller 30, or in a controller different from the controller 30. In this case, the three or more object detection units including the additional ones may detect a monitored object from an input image using algorithms different from each other.

[0256] [Example of object detection method] Next, a method for detecting a monitored object by the detection and determination unit 307, that is, a method for making a final determination regarding the detection of a monitored object, will be described.

[0257] For example, when a monitored object is detected by at least one of the object detection units 302 and 402X, the detection determination unit 307 may determine that the monitored object is present in the vicinity of the shovel 100 and detect the monitored object. As a result, even if an actually existing monitored object is not detected by one of the object detection units 302, 402X, the controller 30 can detect the monitored object as long as the monitored object is detected by the other one. Therefore, the controller 30 can more reliably detect an actually existing monitored object, for example, in a situation where a failure to detect the monitored object is likely to occur or a situation where there is a high need to prevent a failure to detect the monitored object, and as a result, safety can be further improved with the activation of the safety functions of the shovel 100.

[0258] Furthermore, for example, when a monitored object is detected by both the object detection unit 302 and the object detection unit 402X, the detection determination unit 307 may determine that the monitored object is present in the vicinity of the shovel 100 and detect the monitored object. This allows the controller 30 to, for example, prevent a monitored object from being detected even if a non-existent monitored object is erroneously detected by one of the object detection units 302, 402X unless the other detects the monitored object. Therefore, the controller 30 can suppress erroneous detection of a monitored object in, for example, a situation in which erroneous detection of a monitored object is likely to occur or a situation in which it is highly necessary to prevent a decrease in work efficiency due to erroneous detection of a monitored object.

[0259] As described above, when three or more object detection units are provided in the shovel 100, the detection determination unit 307 may detect a monitored object when, for example, a predetermined number (an integer equal to or greater than two) of all object detection units have detected the monitored object. The predetermined number may be fixed or variable. For example, the predetermined number may be variable according to a predetermined input from a user (operator or supervisor) received via the input device 52 or the communication device 60.

[0260] Furthermore, the detection and determination unit 307 may switch between the above two methods in response to a predetermined input from a user (operator or observer) received via the input device 52 or the communication device 60.

[0261] Furthermore, the detection and determination unit 307 may detect a monitored object based on the object detection unit 302 and the object detection unit 402X, for example, by using the importance defined for the detection results of the object detection unit 302 and the object detection unit 402X. Specifically, when a monitored object is detected by one of the object detection unit 302 and the object detection unit 402X whose detection result has a relatively higher importance, the detection and determination unit 307 detects the monitored object.

[0262] The importance may be set (variable) according to a predetermined input from a user (operator or supervisor) received through, for example, the input device 52 or the communication device 60. This allows the user to compare, for example, a peripheral image displayed on the display device 50A or the remote control display device with the actual detection status of the monitored object by the object detection units 302, 402X, and set a higher importance to one detection result that is considered to be more accurate. The importance may also be automatically set (variable) according to the situation around the shovel 100. For example, if there is a difference in the detection accuracy of the monitored object between the object detection units 302, 402X depending on the environmental condition in which the shovel 100 is placed, the importance of one detection result with higher detection accuracy may be set higher than the importance of the other detection result. The environmental condition in which the shovel 100 is placed may include the condition of objects around the shovel 100 that appear as a background in the image captured by the camera 40X (for example, the type of earth and sand on the ground, asphalt, etc.). Furthermore, the environmental conditions in which the shovel 100 is placed may include, for example, the time period in which work is being performed, the lighting conditions during night work, weather conditions, the positional relationship between the light source and the imaging range of the camera 40X, etc. This allows the controller 30 to set the importance of either of the object detection units 302, 402X that is expected to have higher detection accuracy relatively high, for example, in accordance with the situation in which the shovel 100 is placed.

[0263] As described above, when the shovel 100 is provided with three or more object detection units, the detection and determination unit 307 may detect a monitored object such that the relatively higher the importance of the detection result of each object detection unit, the more likely that detection result will be reflected in the result of the final determination. For example, when a monitored object has been detected by some of all object detection units, the detection and determination unit 307 may detect the monitored object if the sum of the importance of the detection results of those some object detection units is greater than the sum of the importance of the detection results of the remaining object detection units. Furthermore, when a monitored object has been detected by some of all object detection units, the detection and determination unit 307 may detect the monitored object if the sum of the importance of the detection results of those some object detection units is equal to or greater than a predetermined threshold.

[0264] [Transformation / Change] Although the embodiments have been described in detail above, the present disclosure is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist described in the claims.

[0265] For example, in the above-described embodiment, a method for detecting a monitored object based on an image captured by the imaging device 40 mounted on the shovel 100 has been described, but a similar method may be applied to a method for detecting a monitored object based on an image captured by an imaging device mounted on another work machine. Examples of other work machines include a lifting magnet machine in which a lifting magnet is attached to the tip of the attachment AT instead of the bucket 6, a bulldozer, a forestry machine (e.g., a harvester), a road machine (e.g., an asphalt finisher), etc. [Explanation of symbols]

[0266] 1 Undercarriage 1C Crawler 1ML Travel Hydraulic Motor 1MR travel hydraulic motor 2. Swivel mechanism 2M Swing Hydraulic Motor 3 Upper rotating body 4. Boom 5 Arm 6 buckets 7 Boom cylinder 8 Arm Cylinder 9 Bucket cylinder 10 Cabins 11 Engine 13 Regulator 14 Main pump 15 Pilot pump 17 Control valve 25 Pilot Line 25A pilot line 25B Pilot Line 26 Operating device 27 Pilot Line 27A Pilot Line 27B Pilot Line 29 Operating pressure sensor 30 Controllers 30A auxiliary storage 30B Memory Device 30C CPU 30D Interface Device 31 Hydraulic control valve 32 Shuttle valve 33 Hydraulic control valve 40 Imaging device 40B Camera 40F Camera 40L camera 40R Camera 50 Output Device 50A display device 50B Sound output device 52 Input Device 60 Communication Equipment 70 Irradiation device 100 Shovel 200 Management device 201 External Interface 201A Recording Media 202 Auxiliary storage 203 Memory Device 204 CPU 205 High-speed calculation device 206 Communication Interface 207 Input Device 208 Display device 301 Display processing unit 302 Object detection unit (detection unit) 303 Position estimation part 304 Safety control section (control section) 305 Irradiation control unit 306 Image correction unit (correction unit) 307 Detection and Judgment Unit 402, 402B, 402F, 402L, 402R Object detection unit (detection unit) 404, 404B, 404F, 404L, 404R Image correction unit (correction unit) AT Attachment HA Hydraulic Actuator HMT Helmet LM pre-trained model Network communication line RV High Visibility Safety Clothing S1 Boom Angle Sensor S2 Arm Angle Sensor S3 Bucket Angle Sensor S4 Aircraft attitude sensor S5 Rotation Angle Sensor SYS Excavator Management System W, W1 to W3 workers

Claims

1. a lower running body; an upper rotating body rotatably mounted on the lower traveling body; an imaging device mounted on the upper rotating body and configured to capture an image of the periphery of the upper rotating body; a detection unit that detects a predetermined object from the input image by recognizing only the former of the predetermined objects in which the difference between the up-down axis of the input image and the vertical axis of the subject is relatively small and the former in which the difference between the up-down axis of the input image and the vertical axis of the subject is relatively large; a correction unit that corrects the captured image so that a difference between an up-down axis of the captured image of the imaging device and a vertical axis of a subject appearing in the captured image becomes small, the correction unit corrects the captured image in a manner that does not correct a first image region in the captured image, in which a difference between an up-down axis of the captured image and a vertical axis for the subject is relatively small, and a second image region in which a difference between an up-down axis of the captured image and a vertical axis for the subject is relatively large, but tilts only the second image region as a whole in a direction in which the difference between the up-down axis of the captured image and the vertical axis for the subject becomes smaller; the detection unit detects the predetermined object from the image corrected by the correction unit as the input image. Shovel.

2. the detection unit detects the predetermined object from the image corrected by the correction unit using a trained model that has been machine-learned using training data that includes only one of an image in which the predetermined object is captured and in which the difference between the up-down axis of the image and the vertical axis of the captured predetermined object is relatively small, and an image in which the predetermined object is captured and in which the difference between the up-down axis of the image and the vertical axis of the captured predetermined object is relatively large; The shovel according to claim 1.

3. The former image included in the training data is an image generated by correcting an image captured by an imaging device of the same type as the imaging device, the image capturing the predetermined object, using the same function as the correction unit. The shovel according to claim 2.

4. The former image is at least one of an image captured by an imaging device of the same type installed at approximately the same position as the imaging device of the same type of shovel and in which the specified object is captured at the left end of the image, and an image captured by an imaging device of the same type and in which the specified object is captured at the right end of the image, and an image generated by correcting an image captured by an imaging device of the same type and in which the specified object is captured in the center in the left-right direction of the image using the same function as the correction unit. The shovel according to claim 3.

5. and a control unit that slows down or stops the operation of the shovel when the detection unit detects the predetermined object within a predetermined range of the shovel. A shovel according to any one of claims 1 to 4.

Images