Shovel

Abstract

To provide a technique for further improving the accuracy of detecting an object around a shovel.SOLUTION: A shovel 100 includes an undercarriage 1, a super structure 3 turnably mounted on the undercarriage 1, an image pick-up device 40 mounted on the super structure 3 for picking up an image around the super structure 3, and object detection parts 302, 402 using mutually different algorithms for detecting a predetermined object (an object to be monitored) around the super structure 3 on the basis of the image picked up by the image pick-up device 40.SELECTED DRAWING: Figure 25

Description

Technical Field

[0001] The present disclosure relates to an excavator.

Background Art

[0002] Conventionally, based on an image of an imaging device mounted on an upper revolving body, a technique for detecting a predetermined object (for example, a person) around an excavator (upper revolving body) and activating a safety function (for example, output of an alarm) is known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, there is a possibility that a non-existent object is detected and the safety function is erroneously activated, or a real object is not detected and the necessary safety function is not activated. Therefore, it is desirable to further improve the accuracy of object detection around the excavator.

[0005] Therefore, in view of the above problems, an object is to provide a technology capable of further improving the accuracy of object detection around an excavator.

Means for Solving the Problems

[0006] To achieve the above object, in one embodiment of the present disclosure, a lower traveling body, an upper revolving body rotatably mounted on the lower traveling body, an imaging device mounted on the upper revolving body and imaging the periphery of the upper revolving body, Each unit comprises a plurality of detection units that detect predetermined objects around the upper rotating body using different algorithms based on the captured image from the imaging device, In the algorithms of one of the plurality of detection units and the other detection units, the predetermined object is detected using machine learning models that include different neural networks and are based on training data images in which the predetermined object is depicted. death , The aforementioned distinct neural networks are neural networks with different structures. A shovel will be provided. In other embodiments of this disclosure, Lower running body and An upper slewing body is mounted on the lower traveling body so as to be rotatable, An imaging device mounted on the upper rotating body for imaging the area around the upper rotating body, Each of the following detection units detects a predetermined object around the shovel using a different algorithm based on the image captured by the imaging device: The system includes a final detection unit that detects a predetermined object in the vicinity of the shovel based on the detection results of the plurality of detection units, The importance level is defined for each of the detection results from the aforementioned plurality of detection units. The final detection unit detects the predetermined object when, in the event that the predetermined object has been detected by some of the plurality of detection units, the sum of the importance values ​​of the detection results of the some detection units is greater than the sum of the importance values ​​of the detection results of the remaining detection units, or when the sum of the importance values ​​of the some detection units is equal to or greater than a predetermined threshold. A shovel will be provided. Furthermore, in yet another embodiment of this disclosure, Lower running body and An upper slewing body is mounted on the lower traveling body so as to be rotatable, An imaging device mounted on the upper rotating body for imaging the area around the upper rotating body, Each of the following detection units detects a predetermined object around the shovel using a different algorithm based on the image captured by the imaging device: The system includes a final detection unit that detects a predetermined object in the vicinity of the shovel based on the detection results of the plurality of detection units, In the algorithms of one of the plurality of detection units and the other detection units, the predetermined object is detected using machine learning models that include different neural networks and are based on training data images in which the predetermined object is depicted. A shovel will be provided.

Advantages of the Invention

[0007] According to the above embodiment, the accuracy of object detection around the excavator can be further improved.

Brief Description of the Drawings

[0008] [Figure 1] It is a side view showing an example of an excavator. [Figure 2] It is a top view showing an example of an excavator. [Figure 3] It is a diagram showing an example of an excavator management system. [Figure 4] It is a diagram showing an example of the hardware configuration of an excavator. [Figure 5] It is a diagram showing an example of the hardware configuration of a management device. [Figure 6] It is a functional block diagram showing an example of the functional configuration of an excavator. [Figure 7] It is a diagram for explaining a specific example of an object detection method. [Figure 8] It is a diagram showing a first example of an image of teaching data. [Figure 9] It is a diagram showing a second example of an image of teaching data. [Figure 10] It is a diagram showing a third example of an image of teaching data. [Figure 11] It is a diagram showing a fourth example of an image of teaching data. [Figure 12] It is a diagram showing a fifth example of an image of teaching data. [Figure 13] It is a diagram showing a sixth example of an image of teaching data. [Figure 14] It is a diagram showing an example of an operator. [Figure 15] It is a diagram showing another example of an operator. [Figure 16] It is a diagram showing an example of a captured image of an imaging device. [Figure 17] It is a diagram showing an example of an image after correction (corrected captured image) for the captured image of the imaging device. [Figure 18]This figure shows the first example of an image captured where only a portion of the entire body of the monitored object (person) is visible. [Figure 19] This figure shows a second example of an image captured where only a portion of the entire body of the monitored object (person) is visible. [Figure 20] This figure shows a third example of an image captured where only a portion of the entire body of the monitored object (person) is visible. [Figure 21] This figure shows the fourth example of an image captured where only a portion of the entire body of the monitored object (person) is visible. [Figure 22] This figure shows an example of the detection status of a monitored object (person). [Figure 23] This figure shows another example of the detection status of a monitored object (person). [Figure 24] This figure shows yet another example of the detection status of a monitored object (person). [Figure 25] This is a functional block diagram showing another example of the functional configuration of an excavator. [Modes for carrying out the invention]

[0009] The embodiments will be described below with reference to the drawings.

[0010] [Shovel Overview] Refer to Figures 1 to 3 to explain the basic structure of the excavator.

[0011] Figures 1 and 2 are a side view and a top view showing an example of the excavator 100 according to this embodiment. Figure 3 is a diagram showing an example of the excavator management system SYS.

[0012] As shown in Figures 1 and 2, the shovel 100 comprises a lower traveling body 1, an upper rotating body 3 mounted on the lower traveling body 1 so as to be rotatable via a slewing mechanism 2, an attachment AT for performing various tasks, and a cabin 10. Hereinafter, the front of the shovel 100 (upper rotating body 3) corresponds to the direction in which the attachment to the upper rotating body 3 extends when the shovel 100 is viewed from directly above in a plan view (top view) along the rotation axis of the upper rotating body 3.

[0013] The lower travel unit 1 includes, for example, a pair of left and right crawlers 1C. The lower travel unit 1 moves the shovel 100 by hydraulically driving each crawler 1C with a left-side travel hydraulic motor 1ML and a right-side travel hydraulic motor 1MR.

[0014] The upper rotating body 3 rotates relative to the lower traveling body 1 when the rotating mechanism 2 is hydraulically driven by the rotating hydraulic motor 2M.

[0015] The attachment AT includes a boom 4, an arm 5, and a bucket 6 as driven elements.

[0016] The boom 4 is mounted to the front center of the upper slewing body 3 so as to be able to be tilted up and down, the arm 5 is mounted to the tip of the boom 4 so as to be able to rotate up and down, and the bucket 6 is mounted to the tip of the arm 5 so as to be able to rotate up and down.

[0017] Bucket 6 is an example of an end attachment. Bucket 6 is used, for example, in excavation work.

[0018] Furthermore, depending on the work to be performed, other end attachments may be attached to the tip of the arm 5 instead of the bucket 6. Other end attachments may be other types of buckets, such as large buckets, slope buckets, or dredging buckets. Other end attachments may also be types of end attachments other than buckets, such as agitators, breakers, or grapples. In addition, spare attachments such as quick couplings or tilt rotators may be interposed between the arm 5 and the end attachment.

[0019] Furthermore, a hook for crane operations may be attached to the bucket 6. The base end of the hook is rotatably connected to a bucket pin that connects the arm 5 and the bucket 6. As a result, when operations other than crane operations (lifting operations), such as excavation work, are performed, the hook is stored in the space formed between the two bucket links.

[0020] The boom 4, arm 5, and bucket 6 are hydraulically driven by the boom cylinder 7, arm cylinder 8, and bucket cylinder 9, respectively, which act as hydraulic actuators.

[0021] Furthermore, some or all of the hydraulic actuators in the shovel 100 may be replaced with electric actuators. In other words, the shovel 100 may be a hybrid shovel or an electric shovel.

[0022] Cabin 10 is the cockpit where the operator sits, and is mounted, for example, on the front left side of the upper rotating body 3.

[0023] Furthermore, if the operator does not board the shovel 100 to operate it, and the shovel 100 operates solely by remote control or fully automated driving functions, as described later, the cabin 10 may be omitted.

[0024] Furthermore, as shown in Figure 3, the shovel 100 may be included in the shovel management system SYS together with the control device 200, and may be able to communicate with the control device 200 via a predetermined communication line NW. This allows the shovel 100 to transmit (upload) various information to the control device 200 and receive various signals (e.g., information signals and control signals) from the control device 200.

[0025] The communication line NW may include, for example, a wide area network (WAN). The wide area network may include, for example, a mobile communication network with base stations as its endpoints. The wide area network may also include, for example, a satellite communication network utilizing communication satellites orbiting above the shovel 100. The wide area network may also include, for example, the Internet network. The communication line NW may also include, for example, a local area network (LAN) of a facility where the management device 200 is installed. The local network may be a wireless line, a wired line, or a line that includes both. The communication line NW may also include, for example, a short-range communication line based on a predetermined wireless communication method such as WiFi or Bluetooth®.

[0026] The shovel management system SYS, within the management device 200, performs management and provides support related to the shovel 100.

[0027] For example, the shovel management system SYS may manage (monitor) the operating status and operational status of the shovel 100 based on various information uploaded from the shovel 100 in the management device 200. Furthermore, as described later, the shovel management system SYS may support the remote operation of the shovel 100 in the management device 200. Also, as described later, when the shovel 100 performs work by fully automated operation, the shovel management system SYS may, for example, support the remote monitoring of the shovel 100's work by fully automated operation in the management device 200. In this case, the shovel management system SYS may support the operation of the shovel 100 by intervention of a supervisor in relation to the fully automated operation function in the management device 200.

[0028] The shovel management system SYS may include one shovel 100 or multiple shovels. Similarly, the shovel management system SYS may include one management device 200 or multiple management devices. In other words, multiple management devices 200 may distribute the processing related to the shovel management system SYS. For example, each of the multiple management devices 200 may communicate with a subset of the shovels 100 it is responsible for among all the shovels 100 included in the shovel management system SYS and execute processing targeting that subset of shovels 100.

[0029] The management device 200 may be, for example, an on-premise server or cloud server installed in a management center outside the work site where the excavator 100 is operating. Alternatively, the management device 200 may be an edge server located within the work site where the excavator 100 is operating, or in a location relatively close to the work site. Furthermore, the management device 200 may be a stationary terminal device or a portable terminal device (mobile terminal) located in a management office or similar location within the work site of the excavator 100. A stationary terminal device may include, for example, a desktop PC (Personal Computer). A portable terminal device may include, for example, a smartphone, tablet device, or laptop PC.

[0030] The excavator 100 may operate actuators (e.g., hydraulic actuators) in response to operations by an operator seated in the cabin 10, thereby driving driven elements such as the lower traveling body 1, the upper rotating body 3, and the attachment AT.

[0031] Furthermore, the shovel 100 may be configured to be remotely controlled from a predetermined external device (for example, a control device 200). When the shovel 100 is remotely controlled, the interior of the cabin 10 may be unoccupied. The following explanation will proceed on the premise that operator operation includes both operation of the control device 26 by the operator in the cabin 10 and remote operation by an external operator.

[0032] Specifically, the shovel 100 is operated by user (operator) input regarding the actuator of the shovel 100, which is performed by the control device 200. In this case, the shovel 100 transmits data of images representing the surroundings of the shovel 100 (hereinafter referred to as "surrounding images") to an external device, such as images captured by the imaging device 40 described later and processed images based on those captured images (e.g., viewpoint transformation images). The surrounding images are then displayed on a remote control display device (hereinafter referred to as "remote control display device") provided in the control device 200. In addition, various information images (information screens) displayed on the display device 50A (see Figure 6) inside the cabin 10 of the shovel 100 may also be displayed on the remote control display device of the control device 200. This allows the operator to remotely operate the shovel 100 while checking the displayed content, such as the surrounding images representing the surroundings of the shovel 100 and various information images, displayed on the remote control display device. The excavator 100 can then operate actuators in response to remote control signals received from the control device 200, which represent the content of the remote control operation, thereby driving the driven elements such as the lower traveling body 1, the upper rotating body 3, and the attachment AT.

[0033] Furthermore, the excavator 100 may operate its actuators automatically, regardless of the operator's actions. This allows the excavator 100 to automatically operate at least some of its driven elements, such as the lower traveling body 1, the upper rotating body 3, and the attachment AT, thus realizing what is known as an "automatic driving function" or "machine control (MC) function."

[0034] The automatic driving function may include a function that automatically operates driven elements (actuators) other than the target driven element (actuator) in response to operator input, i.e., a so-called "semi-automatic driving function" or "operation-assist type MC function". The automatic driving function may also include a function that automatically operates at least some of multiple driven elements (hydraulic actuators) without operator input, i.e., a so-called "fully automatic driving function" or "fully automatic MC function". In the case of the excavator 100, if the fully automatic driving function is enabled, the interior of the cabin 10 may be unoccupied. Furthermore, the semi-automatic driving function and fully automatic driving function may include a mode in which the operation content of the driven elements (actuators) subject to automatic driving is automatically determined according to predetermined rules. Furthermore, the semi-automatic driving function and fully automatic driving function may also include a so-called "autonomous driving function" in which the excavator 100 autonomously makes various judgments, and the operation content of the driven elements (hydraulic actuators) subject to automatic driving is autonomously determined according to the results of those judgments.

[0035] [Excavator hardware configuration] Next, with reference to Figure 4, the hardware configuration of the shovel 100 will be described.

[0036] Figure 4 shows an example of the hardware configuration of the Shovel 100.

[0037] In Figure 4, the paths through which mechanical power is transmitted are shown by double lines, the paths through which high-pressure hydraulic fluid that drives the hydraulic actuators are flowed are shown by solid lines, the paths through which pilot pressure is transmitted are shown by dashed lines, and the paths through which electrical signals are transmitted are shown by dotted lines.

[0038] The shovel 100 includes various components such as a hydraulic drive system for hydraulically driving the driven element, an operating system for operating the driven element, a user interface system for exchanging information with the user, a communication system for communication with the outside, and a control system for various types of control.

[0039] <Hydraulic drive system> As shown in Figure 4, the hydraulic drive system of the excavator 100 according to this embodiment includes hydraulic actuators HA that hydraulically drive each of the driven elements, such as the lower traveling body 1 (left and right crawlers 1C), the upper rotating body 3, and the attachment AT, as described above. The hydraulic drive system of the excavator 100 according to this embodiment also includes an engine 11, a regulator 13, a main pump 14, and a control valve 17.

[0040] As shown in Figure 6, the hydraulic actuator HA includes travel hydraulic motors 1ML and 1MR, slewing hydraulic motor 2M, boom cylinder 7, arm cylinder 8, and bucket cylinder 9, etc.

[0041] Engine 11 is the prime mover for the shovel 100 and the main power source in the hydraulic drive system. Engine 11 is, for example, a diesel engine that uses light oil as fuel. Engine 11 is mounted, for example, at the rear of the upper rotating body 3. Under direct or indirect control by the controller 30, which will be described later, Engine 11 rotates at a constant speed at a preset target speed and drives the main pump 14 and the pilot pump 15.

[0042] Furthermore, in place of or in addition to engine 11, another prime mover (for example, an electric motor) may be mounted on the shovel 100.

[0043] The regulator 13 controls (adjusts) the discharge rate of the main pump 14 under the control of the controller 30. For example, the regulator 13 adjusts the angle of the swash plate of the main pump 14 (hereinafter referred to as the "tilt angle") in response to a control command from the controller 30.

[0044] The main pump 14 supplies hydraulic fluid to the control valve 17 through a high-pressure hydraulic line. The main pump 14 is mounted, for example, at the rear of the upper slewing body 3, similar to the engine 11. The main pump 14 is driven by the engine 11, as described above. The main pump 14 is, for example, a variable displacement hydraulic pump, and as described above, under the control of the controller 30, the piston stroke length is adjusted by adjusting the tilt angle of the swash plate by the regulator 13, thereby controlling the discharge flow rate (discharge pressure).

[0045] The control valve 17 is a hydraulic control device that controls the hydraulic actuator HA in accordance with the operator's operation of the operating device 26, the content of remote operation, or operation commands related to the automatic operation function output from the controller 30. The operation commands corresponding to the automatic operation function may be generated by the controller 30 or by another control device (computer) that controls the automatic operation function. The control valve 17 is mounted, for example, in the center of the upper slewing body 3. As described above, the control valve 17 is connected to the main pump 14 via a high-pressure hydraulic line and selectively supplies hydraulic fluid supplied from the main pump 14 to each hydraulic actuator in accordance with the operator's operation or operation commands corresponding to the automatic operation function. Specifically, the control valve 17 includes a plurality of control valves (also called "direction control valves") that control the flow rate and direction of the hydraulic fluid supplied from the main pump 14 to each of the hydraulic actuators HA.

[0046] <Operation system> As shown in Figure 4, the operating system of the shovel 100 according to this embodiment includes a pilot pump 15, an operating device 26, a hydraulic control valve 31, a shuttle valve 32, and a hydraulic control valve 33.

[0047] The pilot pump 15 supplies pilot pressure to various hydraulic devices via the pilot line 25. The pilot pump 15 is mounted, for example, at the rear of the upper slewing body 3, similar to the engine 11. The pilot pump 15 is, for example, a fixed-displacement hydraulic pump and is driven by the engine 11 as described above.

[0048] The pilot pump 15 may be omitted. In this case, the relatively high-pressure hydraulic fluid discharged from the main pump 14 is reduced in pressure by a predetermined pressure reducing valve, and the resulting relatively low-pressure hydraulic fluid is supplied to various hydraulic devices as pilot pressure.

[0049] The control device 26 is located near the cockpit of the cabin 10 and is used by the operator to operate various driven elements. In other words, the control device 26 is used by the operator to operate the hydraulic actuators HA that drive each driven element. The control device 26 includes pedal devices and lever devices for operating each driven element (hydraulic actuator HA).

[0050] For example, as shown in Figure 4, the operating device 26 is hydraulically pilot operated. Specifically, the operating device 26 uses hydraulic fluid supplied from the pilot pump 15 through the pilot line 25 and the pilot line 25A branched therefrom, and outputs a pilot pressure corresponding to the operation to the secondary pilot line 27A. The pilot line 27A is connected to one inlet port of the shuttle valve 32 and, via the pilot line 27 connected to the outlet port of the shuttle valve 32, is connected to the control valve 17. As a result, the control valve 17 can receive a pilot pressure via the shuttle valve 32 corresponding to the operation of various driven elements (hydraulic actuators) in the operating device 26. Therefore, the control valve 17 can drive each hydraulic actuator HA according to the operation performed on the operating device 26 by an operator or the like.

[0051] The operating device 26 may also be electrically operated. In this case, the pilot line 27A, shuttle valve 32, and hydraulic control valve 33 are omitted. Specifically, the operating device 26 outputs an electrical signal (hereinafter referred to as "operating signal") corresponding to the operation content, and the operating signal is received by the controller 30. The controller 30 then outputs a control command corresponding to the content of the operating signal, that is, a control signal corresponding to the operation content of the operating device 26, to the hydraulic control valve 31. As a result, a pilot pressure corresponding to the operation content of the operating device 26 is input from the hydraulic control valve 31 to the control valve 17, and the control valve 17 can drive each hydraulic actuator HA according to the operation content of the operating device 26. In addition, the control valve (directional control valve) that drives each hydraulic actuator, which is built into the control valve 17, may also be of the electromagnetic solenoid type. In this case, the operating signal output from the operating device 26 may be directly input to the control valve 17, that is, to the electromagnetic solenoid type control valve. Furthermore, the operating device 26 may be omitted when the shovel 100 is remotely operated or when it is operated by an automatic operation function.

[0052] A hydraulic control valve 31 is provided for each driven element (hydraulic actuator HA) that is the target of the operating device 26. The hydraulic control valve 31 is provided, for example, in the pilot line 25B between the pilot pump 15 and the control valve 17, and may be configured to change its flow area (i.e., the cross-sectional area through which hydraulic fluid can flow). This allows the hydraulic control valve 31 to output a predetermined pilot pressure to the secondary pilot line 27B using the hydraulic fluid from the pilot pump 15 supplied through the pilot line 25B. Therefore, as shown in Figure 4, the hydraulic control valve 31 can indirectly apply a predetermined pilot pressure to the control valve 17 in accordance with the control signal from the controller 30 through the shuttle valve 32 between the pilot line 27B and the pilot line 27. Alternatively, as shown in Figure 5, the hydraulic control valve 31 can directly apply a predetermined pilot pressure to the control valve 17 in accordance with the control signal from the controller 30 through the pilot line 27B and the pilot line 27. Therefore, the controller 30 supplies pilot pressure from the hydraulic control valve 31 to the control valve 17 according to the operation of the electric operating device 26, thereby enabling the operation of the shovel 100 based on the operator's input.

[0053] Furthermore, the controller 30 may, for example, control the hydraulic control valve 31 to realize an automatic operation function. Specifically, the controller 30 outputs a control signal to the hydraulic control valve 31 corresponding to an operation command related to the automatic operation function, regardless of whether the operating device 26 is operated or not. As a result, the controller 30 causes the hydraulic control valve 31 to supply pilot pressure corresponding to the operation command related to the automatic operation function to the control valve 17, thereby realizing the operation of the shovel 100 based on the automatic operation function.

[0054] Furthermore, the controller 30 may, for example, control the hydraulic control valve 31 to enable remote operation of the shovel 100. Specifically, the controller 30 outputs a control signal to the hydraulic control valve 31 via the communication device 60 that corresponds to the content of the remote operation specified by the remote operation signal received from the management device 200. As a result, the controller 30 causes the hydraulic control valve 31 to supply pilot pressure corresponding to the content of the remote operation to the control valve 17, thereby enabling operation of the shovel 100 based on the operator's remote operation.

[0055] The shuttle valve 32 has two inlet ports and one outlet port, and outputs hydraulic fluid with the higher of the two pilot pressures input to the two inlet ports to the outlet port. A shuttle valve 32 is provided for each driven element (hydraulic actuator HA) that is operated by the operating device 26. Furthermore, a shuttle valve 32 is provided for each direction of movement of the driven element (for example, the raising direction and the lowering direction of the boom 4). One of the two inlet ports of the shuttle valve 32 is connected to the pilot line 27A on the secondary side of the operating device 26 (specifically, the lever device or pedal device included in the operating device 26), and the other is connected to the pilot line 27B on the secondary side of the hydraulic control valve 31. The outlet port of the shuttle valve 32 is connected to the pilot port of the corresponding control valve of the control valve 17 via the pilot line 27. The corresponding control valve is the control valve that drives the hydraulic actuator that is operated by the lever device or pedal device connected to one of the inlet ports of the shuttle valve 32. Therefore, each of these shuttle valves 32 can apply the higher of the pilot pressure of the pilot line 27A on the secondary side of the operating device 26 and the pilot pressure of the pilot line 27B on the secondary side of the hydraulic control valve 31 to the pilot port of the corresponding control valve. In other words, the controller 30 can control the corresponding control valve regardless of the operator's operation of the operating device 26 by outputting a pilot pressure from the hydraulic control valve 31 that is higher than the pilot pressure on the secondary side of the operating device 26. Thus, the controller 30 can control the operation of the driven elements (lower traveling body 1, upper rotating body 3, attachment AT) regardless of the operator's operation of the operating device 26, and realize an automatic driving function.

[0056] The hydraulic control valve 33 is provided in the pilot line 27A connecting the operating device 26 and the shuttle valve 32. The hydraulic control valve 33 is configured, for example, to allow the flow path area to be changed. The hydraulic control valve 33 operates in response to a control signal input from the controller 30. This allows the controller 30 to forcibly reduce the pilot pressure output from the operating device 26 when the operating device 26 is operated by an operator. Therefore, even when the operating device 26 is being operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator corresponding to the operation of the operating device 26. In addition, the controller 30 can reduce the pilot pressure output from the operating device 26, for example, even when the operating device 26 is being operated, to a level lower than the pilot pressure output from the hydraulic control valve 31. Therefore, by controlling the hydraulic control valve 31 and the hydraulic control valve 33, the controller 30 can reliably apply a desired pilot pressure to the pilot port of the control valve in the control valve 17, for example, regardless of the operation of the operating device 26. Therefore, the controller 30 can more effectively realize the automatic operation function and remote control function of the excavator 100 by controlling, for example, the hydraulic control valve 33 in addition to the hydraulic control valve 31.

[0057] <User Interface System> As shown in Figure 4, the user interface system of the shovel 100 according to this embodiment includes an operating device 26, an output device 50, and an input device 52.

[0058] The output device 50 outputs various information to the users of the shovel 100 (for example, the operator of the cabin 10 or work vehicles around the shovel 100).

[0059] For example, the output device 50 includes lighting equipment and display devices 50A (see Figure 6) that output various information in a visual manner. Lighting equipment is, for example, a warning light. Display devices 50A are, for example, liquid crystal displays or organic EL (electroluminescence) displays. The lighting equipment and display devices 50A may be installed, for example, inside the cabin 10 and output various information in a visual manner to operators, etc., inside the cabin 10. Alternatively, the lighting equipment and display devices 50A may be installed, for example, on the side of the upper rotating body 3 and output various information in a visual manner to workers, etc., around the shovel 100.

[0060] Furthermore, for example, the output device 50 includes a sound output device 50B (see Figure 6) that outputs various information in an audible manner. The sound output device 50B includes, for example, a buzzer or a speaker. The sound output device 50B may be provided, for example, inside and outside the cabin 10, and may output various information in an audible manner to the operator inside the cabin 10 or to people (workers, etc.) around the shovel 100.

[0061] Furthermore, for example, the output device 50 may include a device that outputs various types of information through tactile means such as vibrations in the cockpit.

[0062] The input device 52 receives various inputs from the user of the shovel 100, and the signals corresponding to the received inputs are taken into the controller 30. The input device 52 is installed, for example, inside the cabin 10 and receives inputs from operators, etc., inside the cabin 10. Alternatively, the input device 52 may be installed, for example, on the side of the upper rotating body 3 and receive inputs from workers, etc., around the shovel 100.

[0063] For example, the input device 52 includes an operation input device that accepts operation input. The operation input device may include a touch panel mounted on the display device, a touch pad installed around the display device, a button switch, a lever, a toggle, a knob switch provided on the operation device 26 (lever device), etc.

[0064] Furthermore, for example, the input device 52 may include a voice input device that accepts voice input from the user. The voice input device may include, for example, a microphone.

[0065] Furthermore, for example, the input device 52 may include a gesture input device that receives gesture input from the user. The gesture input device may include, for example, an imaging device that captures images of the gestures performed by the user.

[0066] Furthermore, for example, the input device 52 may include a biometric input device that accepts biometric input from the user. Biometric input may include, for example, the input of biometric information such as the user's fingerprints or iris.

[0067] <Communications System> As shown in Figure 4, the communication system of the shovel 100 according to this embodiment includes a communication device 60.

[0068] The communication device 60 connects to a communication line NW and communicates with a device (for example, a management device 200) that is provided separately from the shovel 100. The device provided separately from the shovel 100 may include not only devices located outside the shovel 100, but also portable terminal devices brought into the cabin 10 by the user of the shovel 100. The communication device 60 uses, for example, 4G (4 th Generation) and 5G (5 th The communication device 60 may include a mobile communication module that conforms to standards such as Generation. Furthermore, the communication device 60 may include, for example, a satellite communication module. Additionally, the communication device 60 may include, for example, a WiFi communication module or a Bluetooth® communication module.

[0069] <Control System> As shown in Figure 4, the control system of the excavator 100 according to this embodiment includes a controller 30. The control system of the excavator 100 according to this embodiment also includes an operating pressure sensor 29, an imaging device 40, an illumination device 70, a boom angle sensor S1, an arm angle sensor S2, a bucket angle sensor S3, a machine attitude sensor S4, and a slewing angle sensor S5.

[0070] The controller 30 (an example of a control device) performs various controls related to the shovel 100.

[0071] The functions of the controller 30 may be realized by any hardware, or any combination of hardware and software. For example, as shown in Figure 4, the controller 30 includes an auxiliary storage device 30A, a memory device 30B, a CPU (Central Processing Unit) 30C, and an interface device 30D, all connected by bus B1.

[0072] The auxiliary storage device 30A is a non-volatile storage means that stores the installed program as well as necessary files and data. The auxiliary storage device 30A is, for example, flash memory.

[0073] The memory device 30B loads the program from the auxiliary storage device 30A into the CPU 30C's readable state, for example, when a program startup command is received. The memory device 30B is, for example, SRAM (Static Random Access Memory).

[0074] The CPU 30C, for example, executes a program loaded into the memory device 30B and implements various functions of the controller 30 according to the program's instructions.

[0075] The interface device 30D is used, for example, as an interface for connecting to a communication line inside the shovel 100. The interface device 30D may include multiple different types of interface devices depending on the type of communication line to be connected.

[0076] The program that implements the various functions of the controller 30 is provided, for example, by a portable recording medium. In this case, the interface device 30D functions as an interface for reading data from and writing data to the recording medium. The recording medium is, for example, a dedicated tool connected by a detachable cable to a connector installed inside the cabin 10. Alternatively, the recording medium may be a general-purpose recording medium such as an SD memory card or a USB (Universal Serial Bus) memory. The program may also be downloaded from another computer outside the excavator 100 (for example, a management device 200) via a predetermined communication line and installed in the auxiliary storage device 30A.

[0077] Furthermore, some of the functions of controller 30 may be implemented by other controllers (control devices). That is, the functions of controller 30 may be implemented in a distributed manner by multiple controllers. For example, the functions related to image processing of images captured by the imaging device 40, the functions related to detecting objects around the shovel 100, and the functions related to ensuring the safety of the shovel 100 may be implemented by different controllers. The functions related to image processing of images captured by the imaging device 40 include, for example, the display processing unit 301 and the image correction unit 306 described later. The functions related to detecting objects around the shovel 100 include, for example, the object detection unit 302 and the detection judgment unit 307 described later. The functions related to ensuring the safety of the shovel 100 include, for example, the functions of the safety control unit 304 described later. In addition, the functions related to detecting objects around the shovel 100 may be implemented in a distributed manner between the controller that implements the functions related to image processing of images captured by the imaging device 40 and the controller that implements the functions related to ensuring the safety of the shovel 100. Specifically, among the functions related to detecting objects around the shovel 100, functions that are highly related to image processing, such as the function of recognizing objects from the image captured by the imaging device 40 (function of the object detection unit 302), may be installed in the former controller. On the other hand, among the functions related to detecting objects around the shovel 100, functions that are highly related to ensuring the safety of the shovel 100, such as the function of finally determining whether or not a recognized object has been detected (function of the detection determination unit 307), may be installed in the latter controller.

[0078] The operating pressure sensor 29 detects the pilot pressure on the secondary side (pilot line 27A) of the hydraulic pilot-operated operating device 26, that is, the pilot pressure corresponding to the operating state of each driven element (hydraulic actuator) in the operating device 26. The detection signal of the pilot pressure corresponding to the operating state of each driven element (hydraulic actuator HA) in the operating device 26, detected by the operating pressure sensor 29, is received by the controller 30.

[0079] Furthermore, if the operating device 26 is electrically operated, the operating pressure sensor 29 is omitted. This is because the controller 30 can understand the operating state of each driven element through the operating device 26 based on the operating signals received from the operating device 26.

[0080] The imaging device 40 acquires images around the shovel 100 to interpolate blind spots and areas that are difficult to see from the operator's perspective. The output (captured image) of the imaging device 40 is taken in by the controller 30.

[0081] The imaging device 40 may be, for example, a monocular camera, a stereo camera, or a depth camera. The imaging device 40 may also acquire three-dimensional data (for example, point cloud data or surface data) representing the position and outline of objects around the shovel 100 within a predetermined imaging range (angle of view) based on the captured image.

[0082] For example, as shown in Figures 1 and 2, the imaging device 40 includes a camera 40F that images the front of the upper rotating body 3, a camera 40B that images the rear of the upper rotating body 3, a camera 40L that images the left side of the upper rotating body 3, and a camera 40R that images the right side of the upper rotating body 3. This allows the operator to view the images captured by cameras 40B, 40L, and 40R, as well as surrounding images such as processed images generated based on those images, through the display device 50A and the remote control display device, and to confirm the state of the left, right, and rear of the upper rotating body 3. In addition, the operator can remotely control the shovel 100 while confirming the operation of the attachment AT, including the bucket 6, by viewing the images captured by camera 40F and surrounding images such as processed images generated based on those images through the remote control display device. Hereinafter, cameras 40F, 40B, 40L, and 40R may be collectively or individually referred to as "camera 40X".

[0083] Each imaging device 40 (camera 40X) includes an image sensor 41 and an image processing engine 42. Based on the output (electrical signal) of the image sensor 41, the camera 40X generates an image using the image processing engine 42 and outputs the generated image (image). The image processing engine 42 is a computer dedicated to image processing, including, for example, a CPU, a memory device, and an auxiliary storage device, and realizes various image processing by executing a program installed on the auxiliary storage device on the CPU. Hereinafter, the function of the image processing engine 42 that generates an image will be referred to as the image generation function, and the image of the imaging device 40 means the image generated by the image generation function.

[0084] The dashed lines in Figure 2 represent the top-view angle (imaging range) of cameras 40F, 40B, 40L, and 40R.

[0085] The illumination device 70, under the control of the controller 30, illuminates the imaging range of the imaging device 40 with a predetermined light. The predetermined light is, for example, visible light. Alternatively, the predetermined light may be, for example, infrared light. There may be one illumination device 70 or multiple illumination devices, as long as they can illuminate the entire imaging range of the imaging device 40. For example, an illumination device 70 is provided for each camera 40X and is installed on the upper rotating body 3 so as to be close to the camera 40X.

[0086] The irradiation device 70 may be omitted.

[0087] The boom angle sensor S1 acquires detection information regarding the attitude angle of the boom 4 (hereinafter referred to as "boom angle") with respect to a predetermined reference (for example, the horizontal plane or the state of either end of the movable angle range of the boom 4). The boom angle sensor S1 may include, for example, a rotary encoder, an acceleration sensor, an angular velocity sensor, a six-axis sensor, an IMU (Inertial Measurement Unit), etc. The boom angle sensor S1 may also include a cylinder sensor capable of detecting the extension and retraction position of the boom cylinder 7.

[0088] The arm angle sensor S2 acquires detection information regarding the attitude angle of the arm 5 (hereinafter referred to as "arm angle") relative to a predetermined reference (for example, a straight line connecting the connection points at both ends of the boom 4, or the state of either end of the movable angle range of the arm 5). The arm angle sensor S2 may include, for example, a rotary encoder, an acceleration sensor, an angular velocity sensor, a six-axis sensor, an IMU, etc. The arm angle sensor S2 may also include a cylinder sensor capable of detecting the extension and retraction position of the arm cylinder 8.

[0089] The bucket angle sensor S3 acquires detection information regarding the attitude angle of the bucket 6 (hereinafter referred to as "bucket angle") with respect to a predetermined reference (for example, a straight line connecting the connection points at both ends of the arm 5, or the state of either end of the movable angle range of the bucket 6). The bucket angle sensor S3 may include, for example, a rotary encoder, an acceleration sensor, an angular velocity sensor, a six-axis sensor, an IMU, etc. The bucket angle sensor S3 may also include a cylinder sensor capable of detecting the extension and retraction position of the bucket cylinder 9.

[0090] The machine attitude sensor S4 acquires detection information regarding the attitude state of the excavator 100, including the lower traveling body 1 and the upper rotating body 3. The machine attitude sensor S4 is mounted on the upper rotating body 3, for example, and acquires detection information regarding the tilt angle of the upper rotating body 3 with respect to the horizontal plane and the attitude angle around the rotation axis (i.e., the orientation of the upper rotating body 3 with respect to the ground). The machine attitude sensor S4 may include, for example, an acceleration sensor (tilt sensor), an angular velocity sensor, a six-axis sensor, an IMU, etc.

[0091] The rotation angle sensor S5 acquires detection information regarding the rotation angle of the upper rotating body 3 relative to the lower traveling body 1 (i.e., the orientation of the upper rotating body 3). The rotation angle sensor S5 includes, for example, a potentiometer, a rotary encoder, a resolver, etc.

[0092] Furthermore, for example, the shovel 100 may include a positioning device capable of determining its absolute position. The positioning device is, for example, a GNSS (Global Navigation Satellite System) sensor. This can improve the accuracy of estimating the attitude state of the shovel 100.

[0093] Furthermore, for example, the shovel 100 may include, in addition to the imaging device 40, a distance sensor for detecting the distance to objects around the shovel 100. The distance sensor may include, for example, a LiDAR (Light Detecting and Ranging), millimeter-wave radar, ultrasonic sensor, infrared sensor, or distance image sensor. This allows the controller 30 to detect objects around the shovel 100 using, for example, the output of the distance sensor in addition to the output of the imaging device 40.

[0094] Furthermore, some or all of the boom angle sensor S1, arm angle sensor S2, bucket angle sensor S3, machine attitude sensor S4, and slewing angle sensor S5 may be omitted. For example, if remote control or automatic driving functions are not employed, it may not be necessary to estimate the attitude state of the attachment AT or the machine (upper slewing body 3) of the shovel 100. Also, for example, it may be possible to estimate the attitude state of the shovel 100 from information about the shovel 100's surroundings acquired by the imaging device 40 or the distance sensors described later. Specifically, the information about the shovel 100's surroundings acquired by the imaging device 40 or the distance sensors described later may include information about the position and shape of surrounding objects and attachments as seen from the machine (upper slewing body 3). In this case, the controller 30 can estimate the attitude state of the attachment AT or the machine (upper slewing body 3) from that information, depending on the required accuracy.

[0095] [Hardware configuration of the management device] Next, the hardware configuration of the management device 200 will be described with reference to Figure 5.

[0096] Figure 5 shows an example of the hardware configuration of the management device 200 according to this embodiment.

[0097] The functions of the management device 200 are realized by any hardware or any combination of hardware and software. For example, as shown in Figure 5, the management device 200 includes an external interface 201, an auxiliary storage device 202, a memory device 203, a CPU 204, a communication interface 206, an input device 207, and a display device 208, all connected via bus B2.

[0098] The external interface 201 functions as an interface for reading data from and writing data to the recording medium 201A. The recording medium 201A includes, for example, flexible disks, CDs (Compact Discs), DVDs (Digital Versatile Discs), BDs (Blu-ray® Discs), SD memory cards, USB memory, etc. This allows the management device 200 to read various data used in processing through the recording medium 201A, store it in the auxiliary storage device 202, and install programs that realize various functions.

[0099] Furthermore, the management device 200 may acquire various data and programs from external devices through the communication interface 206.

[0100] The auxiliary storage device 202 stores various installed programs, as well as files and data necessary for various processes. The auxiliary storage device 202 includes, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or flash memory.

[0101] When a program startup command is received, the memory device 203 reads the program from the auxiliary storage device 202 and stores it. The memory device 203 includes, for example, DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory).

[0102] The CPU 204 executes various programs loaded from the auxiliary storage device 202 into the memory device 203, and implements various functions related to the management device 200 according to the programs.

[0103] The high-speed computing unit 205 works in conjunction with the CPU 204 to perform calculations at a relatively high speed. The high-speed computing unit 205 includes, for example, a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field-Programmable Gate Array).

[0104] Furthermore, the high-speed arithmetic unit 205 may be omitted depending on the required processing speed.

[0105] The communication interface 206 is used as an interface for communicating with external devices. This allows the management device 200 to communicate with external devices, such as a shovel 100, through the communication interface 206. Furthermore, the communication interface 206 may have multiple types of communication interfaces depending on the communication method between the connected devices.

[0106] The input device 207 receives various inputs from the user. For example, the input device 207 includes an input device (remote control device) for an operator to perform remote operations.

[0107] The input device 207 includes, for example, an operation input device that receives mechanical operation input from a user. The operation input device includes, for example, buttons, toggles, levers, etc. The operation input device also includes, for example, a touch panel mounted on the display device 208, a touch pad provided separately from the display device 208, etc.

[0108] Furthermore, the input device 207 includes, for example, a voice input device capable of receiving voice input from a user. The voice input device includes, for example, a microphone capable of collecting the user's voice.

[0109] Furthermore, the input device 207 includes, for example, a gesture input device capable of receiving gesture input from the user. The gesture input device includes, for example, a camera capable of capturing images of the user's gestures.

[0110] Furthermore, the input device 207 includes, for example, a biometric input device capable of receiving biometric input from a user. The biometric input device includes, for example, a camera capable of acquiring image data containing information about the user's fingerprints or iris.

[0111] The display device 208 displays information screens and operation screens to the user. For example, the display device 208 includes the remote control display device mentioned above. The display device 208 is, for example, a liquid crystal display or an organic EL (electroluminescence) display.

[0112] [An example of the functional configuration of an excavator] Next, with reference to Figure 6, an example of the functional configuration of the shovel 100 will be described.

[0113] Figure 6 is a functional block diagram showing an example of the functional configuration of the shovel 100 according to this embodiment.

[0114] As shown in Figure 6, the controller 30 includes, as functional units, a display processing unit 301, an object detection unit 302, a position estimation unit 303, a safety control unit 304, and an irradiation control unit 305. The controller 30 may also include, as a functional unit, an image correction unit 306. The functions of the display processing unit 301, object detection unit 302, position estimation unit 303, safety control unit 304, and irradiation control unit 305 are realized, for example, by loading a program installed in the auxiliary storage device 30A into the memory device 30B and executing it on the CPU 30C.

[0115] The display processing unit 301 displays the surrounding image on the display device 50A inside the cabin 10 based on the captured image input from the imaging device 40.

[0116] The display processing unit 301 may display a peripheral image on the display device 50A that corresponds to the imaging range of any one of the cameras 40F, 40B, 40L, or 40R, or it may display the imaging ranges of any two or more of the cameras 40F, 40B, 40L, or 40R on the display device 50A. For example, the display processing unit 301 may display a peripheral image on the display device 50A that includes the imaging range of at least cameras 40B and 40R. This is because the rear and right side of the upper rotating body 3, which corresponds to the imaging range of cameras 40B and 40R, are likely to be blind spots from the perspective of the operator in the cabin 10.

[0117] The object detection unit 302 (an example of a detection unit) detects a predetermined object (hereinafter referred to as "monitoring object") in the vicinity of the shovel 100 based on the output (captured image) of the imaging device 40. Specifically, the object detection unit 302 may recognize the monitoring object from the captured image of the imaging device 40 and identify the position (region) in which the monitoring object is captured. Alternatively, the object detection unit 302 may recognize the monitoring object from the image after the captured image of the imaging device 40 (camera 40X) has been corrected by the image correction unit 306 (hereinafter referred to as "corrected captured image") and identify the position (region) in which the monitoring object is captured. In other words, the detection of a monitoring object may mean recognizing a monitoring object in the captured image of the imaging device 40 and identifying the position (region) in the input image that contains the monitoring object. Hereinafter, the image input to the object detection unit 302 (the captured image itself, which is the output of the imaging device 40, or the corrected captured image output from the image correction unit 306) may be conveniently referred to as the "input image".

[0118] The monitored objects may include people such as workers working around the shovel 100 or supervisors at the work site. The monitored objects may also include any objects (obstacles) other than people at the work site. Obstacles other than people at the work site may include, for example, specific terrain such as holes, ditches, and piles of soil; fixed obstacles (i.e., those that do not move under their own power) such as road cones, fences, utility poles, temporarily placed materials, and temporary offices at the work site. Furthermore, obstacles other than people at the work site may also include movable obstacles such as other work machinery or work vehicles. The number of types of monitored objects detected by the object detection unit 302 may be one or multiple. The following explanation will focus primarily on the case where the monitored object is a person.

[0119] Details of the object detection method by the object detection unit 302 will be described later.

[0120] Furthermore, the function of the object detection unit 302 may be switched between ON (enabled) and OFF (disabled) in response to a predetermined input from an operator or the like to the input device 52. Alternatively, the function of the object detection unit 302 may be transferred to the camera 40X, and the detection results may be transmitted to the controller 30. Alternatively, the function of the object detection unit 302 may be transferred to the display device 50A, and the detection results may be transferred to the controller 30. In this case, the image captured by the camera 40X is directly input to the display device 50A. Therefore, in this case, the function of the display processing unit 301 may also be transferred to the display device 50A. Alternatively, the function of the object detection unit 302 may be transferred to the imaging device 40. In this case, the function of the object detection unit 302 is divided into object detection units that detect monitored objects based on the images captured by cameras 40F, 40B, 40L, and 40R, and these units are built into each of the cameras 40F, 40B, 40L, and 40R (see Figure 25 described later).

[0121] When the object detection unit 302 detects a monitored object from the image captured by the imaging device 40, the position estimation unit 303 estimates the actual location of the detected monitored object, that is, the location of the monitored object around the shovel 100 (hereinafter referred to as the "actual location").

[0122] Specifically, the position estimation unit 303 estimates the actual location of the monitored object based on the detection position (detection area) of the monitored object in the captured image identified by the object detection unit 302. Furthermore, if multiple monitored objects are detected by the object detection unit 302, the position estimation unit 303 estimates the actual location of each of the multiple monitored objects.

[0123] For example, if camera 40X is a monocular camera, the position estimation unit 303 identifies a reference point (detection position) in the detection area of ​​the monitored object in the input image, and estimates the direction as seen from the shovel 100 (upper rotating body 3) based on the left-right position of that reference point in the input image. The position estimation unit 303 also estimates the distance from the shovel 100 to the monitored object (specifically, the distance in the direction along the work plane on which the shovel 100 is located) based on the size of the detection area of ​​the monitored object in the input image (for example, the size in the vertical direction). This is because there is a correlation between the size of the recognized monitored object in the input image and the size of the monitored object as it moves away from the shovel 100 (upper rotating body 3). Specifically, since the monitored object has a range of assumed size (for example, a range of assumed human height), the correlation between the position of the monitored object as seen from the shovel 100 within the assumed size range and its size in the input image can be predetermined. Therefore, the object detection unit 302 can estimate the actual location of the recognized monitoring object based on a map or conversion formula that represents the correlation between the size of the monitoring object on the input image and the distance from the upper rotating body 3, which is pre-stored in, for example, the auxiliary storage device 30A. Thus, the position estimation unit 303 can estimate the location of the recognized monitoring object by estimating the direction and distance of the monitoring object as seen from the shovel 100 (upper rotating body 3).

[0124] Furthermore, for example, if camera 40X is a monocular camera, the position estimation unit 303 may estimate the actual location of the monitored object by projecting the input image onto the same plane as the lower mobile body 1, assuming that the monitored object is on the same plane as the lower mobile body 1. Specifically, the position estimation unit 303 may identify a reference point (detection position) that represents the point of contact between the monitored object and the ground, and calculate the actual location of the monitored object with respect to that reference point by projecting the camera 40X onto the upper rotating body 3 according to its installation position and installation angle. In this case, a certain part of the input image (for example, a certain pixel) is associated one-to-one with a certain position on the same plane as the shovel 100 (lower mobile body 1).

[0125] Furthermore, for example, if camera 40X is a stereo camera, the actual location of the monitored object is estimated based on the difference (parallax) in the reference position of the detected area of ​​the monitored object for each of the two captured images.

[0126] The safety control unit 304 performs functional safety control of the shovel 100.

[0127] The safety control unit 304 activates the safety function, for example, when the object detection unit 302 detects a monitored object within a predetermined range around the shovel 100. Specifically, the safety control unit 304 may activate the safety function if the actual location (estimated value) of the monitored object, as estimated by the position estimation unit 303, is within a predetermined range around the shovel 100.

[0128] Safety features may include, for example, a notification function that outputs an alarm to at least one of the following locations: inside the cabin 10, outside the cabin 10, and a remote operator or monitor of the shovel 100, to notify the detection of a monitored object. This allows operators inside the cabin 10, workers around the shovel 100, and operators or monitors remotely controlling or monitoring the shovel 100 to be alerted to the presence of a monitored object within the monitoring area around the shovel 100. Hereinafter, the notification function to the inside of the cabin 10 (operator, etc.) may be referred to as the "internal notification function," the notification function to the outside of the shovel 100 (workers, etc.) as the "external notification function," and the notification function to operators or monitors remotely controlling or monitoring the shovel 100 as the "remote notification function," and these may be distinguished accordingly.

[0129] Furthermore, the safety features may include, for example, an operation restriction function that limits the movement of the shovel 100 in response to operation commands corresponding to the operation of the control device 26, remote control, or automatic driving function. This forcibly restricts the movement of the shovel 100, reducing the possibility of the shovel 100 approaching or coming into contact with surrounding objects. The operation restriction function may also include an operation deceleration function that slows down the operating speed of the shovel 100 to a slower rate than normal in response to operation commands corresponding to the operation of the control device 26, remote control, or automatic driving function. In addition, the operation restriction function may include an operation stop function that stops the movement of the shovel 100 and maintains a stopped state regardless of operation commands corresponding to the operation of the control device 26, remote control, or automatic driving function.

[0130] The safety control unit 304 activates the notification function when, for example, the object detection unit 302 detects a monitored object within a predetermined range around the shovel 100 (hereinafter referred to as the "notification range"). The notification range is, for example, the range in which the distance D from a predetermined part of the shovel 100 is less than or equal to a threshold Dth1. The predetermined part of the shovel 100 is, for example, the upper rotating body 3. Alternatively, the predetermined part of the shovel 100 may be, for example, the bucket 6 or hook at the tip of the attachment AT. The threshold Dth1 may be constant regardless of the direction viewed from the predetermined part of the shovel 100, or it may change depending on the direction viewed from the predetermined part of the shovel 100.

[0131] The safety control unit 304 activates internal and external notification functions by sound (i.e., by auditory means) to at least one of the inside and outside of the cabin 10, for example, by controlling the sound output device 50B. At this time, the safety control unit 304 may vary the pitch, sound pressure, timbre, blowing period (if the sound is blown periodically), and content of the sound output, depending on various conditions.

[0132] Furthermore, the safety control unit 304 may activate an internal notification function, for example, by visual means. Specifically, the safety control unit 304 may, through the display processing unit 301, control the display device 50A inside the cabin 10 to display an image on the display device 50A indicating that a monitored object has been detected, along with the surrounding image. The safety control unit 304 may also, through the display processing unit 301, highlight the monitored object shown in the surrounding image displayed on the display device 50A inside the cabin 10, or the position on the surrounding image corresponding to the detected monitored object. More specifically, the safety control unit 304 may superimpose a frame surrounding the detected monitored object onto the surrounding image displayed on the display device 50A inside the cabin 10, or superimpose a marker onto the position on the surrounding image corresponding to the actual location of the detected monitored object. This enables the display device 50A to provide a visual notification function to the operator. The safety control unit 304 may also use warning lights or lighting devices inside the cabin 10 to notify the operator inside the cabin 10 that a monitored object has been detected.

[0133] Furthermore, the safety control unit 304 may activate an external notification function in a visual manner by controlling an output device 50 (for example, a lighting device such as a headlight or a display device 50A) provided on the side of the housing section of the upper rotating body 3. Alternatively, the safety control unit 304 may activate an internal notification function in a tactile manner by controlling a vibration generating device that vibrates the cockpit where the operator sits. This allows the controller 30 to make the operator, workers and supervisors around the shovel 100 aware that there is a monitored object (for example, a person such as a worker) in a relatively close location around the shovel 100. As a result, the controller 30 can prompt the operator to check the safety conditions around the shovel 100, or prompt workers in the monitoring area to evacuate from the monitoring area.

[0134] Furthermore, the safety control unit 304 may activate the remote notification function by, for example, transmitting a command signal indicating the activation of the notification function to the management device 200 via the communication device 60. In this case, when the management device 200 receives a command signal from the shovel 100 via the communication interface 206, it may output an alarm in a visual or auditory way via the display device 208 or the like. This allows operators and monitors who remotely control or monitor the shovel 100 via the management device 200 to understand that a monitored object has entered the notification range around the shovel 100.

[0135] Furthermore, the remote notification function of the safety control unit 304 may be transferred to the management device 200. In this case, the management device 200 receives information from the shovel 100 regarding the detection status of monitored objects by the object detection unit 302 and the estimated location of the monitored objects by the position estimation unit 303. Based on the received information, the management device 200 determines whether or not a monitored object has entered the notification range, and activates the external notification function if a monitored object is present within the notification range.

[0136] Furthermore, the safety control unit 304 may vary the notification method (i.e., the way of notification) depending on the positional relationship between the monitored object detected within the notification range and the upper rotating body 3.

[0137] For example, if the monitoring object detected by the object detection unit 302 within the notification range is located relatively far from a predetermined part of the shovel 100, the safety control unit 304 may output a relatively low-urgency alarm (hereinafter referred to as a "caution level alarm") that draws attention to the monitoring object. Hereinafter, the range within the notification range where the distance to the predetermined part of the shovel 100 is relatively far, i.e., the range corresponding to a caution level alarm, may be conveniently referred to as the "caution notification range." On the other hand, if the monitoring object detected by the object detection unit 302 within the notification range is located relatively close to a predetermined part of the shovel 100, the safety control unit 304 may output a relatively high-urgency alarm (hereinafter referred to as a "warning level alarm") that indicates the monitoring object is approaching the predetermined part of the shovel 100 and the level of danger is increasing. Hereinafter, the range within the notification range where the distance to the predetermined part of the shovel 100 is relatively close, i.e., the range corresponding to a warning level alarm, may be referred to as the "warning notification range."

[0138] In this case, the safety control unit 304 may differentiate the pitch, sound pressure, timbre, and sounding cycle of the sound output from the sound output device 50B between a warning level alarm and a alert level alarm. The safety control unit 304 may also differentiate the color, shape, size, presence or absence of flashing, and flashing cycle of the image displayed on the display device 50A indicating that a monitored object has been detected, or of the image (e.g., frame or marker) that highlights the monitored object or the position of the monitored object on the surrounding image displayed on the display device 50A, between a warning level alarm and an alert level alarm. As a result, the controller 30 can allow the operator to understand the urgency, or in other words, the degree to which the monitored object is approaching a predetermined part of the shovel 100, based on the differences in the notification sound (alarm sound) output from the sound output device 50B and the notification image displayed on the display device 50A.

[0139] The safety control unit 304 may stop the notification function if, after the notification function has started, the monitored object detected by the object detection unit 302 is no longer detected within the notification range. Alternatively, the safety control unit 304 may stop the notification function if, after the notification function has started, a predetermined input to cancel the notification function is received via the input device 52.

[0140] Furthermore, the safety control unit 304 activates the operation restriction function when, for example, the object detection unit 302 detects a monitored object within a predetermined range around the shovel 100 (hereinafter referred to as the "operation restriction range"). The operation restriction range is set to be, for example, the same as the notification range described above. Alternatively, the operation restriction range may be set to a range where its outer edge is relatively closer to a predetermined part of the shovel 100 than the notification range. This allows the safety control unit 304 to, for example, activate the notification function first when a monitored object enters the notification range from the outside, and then activate the operation restriction function again when the monitored object enters the inner operation restriction range. Therefore, the controller 30 can activate the notification function and the operation restriction function in stages in accordance with the movement of the monitored object inward within the monitoring area.

[0141] Specifically, the safety control unit 304 may activate the operation limiting function when a monitored object is detected within the operation limiting range where the distance D from a predetermined part of the shovel 100 is within the threshold Dth2 (≤ Dth1). The threshold Dth2 may be constant regardless of the direction viewed from the predetermined part of the shovel 100, or it may change depending on the direction viewed from the predetermined part of the shovel 100.

[0142] Furthermore, the operating restriction range includes at least one of the following: an operating deceleration range that slows down the operating speed of the shovel 100 to a slower rate than normal in response to operating commands corresponding to the operation of the control device 26, remote operation, and automatic operation functions; and an operating stop range that stops the operation of the shovel 100 and maintains a stopped state regardless of the operating commands corresponding to the operation of the control device 26, remote operation, and automatic operation functions. For example, if the operating restriction range includes both the operating deceleration range and the operating stop range, the operating stop range is the range within the operating restriction range that is close to a predetermined part of the shovel 100. The operating deceleration range is the range set outside the operating stop range within the operating restriction range.

[0143] The safety control unit 304 activates an operation limiting function that restricts the operation of the shovel 100 by controlling the hydraulic control valve 31. In this case, the safety control unit 304 may restrict the operation of all driven elements (i.e., the corresponding hydraulic actuators), or it may restrict the operation of some driven elements (hydraulic actuators). This allows the controller 30 to slow down or stop the operation of the shovel 100 if a monitored object is present around the shovel 100. Therefore, the controller 30 can suppress the occurrence of contact between the monitored object around the shovel 100 and the shovel 100 or the suspended load. The safety control unit 304 may also activate the operation limiting function (operation stop function) by controlling an electromagnetic switching valve (not shown) of the pilot line 25 and shutting off the pilot line 25.

[0144] Furthermore, the safety control unit 304 may stop the operation restriction function if, after the operation restriction function has started, the monitored object detected by the object detection unit 302 is no longer detected within the operation restriction range. Also, the safety control unit 304 may stop the operation restriction function if, after the operation restriction function has started, a predetermined input for canceling the operation of the operation restriction function is received through the input device 52. The content of the input for canceling the notification function to the input device 52 and the content of the input for canceling the operation restriction function may be the same or different.

[0145] Furthermore, the safety control unit 304 may be switched between ON (enabled) and OFF (disabled) in response to a predetermined input from an operator or the like to the input device 52.

[0146] The irradiation control unit 305 performs control related to the irradiation device 70.

[0147] Furthermore, as mentioned above, if the irradiation device 70 is omitted, the irradiation control unit 305 will naturally also be omitted.

[0148] The image correction unit 306 performs a predetermined correction on the output (captured image) of the imaging device 40 (camera 40X) and outputs the corrected captured image to the object detection unit 302.

[0149] [Overview of object detection method] Next, with reference to Figures 7 to 15, an overview of the object detection method by the object detection unit will be described.

[0150] Figure 7 illustrates a specific example of the object detection method. Figures 8 to 13 show the first to sixth examples (images 800 to 1300) of the training data images. Specifically, Figures 8 to 13 are images captured by camera 40X and a similar camera installed at the same position on a shovel of the same model as shovel 100, and the side of the upper rotating body 3 is visible in the foreground (lower end) of the captured images. Figures 14 and 15 show an example and other examples of worker W.

[0151] The object detection unit 302 detects the monitored object from the image captured by the camera 40X by simply applying image processing techniques such as shape detection or pattern recognition (template matching).

[0152] Furthermore, as shown in Figure 7, the object detection unit 302 detects monitored objects from the image captured by the camera 40X by applying machine learning in addition to image processing technology. Specifically, the object detection unit 302 uses a trained model LM, which has been trained on the characteristics of monitored objects in the input image, to output a rectangular frame (hereinafter referred to as the "detection frame") representing the area in which the monitored object is located, and a label representing the type of monitored object, from the input image (the image captured by the camera 40X or the corrected image thereof). The labels include, for example, a label indicating that no monitored object is present, and labels set for each type of monitored object. Only one label may be set for each type of monitored object, or multiple labels may be set as described later. Also, if there are multiple types of monitored objects, a single trained model LM may be configured to output labels for all types of monitored objects, i.e., to detect all types of monitored objects, or multiple trained models LM may be provided, each capable of detecting only some of the types of monitored objects. For example, there may be a pre-trained model LM for each type of monitored object, and the labels for each pre-trained model LM may consist only of a label indicating the existence of a certain type of monitored object and a label indicating the absence of that type of monitored object.

[0153] A pre-trained model (LM) is generated by applying supervised learning to a base model. Specifically, a pre-trained model (LM) is generated by having the base model learn from a set of training data (training dataset) consisting of combinations of input images and output ground truth (detection frame and label). Alternatively, a pre-trained model (LM) may be generated (updated) by adding a new training dataset to an existing pre-trained model (LM). Naturally, the input images included in the training dataset include both images that contain (show) the monitored object and images that do not contain (show) the monitored object.

[0154] The trained model LM is generated, for example, by an external device such as the management device 200, and written to the auxiliary storage device 30A from a predetermined recording medium via the interface device 30D during the manufacturing of the shovel 100. Alternatively, the trained model LM may be downloaded from the external device such as the management device 200 to the shovel 100 via a predetermined communication line and registered in the auxiliary storage device 30A of the controller 30. As a result, the object detection unit 302 can detect monitored objects using the trained model LM registered in the auxiliary storage device 30A.

[0155] Furthermore, the trained model LM may be updated by installing update data from a predetermined recording medium to the auxiliary storage device 30A via the interface device 30D. Alternatively, the trained model LM may be updated by downloading update data from an external device such as the management device 200 to the shovel 100 via a predetermined communication line and installing it to the auxiliary storage device 30A. As a result, the object detection unit 302 can detect monitored objects using the updated, latest trained model LM.

[0156] For example, the object detection unit 302 detects a monitored object from an input image using a Support Vector Machine (SVM) that has been trained to recognize the trends in the image features of monitored objects in the image. In this case, the trained model LM includes a processing unit that extracts image features from the image captured by the camera 40X as a preliminary step. The image features are, for example, HOG (Histogram of Oriented Gradients) features.

[0157] Furthermore, for example, the object detection unit 302 detects monitored objects from the input image using machine learning with a deep neural network (DNN), that is, using a pre-trained model LM obtained through deep learning. Specifically, the object detection unit 302 may use a pre-trained model LM obtained through deep learning using a convolutional neural network (CNN) to detect monitored objects from the input image. A CNN is constructed by connecting multiple combinations of convolutional layers that perform convolutional processing and pooling layers that perform pooling processing using an activation function, and the final fully connected layer makes a final decision based on features (feature maps). An example of an activation function is ReLU (Rectified Linear Unit). As a result, the pre-trained model LM can handle the input image directly without requiring any prior processing.

[0158] For example, the object detection unit 302 detects a monitored object by using a pre-trained CNN-based model LM to generate candidate regions of monitored objects from the input image and classifying these candidates into labels. That is, the pre-trained CNN-based model LM may be a classification model that, for example, treats the detection of monitored objects from the input image as a classification problem, generates candidate regions of monitored objects from the image captured by the camera 40X, and classifies these candidates into labels. The classification model may be, for example, R (Region-based)-CNN or its derivatives (Fast R-CNN, Faster R-CNN, etc.).

[0159] Furthermore, for example, the object detection unit 302 detects a monitored object by simultaneously recognizing the monitored object and identifying its location (region) from the input image using a pre-trained model LM based on a CNN. That is, the pre-trained model LM based on a CNN may be a regression model that treats the detection of a monitored object from the image captured by the camera 40X as a regression problem and simultaneously performs the recognition of the monitored object and the identification of its location (region) from the input image. Examples of regression models include YOLO (You Only Look Once) and SSD (Single Shot Detector).

[0160] A trained model LM is generated, for example, by machine learning a base trained model or an existing trained model LM so that it can detect monitored objects with different poses.

[0161] For example, as shown in Figure 8, training data image 800 shows worker W in an upright position facing the camera. Also, as shown in Figure 9, training data image 900 shows worker W in a crouching position facing the camera. Furthermore, as shown in Figure 10, training data image 1000 shows worker W in a squatting position facing the camera. The trained model LM is trained using a training dataset that includes images of monitored objects in different postures, such as images 800 to 1000, enabling it to detect monitored objects (in this example, people) in different postures from the input images.

[0162] Furthermore, the trained model LM may be generated, for example, by machine learning a base trained model or an existing trained model LM so that it can detect monitored objects facing different orientations.

[0163] For example, as shown in Figure 11, training data image 1100 shows worker W in an upright position facing sideways (right) to the camera. Also, as shown in Figure 12, training data image 1200 shows worker W in a crouching position with their back to the camera. Furthermore, as shown in Figure 13, training data image 1300 shows worker W in a squatting position facing sideways (left) to the camera. The trained model LM can detect monitoring objects facing different directions from input images by being trained on a training dataset that includes images of monitoring objects facing different directions, such as images 800, 1100, images 900, 1200, and images 1000, 1300.

[0164] Furthermore, the trained model LM may be generated, for example, by machine learning a base trained model or an existing trained model LM so that it can detect surveillance objects whose orientation and orientation differ from each other. Specifically, the trained model LM can be machine-learned using a training dataset that includes images showing surveillance objects with different orientations and orientations, such as images 800 to 1300, so that surveillance objects with different orientations and orientations can be detected from the input image.

[0165] Furthermore, if the object being monitored is a person, the trained model LM may be generated by machine learning to detect (recognize) a person or garment that is worn relatively frequently by workers around the shovel 100. This allows the controller 30 to appropriately recognize the characteristics of the garment, detect the person or garment, and understand the presence of a person near the shovel 100, even in the case of captured images where it is difficult to distinguish between work clothes and the background.

[0166] For example, as shown in Figure 14, the trained model LM may be generated by machine learning to detect (recognize) a person wearing a helmet (worker W) and the helmet HMT they are wearing. Specifically, the trained model LM can be trained using a training dataset containing many images of workers W wearing helmet HMTs, thereby enabling the detection of people wearing helmets and the helmets they are wearing from input images.

[0167] Furthermore, as shown in Figure 15, for example, the trained model LM may be generated by machine learning to detect (recognize) a person (worker W) wearing high-visibility safety clothing RV and the high-visibility safety clothing RV being worn by that person. High-visibility safety clothing RV is configured to be relatively visible from the surroundings of the wearer. For example, retroreflective material is attached to the high-visibility safety clothing RV. Also, fluorescent fabric colored with fluorescent yellow or fluorescent green is used for the high-visibility safety clothing RV. Specifically, the trained model LM can detect a person wearing high-visibility safety clothing and the high-visibility safety clothing being worn by that person from the input image by machine learning using a training dataset that includes many images of workers W wearing high-visibility safety clothing RV.

[0168] Furthermore, as shown in Figure 15, for example, the trained model LM may be machine-learned to detect (recognize) both a person (worker W) wearing both a helmet HMT and high-visibility safety suit RV, as well as both the high-visibility safety suit RV and the helmet HMT being worn. In this case, the trained model LM may be capable of outputting both a label indicating the presence of a helmet and a label indicating the presence of high-visibility safety suit. Specifically, the trained model LM can be machine-learned using a training dataset containing many images of workers W wearing helmet HMT and high-visibility safety suit RV, thereby enabling it to detect people wearing helmets and high-visibility safety suit, as well as the helmets and safety suit being worn, from input images.

[0169] Furthermore, the training dataset may include, for example, images of monitored objects captured by a camera of the same model as camera 40X. Alternatively, the training dataset may include images of monitored objects captured by a camera of the same model as camera 40X, installed in approximately the same position and orientation as excavator 100, as shown in Figures 8 to 13. This allows the trained model LM to be machine-learned to detect monitored objects from images captured by camera 40X, taking into account how the monitored objects appear in the images captured by camera 40X. Therefore, the object detection unit 302 can detect monitored objects with greater accuracy.

[0170] Furthermore, the training dataset includes images in which the monitored object is visible against multiple different types of backgrounds. For example, these background types include soil, asphalt, forests, houses, and urban buildings. This allows the trained model LM to be machine-learned to detect the monitored object from the input image, taking into account how the monitored object appears against different backgrounds. As a result, the object detection unit 302 can detect the monitored object with greater accuracy.

[0171] Furthermore, the training dataset includes, for example, images of the monitored object taken at different time periods. These time periods include, for example, the morning (e.g., 6am-10am), midday (e.g., 10am-3pm), evening (e.g., 4pm-7pm), and nighttime (e.g., 7pm-6am the next day). This allows the trained model LM to be machine-learned to detect the monitored object from the input image, taking into account how the monitored object appears at different time periods. As a result, the object detection unit 302 can detect the monitored object with greater accuracy.

[0172] Furthermore, the training dataset includes, for example, images of monitored objects taken under multiple different lighting conditions during the nighttime hours. Lighting conditions include, for example, the intensity and color of the lighting. As a result, the trained model LM is machine-learned to detect monitored objects from input images, taking into account how monitored objects appear under different lighting conditions. Therefore, the object detection unit 302 can detect monitored objects with greater accuracy.

[0173] Furthermore, the training dataset includes, for example, images of monitored objects taken under multiple different weather conditions. These multiple weather conditions include, for example, sunny, cloudy, rainy, and snowy conditions. This allows the trained model LM to be machine-learned to detect monitored objects from input images, taking into account how monitored objects appear under different weather conditions. As a result, the object detection unit 302 can detect monitored objects with greater accuracy.

[0174] Furthermore, the training dataset includes, for example, multiple images of the monitored object where the positional relationship between the light source and the imaging area differs from one another. The light source is, for example, the sun or nighttime lighting. The multiple images include images corresponding to different positional relationships between the light source and the imaging area, such as front lighting, semi-front lighting, side lighting, and back lighting. In addition, the multiple images include images where, for example, the same front lighting, side lighting, and back lighting conditions are met, but the height (elevation angle) of the sun from the ground differs. As a result, the training dataset is machine-learned to detect monitored objects from the input images, taking into account how monitored objects appear due to differences in the position of the light source. Therefore, the object detection unit 302 can detect monitored objects with greater accuracy.

[0175] [Specific examples of object detection methods] Next, with reference to Figures 16 to 18, a specific example of how the object detection unit 302 detects monitored objects will be described.

[0176] <Example 1> In this example, the object detection unit 302 detects people (workers) around the shovel 100 by detecting (recognizing) high-visibility safety clothing from the input image. As a result, the object detection unit 302 can more easily detect people even from input images where it is difficult to distinguish between the work clothes of people around the shovel 100 and the background, and consequently, it can detect people as monitored objects with greater accuracy.

[0177] Specifically, the object detection unit 302 may detect (recognize) a person wearing high-visibility safety clothing from the input image, or it may detect (recognize) the high-visibility safety clothing itself worn by a person, or it may detect (recognize) both. In this way, when the object detection unit 302 detects a person wearing high-visibility safety clothing or high-visibility safety clothing (worn by a person), it can determine that it has detected a person as a surveillance object.

[0178] For example, the object detection unit 302 detects people around the shovel 100 by recognizing at least one of the shape, color (e.g., fluorescent yellow or fluorescent green) and luminance of the retroreflective material of the high-visibility safety clothing. Specifically, the object detection unit 302 may recognize at least one of the shape, color, and luminance of the retroreflective material of the high-visibility safety clothing by applying shape detection, pattern recognition, etc. to the shape, color, and luminance of the retroreflective material of the high-visibility safety clothing. Alternatively, as described above, the object detection unit 302 may recognize at least one of the shape, color, and luminance of the retroreflective material of the high-visibility safety clothing by using a trained model LM based on a training dataset containing many images of people (workers) wearing high-visibility safety clothing.

[0179] In this example, the imaging device 40 (camera 40X) may acquire images of the area around the shovel 100 while visible light is being emitted from the illumination device 70. In this case, the illumination device 70 may continuously emit visible light while the camera 40X is operating (power on) under the control of the controller 30 (illumination control unit 305), or it may emit visible light in synchronization with the imaging timing of the camera 40X. As a result, if a person (worker) wearing high-visibility safety clothing is present within the imaging range of the imaging device 40, the retroreflective material and fluorescent colors of the high-visibility safety clothing will appear more clearly in the images captured by the camera 40X. Therefore, the object detection unit 302 can more easily detect (recognize) the high-visibility safety clothing from the input image (image captured by the camera 40X or image corrected by the image correction unit 306), and as a result, it can detect people as monitored objects with greater accuracy.

[0180] In this example, the imaging device 40 (camera 40X) may acquire images of the area around the shovel 100 while infrared light is being emitted from the illumination device 70. In this case, the illumination device 70 may continuously emit infrared light while the camera 40X is operating (powered on) under the control of the controller 30 (illumination control unit 305), or it may emit infrared light intermittently (periodically) in synchronization with the imaging timing of the camera 40X. As a result, if a person (worker) wearing high-visibility safety clothing is present within the imaging range of the imaging device 40, the outline of the retroreflective material of the high-visibility safety clothing will be more clearly visible in the image captured by the camera 40X. Therefore, the object detection unit 302 can more easily detect (recognize) the high-visibility safety clothing from the input image, and as a result, it can detect people as monitored objects with greater accuracy. In addition, unlike visible light, infrared light cannot be directly perceived by the human eye, thus suppressing the impact on people around the shovel 100.

[0181] In this example, the irradiation device 70 may also switch between a state of continuously irradiating infrared light and a state of intermittently irradiating infrared light in synchronization with the imaging timing of the camera 40X, under the control of the controller 30. For example, the irradiation control unit 305 determines whether or not the surrounding image displayed on the display device 50A or the display device 208 is affected by infrared light. The irradiation control unit 305 then switches the irradiation state of the infrared light from the irradiation device 70 according to the determination result. The effect of infrared light means, for example, that the surrounding image on the display device 50A becomes reddish when the camera 40X captures near-infrared light reflected from the subject (e.g., high-visibility safety clothing). The irradiation control unit 305 may determine whether or not there is an effect of infrared light by image analysis. Alternatively, the irradiation control unit 305 may determine that there is an effect of infrared light by receiving predetermined input from the user (operator or supervisor) through the input device 52 or the communication device 60. As a result, when an operator or monitor recognizes that the surrounding image of the display device 50A or the display device 208 is in a reddish state, they can notify the controller 30 of that state by making a predetermined input through the input device 52 or the input device 207.

[0182] Specifically, the irradiation control unit 305 normally continues to irradiate infrared light from the irradiation device 70 when the surrounding images displayed on the display device 50A and the display device 208 are not affected by infrared light. On the other hand, when the surrounding images displayed on the display device 50A and the display device 208 are affected by infrared light, the irradiation control unit 305 intermittently irradiates infrared light in synchronization with the imaging timing of the camera 40X. This makes it possible to balance suppressing the control load of the irradiation device 70 by the controller 30 with suppressing the influence of infrared light on the surrounding images of the display device 50A, etc.

[0183] <Example 2> Figure 16 shows an example of an image captured by the imaging device 40 (camera 40X). Figure 17 shows an example of a corrected image (corrected captured image) from the imaging device 40 (camera 40X) by the image correction unit 306.

[0184] In this example, the object detection unit 302 detects the object being monitored using the corrected image (corrected captured image) of the image captured by the imaging device 40, which has been corrected by the image correction unit 306, as the input image.

[0185] For example, from the perspective of the operator in cabin 10, it is difficult to directly see the rear, left side, and right side of the upper rotating body 3. Also, for example, a remote operator cannot directly see the surrounding area, including the front of cabin 10. Therefore, it becomes necessary to cover the 360-degree range around the shovel 100 when viewed from above with four cameras 40F, 40B, 40L, and 40R. As a result, for example, as shown in Figure 2, the four cameras 40F, 40B, 40L, and 40R may be wide-angle cameras with a very large field of view in the left-right direction, and wide-angle distortion (volume distortion) may occur in the captured images.

[0186] Furthermore, as shown in Figures 1 and 2, for example, the camera 40X is installed on top of the upper rotating body 3. Therefore, the camera 40X needs to detect monitored objects, including those near the ground, from a relatively high position, and the optical axis of the camera 40X is set diagonally downward. As a result, distortion due to perspective may occur in the images captured by the camera 40X.

[0187] For example, as shown in Figure 16, workers W1 to W3 are visible in the center, left edge, and right edge of the image captured by camera 40X, respectively. Worker W1 in the center of the image captured by camera 40X is positioned so that its vertical axis approximately coincides with the vertical axis of the image. The term "approximately" is intended to allow for manufacturing errors in the shovel 100 and installation errors in camera 40X. On the other hand, workers W2 and W3 in the image captured by camera 40X have their vertical axes tilted to the left and right, respectively, relative to the vertical axis. This is because the image captured by camera 40X is distorted so that the vertical axes of subjects at the left and right edges are tilted towards the edges relative to the vertical axis.

[0188] In contrast, in this example, the image correction unit 306 corrects the image captured by the camera 40X so that the difference between the vertical axis of the image captured by the camera 40X and the vertical axis for the subject is reduced, and outputs the corrected image.

[0189] For example, as shown in Figure 17, the image correction unit 306 generates a corrected image by tilting only the central part and the left and right edges of the image captured by the camera 40X inward by a predetermined amount. As a result, the difference between the vertical axis of the image and the vertical axis for the subjects in the corrected image becomes relatively small for the workers W2 and W3.

[0190] Furthermore, the image correction unit 306 may correct the distortion of the image captured by the camera 40X itself using known methods such as projection transformation. This allows the image correction unit 306 to reduce the difference between the vertical axis in the corrected image and the vertical axis for the subject.

[0191] In this example, the object detection unit 302 is configured to detect only the former of two types of surveillance objects: those with a relatively small difference between the vertical axis in the input image and the vertical axis relative to the subject, and those with a relatively large difference. Surveillance objects with a relatively large difference between the vertical axis in the image and the vertical axis relative to the subject are, for example, the workers W2 and W3 in Figure 16, which appear at both ends of the image captured by the camera 40X. Surveillance objects with a relatively small difference between the vertical axis in the image and the vertical axis relative to the subject are, for example, the worker W1 in Figure 16, which appear at both ends of the image captured by the camera 40X. Surveillance objects with a relatively small difference between the vertical axis in the image and the vertical axis relative to the subject are, for example, the workers W1 to W3 in Figure 17, which appear in the corrected image captured by the camera 40X. As a result, the object detection unit 302 can detect surveillance objects using the corrected image captured as the input image.

[0192] Specifically, the object detection unit 302 may detect surveillance objects from the input image by applying shape detection, pattern recognition, etc., to surveillance objects where the difference between the vertical axis in the input image and the vertical axis for the subject is relatively small. Alternatively, the object detection unit 302 may use a pre-trained model LM based on a training dataset that includes only images containing surveillance objects where the difference between the vertical axis in the input image and the vertical axis for the subject is relatively small, and images containing surveillance objects where the difference is relatively large. This allows the object detection unit 302 to detect surveillance objects with higher accuracy because it only needs to recognize surveillance objects where the difference between the vertical axis in the image and the vertical axis for the subject is relatively small from the input image (corrected captured image). Furthermore, when a pre-trained model LM is used, the efficiency of machine learning can be improved because it only needs to learn the features of surveillance objects where the difference between the vertical axis in the image and the vertical axis for the subject is relatively small from the image (corrected captured image).

[0193] Furthermore, when the trained model LM is used, the training dataset may include, as described above, images of the monitored object captured by a camera of the same model as camera 40X. Alternatively, as shown in Figures 8 to 13, the training dataset may include images of the monitored object captured by a camera of the same model as camera 40X, which is installed in approximately the same position and orientation as a shovel of the same model as shovel 100. In this case, the images captured by the camera of the same type as camera 40X are corrected by the same function as the image correction unit 306, and the training dataset may include the corrected images. As a result, the trained model LM is machine-learned to detect monitored objects by considering how they appear in the images captured by camera 40X, and to detect only monitored objects where the difference between the vertical axis in the image and the vertical axis for the subject is relatively small. Therefore, the object detection unit 302 can detect monitored objects with greater accuracy. In this case, the training dataset may also include corrected images corresponding to both captured images in which the monitored object is located in the center of the image in the left-right direction, and captured images in which the monitored object is located at at least one edge in the left-right direction. This allows the trained model LM to be machine-learned to detect both the uncorrected monitored object in the center of the input image (corrected captured image) and the corrected monitored object at the edges of the input image in the left-right direction. As a result, the object detection unit 302 can detect the monitored object with greater accuracy.

[0194] <Example 3> Figure 18 shows an example of an image captured where only a portion of the entire body of the monitored object (person) is visible.

[0195] For example, as shown in Figure 18, the image captured by camera 40X may only show a portion of the entire body of a person (worker W) as a monitored object, including the lower half of the body, located at both ends of the left-right imaging range. Specifically, in Figure 18 (first example), the lower half of worker W and part of the upper half (waist area) are visible at a position relatively close to the upper rotating body 3 at the right edge of the image captured by camera 40X. Also, in Figure 19 (second example), the lower half of worker W and part of the upper half (chest, waist, arms, and hands, excluding the head) are visible at a position relatively close to the upper rotating body 3 at the left edge of the image captured by camera 40X. Furthermore, in Figure 20 (third example), part of the lower half of worker W (lower legs and feet) is visible at a position relatively close to the upper rotating body 3 at the right edge of the image captured by camera 40X. Furthermore, in Figure 21 (Fourth Example), a portion of the lower body of worker W (lower legs and feet) is visible at a position relatively far from the upper rotating body 3 at the left edge of the image captured by camera 40X. This is because camera 40X has an optical axis that is angled downward from the top surface of the upper rotating body 3 toward the ground, and the upper body of worker W (person) as the monitored object falls outside the three-dimensional imaging range of camera 40X.

[0196] Furthermore, if an obstacle exists between the monitored object and the camera 40X, the lower part (lower body) of the monitored object (person) may be hidden by the obstacle, and the image captured by the camera 40X may only show a portion of the monitored object, including the upper part (upper body).

[0197] In contrast, in this example, the object detection unit 302 detects (recognizes) a monitored object (person) by detecting (recognizing) only a predetermined part of the entire monitored object (person) from the input image. The predetermined part is, for example, the lower part (lower body) or upper part (upper body) of the monitored object (person), as described above. As a result, the object detection unit 302 can detect the monitored object even if only a portion of the entire monitored object, including the predetermined part, is visible in the input image. Therefore, the object detection unit 302 can detect the monitored object with greater accuracy.

[0198] Specifically, the object detection unit 302 may detect a predetermined part of the monitored object from the input image by applying shape detection, pattern recognition, or the like to that predetermined part of the monitored object. Alternatively, the object detection unit 302 may detect a predetermined part of the monitored object using a trained model LM based on a training dataset that includes images showing only a portion of the monitored object, including a predetermined part of the whole object.

[0199] Furthermore, the object detection unit 302 may be configured to detect (recognize) both the entire monitored object and a predetermined part of the entire monitored object. This allows the object detection unit 302 to determine that a monitored object has been detected, for example, when it detects either the entire monitored object or a predetermined part of the monitored object. Therefore, the object detection unit 302 can detect monitored objects with greater accuracy.

[0200] For example, the object detection unit 302 uses a trained model LM based on a training dataset that includes images showing the entire monitored object and images showing only a portion of the monitored object, including a predetermined part of the entire object, to detect both the entire monitored object and a predetermined part of the monitored object. In this case, the object detection unit 302 may output a label corresponding to the detected monitored object from among a plurality of labels, including a label representing the entire monitored object and a label representing a predetermined part of the entire monitored object. The object detection unit 302 may then determine that a monitored object has been detected if the trained model LM outputs either a label representing the entire monitored object or a label representing a predetermined part of the entire monitored object.

[0201] Furthermore, when the trained model LM is used, the training dataset may include, for example, images captured by a camera of the same model as camera 40X, where only a portion of the monitored object, including its lower part, is visible at at least one of the left and right edges. Alternatively, the training dataset may include, for example, images captured by a camera of the same model as camera 40X, installed in approximately the same position and orientation as shovel 100, where only a portion of the monitored object, including its lower part, is visible at at least one of the left and right edges. This allows the trained model LM to be trained to detect the lower part of the monitored object (e.g., the lower half of a person) by considering how the monitored object appears in the images captured by camera 40X. As a result, the object detection unit 302 can detect a predetermined portion of the monitored object with greater accuracy.

[0202] Furthermore, the object detection unit 302 may detect the entire monitored object based on the image of the central part of the input image in the left-right direction, or detect only the lower part of the monitored object (for example, the lower half of a person) based on the images of both ends of the input image in the left-right direction. This is because, as described above, in the input image, only the lower part of the monitored object (for example, the lower half of a person) may be visible at both ends in the left-right direction.

[0203] Furthermore, the object detection unit 302 may detect a monitored object in a region when it detects (recognizes) a predetermined part of the monitored object in both of the overlapping regions of the imaging ranges of two cameras 40X whose imaging ranges partially overlap. Two cameras 40X whose imaging ranges partially overlap correspond to any one of the following combinations: camera 40F,40R, camera 40R,40B, camera 40B,40L, and camera 40L,40F. As a result, even if the object detection unit 302 erroneously detects a predetermined part of the monitored object based on one camera 40X in a region of the overlapping imaging ranges of the two cameras 40X, it can determine that the object is not detected because the other camera 40X has not detected the predetermined part of the monitored object. Therefore, the object detection unit 302 can suppress erroneous detections when detecting a predetermined part of the entire monitored object.

[0204] [Specific examples of methods for estimating the actual location of a monitored object] Next, with reference to Figures 22 to 24, we will explain a specific example of a method for estimating the actual location of a detected monitored object.

[0205] Figures 22 to 24 show an example, another example, and yet another example of the detection state of a monitored object (person) by the object detection unit 302. Specifically, Figures 22 to 24 show the detection frame FR which represents the detection range of the monitored object in the input image when a monitored object (worker W) is detected from the input image by the object detection unit 302.

[0206] The images in Figures 22 to 24 may be displayed on the display device 50A or the remote control display device when, for example, the object detection unit 302 detects a monitored object (worker W). This allows the operator or supervisor to understand the detection status of monitored objects (worker W) around the shovel 100 while viewing the surrounding image (image captured by camera 40X).

[0207] In the situations shown in Figures 22 and 23, worker W is standing upright in the same location, relatively close to the upper rotating body 3, with their front facing the camera 40X. In the situation shown in Figure 24, worker W is standing upright in a location relatively far from the upper rotating body 3, with their front facing the camera 40X.

[0208] For example, in the situation shown in Figure 22, the object detection unit 302 detects (recognizes) the entire worker W as a monitored object (person) in the input image (for example, the image captured by camera 40X). Therefore, the detection frame FR is represented as a rectangle surrounding the worker W.

[0209] On the other hand, for example, in the situation shown in Figure 23, the object detection unit 302 detects (recognizes) only the upper part (upper body portion) of the worker W shown in the input image as a monitored object (person). Therefore, the detection frame FR is represented as a rectangle that surrounds only the upper part of the worker W, including the head where the helmet is worn and the torso where the high-visibility safety suit is worn.

[0210] When the object detection unit 302 detects (recognizes) a person by detecting (recognizing) clothing such as a helmet or high-visibility safety suit, it may determine that only a portion of the area corresponding to the person's clothing in the input image is within the detection range of the monitored object. Furthermore, depending on the background of the monitored object and the weather conditions at the time, the shape and other features of the monitored object in the input image may become unclear, and the object detection unit 302 may only be able to recognize a portion of the entire monitored object as the monitored object. Therefore, the object detection unit 302 may only detect (recognize) a portion of the entire monitored object in the vertical direction in the input image.

[0211] For example, in the situation shown in Figure 23, if the actual location of worker W is estimated based on the size of the detected (recognized) monitoring object on the input image (the vertical size of the detection frame FR), as described above, it is possible that the system may determine that the worker is located further from the upper rotating body 3 than is actually the case. Specifically, even though worker W is in the same position as in the situation shown in Figure 22, in the situation shown in Figure 23, the estimated actual location may be further from the upper rotating body 3 than the actual location estimated in the situation shown in Figure 22.

[0212] Furthermore, for example, although the distance between the worker W and the upper rotating body 3 differs between the situation in Figure 23 and the situation in Figure 24, the detection position of the person (worker W) on the input image, specifically the lower end position of the detection range (detection frame FR), is approximately the same. Therefore, as described above, when the actual location of the monitored object is estimated by projection transformation, etc., based on the detection position (reference point) of the monitored object on the input image, it is possible that an actual location relatively far from the upper rotating body 3 will be estimated, similar to the worker W detected in the situation in Figure 24.

[0213] In contrast, in this example, the position estimation unit 303 estimates the actual location of the monitored object based on information regarding the vertical detection position of the monitored object on the input image, and information regarding the size and shape of the detection range of the monitored object on the input image. The information regarding the vertical detection position of the monitored object on the input image refers to a coordinate position that represents the detection range of the monitored object on the input image, and includes, for example, the coordinates of the lower end, upper end, and centroid of the detection range on the input image of the monitored object. The information regarding the size of the detection range on the input image of the monitored object includes, for example, information regarding the height (vertical dimension), width (horizontal dimension), aspect ratio, diagonal, and area of ​​the detection range. The information regarding the shape of the detection range on the input image of the monitored object includes, for example, information regarding the aspect ratio.

[0214] Specifically, the position estimation unit 303 may determine whether the shape and size of the detection range of the monitored object on the input image are consistent with the vertical detection position of the monitored object on the input image. Assuming that the entire monitored object is detected in the input image, the vertical detection position of the monitored object on the input image determines the expected range of the shape and size of the detection range of the monitored object. Therefore, the position estimation unit 303 can determine whether the entire monitored object is detected by checking whether the shape and size of the detection range of the monitored object on the input image are consistent with the vertical detection position of the monitored object on the input image, specifically whether they fall within this expected range.

[0215] The expected range of shape and size of the detection area of ​​a monitored object on the input image may be uniformly defined according to the vertical detection position of the monitored object on the input image. Furthermore, the expected range of shape and size of the detection area of ​​a monitored object on the input image may also consider other conditions in addition to the vertical detection position of the monitored object on the input image. For example, the expected range of shape and size of the detection area of ​​a monitored object on the input image may vary depending on the orientation and orientation of the detected monitored object, in addition to the vertical detection position. This is because the appearance of the monitored object on the input image differs depending on its orientation and orientation, and as a result, the shape and size of the detection area of ​​the monitored object on the input image changes. The orientation and orientation of the monitored object can be obtained, for example, by machine learning a trained model LM so that it can detect the monitored object for each different orientation and orientation, that is, so that it can detect the monitored object for each of multiple labels corresponding to different orientations and orientations. Furthermore, the expected range of the shape and size of the detection area on the input image (captured image) of the monitored object may be defined not only by considering the vertical detection position on the input image (captured image) of the monitored object, but also by considering the distortion caused by the lens at that detection position.

[0216] The position estimation unit 303 determines that the entire monitored object has been detected if there is consistency between the shape and size of the detection range of the monitored object on the input image and the vertical detection position of the monitored object on the input image. In this case, the position estimation unit 303 may estimate the actual location of the monitored object based on the detection position and detection range of the monitored object on the input image, as described above.

[0217] On the other hand, if the position estimation unit 303 finds that there is no consistency between the shape and size of the detection range of the monitored object on the input image and the vertical detection position of the monitored object on the input image, it determines that only a part of the monitored object has been detected. In this case, the position estimation unit 303 may correct the detection position or detection range of the monitored object on the input image and estimate the actual location of the monitored object based on the corrected detection position or detection range.

[0218] For example, if the width of the detection range of the monitored object on the input image deviates significantly from the expected range relative to the vertical detection position of the monitored object on the input image, the position estimation unit 303 determines that only a portion of the upper part of the monitored object in the input image has been detected (see Figure 23).

[0219] Furthermore, for example, if the aspect ratio of the detection range of the monitored object on the input image deviates from the expected range relative to the vertical detection position of the monitored object on the input image in the direction of increasing horizontal dimension, the position estimation unit 303 determines that only a portion of the monitored object in the vertical direction of the entire monitored object in the input image has been detected. In addition, if the object detection unit 302 detects only a predetermined portion of the monitored object or only the clothing worn by the monitored object, the position estimation unit 303 may determine whether only the upper or lower portion of the monitored object in the vertical direction of the entire monitored object in the input image has been detected, taking into account that predetermined portion or the part of the clothing being worn. For example, if the object detection unit 302 detects (recognizes) a helmet or high-visibility safety clothing, the position estimation unit 303 determines that only the upper portion of the monitored object (person) in the input image, including the part of the helmet or high-visibility safety clothing being worn, has been detected (see Figure 23). Furthermore, if the position estimation unit 303 determines that only a portion of the monitored object in the input image in the vertical direction has been detected, it may, prioritizing safety, uniformly consider that only the upper portion of the monitored object in the vertical direction has been detected.

[0220] In this case, the position estimation unit 303 may, for example, perform a correction by extending the detection range on the input image of the monitored object downwards so that it conforms to the shape (aspect ratio) expected of the monitored object, based on information regarding the height and width of the detection range on the input image of the monitored object. The shape (aspect ratio) expected of the monitored object is defined, for example, according to the detection position on the input image of the monitored object. Furthermore, the shape (aspect ratio) expected of the monitored object may be defined, for example, according to the orientation and posture of the monitored object, lens distortion at its detection position, etc., in addition to the detection position on the input image of the monitored object. The position estimation unit 303 may also, for example, perform a correction by moving the detection position of the monitored object downwards or extending the detection range of the monitored object downwards, based on information regarding the width of the detection range on the input image of the monitored object, so that it conforms to the width of the detection range on the input image of the monitored object. In this case, the position estimation unit 303 may correct the detection position and detection range of the monitored object by considering, for example, the orientation and posture of the monitored object, lens distortion at its detection position, etc. The position estimation unit 303 may then estimate the actual location of the monitored object based on the corrected detection range and detection position of the monitored object.

[0221] Thus, in this example, the position estimation unit 303 can understand that the object detection unit 302 has detected only a portion of the monitored object in the input image (particularly the upper portion in the vertical direction), and can estimate the actual location of the monitored object by taking this state into consideration. Therefore, the position estimation unit 303 can suppress situations such as estimating the actual location of the monitored object as being further away from the upper rotating body 3 than it actually is, and can estimate the actual location of the monitored object more appropriately. As a result, the controller 30 (safety control unit 304) can activate the safety functions more appropriately, and as a result, the safety of the shovel 100 can be improved.

[0222] [Other examples of excavator functional configurations] Next, with reference to Figure 25, another example of the functional configuration of shovel 100 will be described. The following explanation will focus on the differences from the example described above (Figure 6), and explanations that are the same as or corresponding to the above example may be omitted.

[0223] Figure 25 is a functional block diagram showing another example of the functional configuration of the shovel 100 according to this embodiment.

[0224] As shown in Figure 25, the controller 30 includes a display processing unit 301, an object detection unit 302, a position estimation unit 303, a safety control unit 304, and an irradiation control unit 305, similar to the example described above, and may also include an image correction unit 306. In addition, the controller 30 includes a detection and judgment unit 307, unlike the example described above.

[0225] The imaging device 40 includes an object detection unit 402. The imaging device 40 may also include an image correction unit 404.

[0226] The object detection unit 402 (an example of a detection unit) detects monitored objects based on the image captured by the imaging device 40, similar to the object detection unit 302. The object detection unit 402 includes object detection units 402F, 402B, 402L, and 402R. Hereinafter, the object detection units 402F, 402B, 402L, and 402R may be collectively referred to as "object detection unit 402X," or any one of them may be referred to individually.

[0227] The object detection unit 402F is mounted on the camera 40F and, like the object detection unit 302, detects monitored objects based on the image captured by the camera 40F. Specifically, the object detection unit 402F may recognize a monitored object from the image captured by the camera 40F and identify the location (area) in which the monitored object is visible. Alternatively, the object detection unit 402F may recognize a monitored object from the image after the image captured by the camera 40F (i.e., the image generated by the image capture generation function) has been corrected by the image correction unit 404F (corrected image), and identify the location (area) in which the monitored object is visible. The function of the object detection unit 402F can be realized by any hardware or any combination of hardware and software. For example, the function of the object detection unit 402F can be realized by loading a program installed in the auxiliary storage device of the image processing engine 42 of the camera 40F into a memory device and executing it on the CPU. The same applies to the functions of the object detection units 402B, 402L, and 402R.

[0228] The object detection unit 402B is mounted on the camera 40B and, like the object detection unit 302, detects monitored objects based on the image captured by the camera 40B. Specifically, the object detection unit 402B may recognize a monitored object from the image captured by the camera 40B and identify the location (area) in which the monitored object is visible. Alternatively, the object detection unit 402B may recognize a monitored object from the image after the image captured by the camera 40B (i.e., the image generated by the image capture generation function) has been corrected by the image correction unit 404B (corrected image), and identify the location (area) in which the monitored object is visible.

[0229] The object detection unit 402L is mounted on the camera 40L and, like the object detection unit 302, detects monitored objects based on the image captured by the camera 40L. Specifically, the object detection unit 402L may recognize a monitored object from the image captured by the camera 40L and identify the location (area) in which the monitored object is visible. Alternatively, the object detection unit 402L may recognize a monitored object from the image after the image captured by the camera 40L (i.e., the image generated by the image capture generation function) has been corrected by the image correction unit 404L (corrected image), and identify the location (area) in which the monitored object is visible.

[0230] The object detection unit 402R is mounted on the camera 40R and, like the object detection unit 302, detects monitored objects based on the image captured by the camera 40R. Specifically, the object detection unit 402R may recognize a monitored object from the image captured by the camera 40R and identify the location (area) in which the monitored object is visible. Alternatively, the object detection unit 402R may recognize a monitored object from the image after the image captured by the camera 40R (i.e., the image generated by the image capture generation function) has been corrected by the image correction unit 404R (corrected image), and identify the location (area) in which the monitored object is visible.

[0231] The image correction unit 404, like the image correction unit 306, performs a predetermined correction on the output (captured image) of the imaging device 40 (camera 40X) and outputs the corrected captured image to the object detection unit 402. This unit includes image correction units 404F, 404B, 404L, and 404R.

[0232] The image correction unit 404F is mounted on the camera 40F and, similar to the image correction unit 306 described above, performs predetermined corrections on the image captured by the camera 40F and outputs the corrected image to the object detection unit 402F.

[0233] Furthermore, camera 40F may output to controller 30 a corrected image, corrected by image correction unit 404F, in addition to the image generated by the image capture generation function. The same may apply to cameras 40B, 40L, and 40R. In this case, the image correction unit 306 of controller 30 may be omitted, and object detection unit 302 may detect the monitored object based on the corrected image input from camera 40X.

[0234] The image correction unit 404B is mounted on the camera 40B and, similar to the image correction unit 306 described above, performs predetermined corrections on the image captured by the camera 40B and outputs the corrected image to the object detection unit 402B.

[0235] The image correction unit 404L is mounted on the camera 40L and, similar to the image correction unit 306 described above, performs predetermined corrections on the image captured by the camera 40L and outputs the corrected image to the object detection unit 402L.

[0236] The image correction unit 404R is mounted on the camera 40R and, similar to the image correction unit 306 described above, performs predetermined corrections on the image captured by the camera 40R and outputs the corrected image to the object detection unit 402R.

[0237] The object detection units 402X use the same algorithm to detect monitored objects from the input image. On the other hand, the object detection unit 402X uses a different algorithm than the object detection unit 302 to detect monitored objects from the input image. This increases the likelihood that the other object detection unit will detect the monitored object even if one of the object detection units 302 or 402X fails to do so. As a result, it is possible to suppress situations where monitored objects around the shovel 100 cannot be detected. Furthermore, even if one of the object detection units 302 or 402X detects a monitored object that does not exist, the other object detection unit can choose not to detect the object because it has not detected it.

[0238] For example, one of the object detection units 302 and 402X may use only the former of image processing technology and machine learning technology, while the other may use both image processing technology and machine learning technology, i.e., a trained model LM, to detect the monitored object.

[0239] Alternatively, for example, the object detection units 302 and 402X may either use a pre-trained SVM with image features of the monitored object as a pre-trained model LM, while the other uses a pre-trained model LM obtained through deep learning to detect the monitored object from the input image.

[0240] Furthermore, the object detection units 302 and 402X may detect monitored objects from the input image using, for example, pre-trained models LM based on different networks (DNNs). For example, one of the object detection units 302 and 402X may use a classification model based on a CNN as the pre-trained model LM, and the other may use a regression model based on a CNN as the pre-trained model LM to detect monitored objects from the input image. Specifically, one of the object detection units 302 and 402X may use a pre-trained model LM based on Faster R-CNN, and the other may use a pre-trained model LM based on YOLO or SSD to detect monitored objects from the input image. Also, for example, the object detection units 302 and 402X may use CNN-based pre-trained models LM, and each pre-trained model LM may differ from the others in at least one of the following: network structure, number of network layers, feature map structure, or candidate region characteristics. Candidate region characteristics may include differences in aspect ratio and whether the candidate region is fixed or variable. Specifically, the object detection units 302 and 402X may detect monitored objects from the input image, with one unit using a YOLO-trained model LM and the other using an SSD-trained model LM. Alternatively, for example, the object detection units 302 and 402X may both use the same SSD-trained model LM, but each trained model LM may differ from the others in at least one of its parameters, such as network structure, number of network layers, feature map structure, and candidate region characteristics.

[0241] Alternatively, for example, the object detection units 302 and 402X may detect monitored objects from the input image using trained models LM that have been machine-trained on different training datasets.

[0242] For example, the object detection units 302 and 402X may detect monitored objects from input images using a trained model LM that has been trained on a training dataset of images in which at least one of the postures and orientations of the monitored objects differs from one another. For example, the trained model LM of the object detection units 302 and 402X may be trained on a training dataset of images in which one of the models shows a person standing upright, and the other model may be trained on a training dataset of images in which the other model shows a person in a crouching or squatting position. Alternatively, for example, the trained models of the object detection units 302 and 402X may be trained on a training dataset of images in which one of the models shows a person facing forward or backward towards the camera, and the other model may be trained on a training dataset of images in which the other model shows a person facing sideways towards the camera. This increases the likelihood that the object detection units 302 and 402X can properly detect a monitored object from the input image even when one of the models cannot properly detect the monitored object due to the influence of the orientation or posture of the monitored object in the input image. Therefore, the object detection units 302 and 402X as a whole can detect monitored objects more reliably.

[0243] Furthermore, for example, the object detection units 302 and 402X may detect a monitored object (person) from an input image by having one of them detect (recognize) the entire shape of the monitored object (person), and the other detect (recognize) the clothing worn by the monitored object (person). For example, the object detection units 302 and 402X may detect a person from an input image by having one of them detect the entire person from the input image, and the other detect (recognize) the helmet or safety vest worn by the person. This allows the object detection units 302 and 402X to more reliably detect the monitored object (person) even if, for some reason, one of them cannot properly detect (recognize) the entire person.

[0244] Furthermore, for example, the object detection units 302 and 402X may detect the monitored object (person) from the input image by detecting different pieces of clothing worn by the monitored object (person) from each other. For example, one of the object detection units 302 and 402X may detect (recognize) a helmet worn by a person from the input image, and the other may detect (recognize) a safety vest worn by a person from the input image, thereby detecting a person from the input image. In this way, even if one of the object detection units 302 and 402X cannot recognize one piece of clothing, for example, because the worker is not wearing one piece of clothing, the other can recognize the other piece of clothing, thereby more reliably detecting the monitored object (person).

[0245] Furthermore, for example, the object detection units 302 and 402X may use pre-trained models LM, which have been trained on training datasets of images with different background types for the monitored objects, to detect monitored objects from input images. For example, one of the pre-trained models LM of the object detection units 302 and 402X may be trained on a training dataset of images with soil or sand in the background, and the other on a training dataset of images with asphalt or other type of ground in the background. This increases the likelihood that the other object detection unit can properly detect the monitored object from the input image even when one of the object detection units 302 and 402X cannot properly detect the monitored object due to the influence of the input image background. As a result, the object detection units 302 and 402X as a whole can more reliably detect monitored objects.

[0246] Furthermore, for example, the object detection units 302 and 402X may each use a trained model LM, which has been machine-trained on a training dataset of images acquired (imaged) at different time periods, to detect monitored objects from the input image. For example, one of the trained models of the object detection units 302 and 402X may be machine-trained on a training dataset of images taken in the morning or midday, while the other is machine-trained on a training dataset of images taken in the evening or nighttime. This allows the controller 30 to, for example, switch between the object detection units 302 and 402X according to the time of day when the shovel 100 is operating, or to switch the method of making the final decision regarding the detection of monitored objects based on the detection results of the object detection units 302 and 402X.

[0247] Furthermore, if the shovel 100 is used at night, for example, the object detection units 302 and 402X may use a trained model LM, which has been machine-trained on a training dataset of images with different nighttime lighting conditions, to detect the monitored object from the input image. For example, the trained model LM of the object detection units 302 and 402X may be machine-trained on a training dataset of images acquired at relatively high illumination for one unit and on a training dataset of images acquired at relatively low illumination for the other unit. Alternatively, the trained model LM of the object detection units 302 and 402X may be machine-trained on a training dataset of images acquired under different color lighting conditions for the other unit. This increases the likelihood that the other unit can properly detect the monitored object even if one unit cannot properly detect it due to the influence of lighting conditions. As a result, the object detection units 302 and 402X as a whole can more reliably detect the monitored object.

[0248] Furthermore, for example, the object detection units 302 and 402X may detect monitored objects from input images using a trained model LM that has been machine-trained on a training dataset of images acquired (imaged) under different weather conditions. For example, the trained model LM of the object detection units 302 and 402X may be machine-trained on a training dataset of images acquired under sunny or cloudy conditions for one of them, and on a training dataset of images acquired under rainy or snowy conditions for the other. This increases the likelihood that the object detection units 302 and 402X can properly detect monitored objects even when one of them cannot properly detect them due to weather conditions. As a result, the object detection units 302 and 402X as a whole can more reliably detect monitored objects.

[0249] Furthermore, for example, the object detection units 302 and 402X may detect monitored objects from input images using a trained model LM that has been machine-trained on a training dataset of images acquired under different positional relationships between light sources and imaging ranges. For example, the trained model LM of the object detection units 302 and 402X may be machine-trained on a training dataset of images corresponding to front lighting, semi-front lighting, or side lighting for one of the models, and on a training dataset of images corresponding to backlighting for the other model. This increases the likelihood that the object detection units 302 and 402X can properly detect monitored objects even if one of them cannot properly detect them due to the positional relationship between the light source and imaging range. As a result, the object detection units 302 and 402X can more reliably detect monitored objects overall.

[0250] The detection and determination unit 307 (an example of a final detection unit) detects the monitored object based on the detection results of the object detection unit 302 and the object detection unit 402X. Specifically, the detection and determination unit 307 makes a final decision regarding the detection of the monitored object based on the detection results of the object detection unit 302 and the object detection unit 402X. Details of the method by which the detection and determination unit 307 detects the monitored object will be described later.

[0251] The position estimation unit 303 estimates the actual location of a monitored object when the detection and determination unit 307 detects one. Furthermore, if the detection and determination unit 307 detects multiple monitored objects, the position estimation unit 303 estimates the actual location of each of the multiple monitored objects.

[0252] For example, if a monitored object is detected by either the object detection unit 302 or 402X, the position of the monitored object is estimated by arbitrarily applying the above method based on the detection range on the input image identified by either of them.

[0253] Furthermore, for example, if the same monitored object is detected by both object detection units 302 and 402X, the actual location of the monitored object may be estimated for each of the detection results from object detection units 302 and 402X, and the actual location of the monitored object may be estimated from these respective estimation results. For example, the actual location of the detected monitored object may be estimated by taking the average of the estimated results of the actual location of the monitored object for each of the detection results from object detection units 302 and 402X.

[0254] The safety control unit 304 (an example of a control unit) activates the safety function when, for example, the detection and determination unit 307 detects a monitored object within a predetermined range around the shovel 100. Specifically, the safety control unit 304 may activate the safety function when the actual location (estimated value) of the monitored object, estimated by the position estimation unit 303, is within a predetermined range around the shovel 100.

[0255] Furthermore, object detection units 302, 402 X Either of these functions may be transferred to the display device 50A. Also, object detection units 302, 402 XIn addition, one or more object detection units for detecting a monitoring object from the input image may be provided on the excavator 100. The additional object detection unit may be provided in the display device 50A, may be provided in the controller 30, or may be provided in a controller different from the controller 30. In this case, three or more object detection units including the additional ones may detect the monitoring object from the input image using different algorithms from each other.

[0256] [Specific example of object detection method] Next, a method for detecting a monitoring object by the detection determination unit 307, that is, a method for making a final determination regarding the detection of a monitoring object will be described.

[0257] For example, when a monitoring object is detected by at least one of the object detection unit 302 and the object detection unit 402X, the detection determination unit 307 may determine that the monitoring object exists around the excavator 100 and detect the monitoring object. Thereby, for example, even if the controller 30 fails to detect a monitoring object that actually exists by either one of the object detection units 302 and 402X, if the other one detects the monitoring object, the controller 30 can detect the monitoring object. Therefore, for example, in a situation where detection failure of a monitoring object is likely to occur or in a situation where there is a high need to prevent detection failure of a monitoring object, the controller 30 can more reliably detect an actually existing monitoring object, and as a result, the safety can be further improved along with the operation of the safety function of the excavator 100.

[0258] Further, for example, when the monitoring object is detected by both the object detection unit 302 and the object detection unit 402X, the detection determination unit 307 may determine that the monitoring object exists around the excavator 100 and detect the monitoring object. Thereby, for example, even if a non-existent monitoring object is erroneously detected by either one of the object detection units 302 and 402X, the controller 30 can prevent detecting the monitoring object if the monitoring object is not detected by the other one. Therefore, the controller 30 can suppress the erroneous detection of the monitoring object, for example, in a situation where the erroneous detection of the monitoring object is likely to occur or in a situation where it is highly necessary to prevent the reduction of work efficiency due to the erroneous detection of the monitoring object.

[0259] In addition, as described above, when three or more object detection units are provided in the excavator 100, the detection determination unit 307 may detect the monitoring object, for example, when the monitoring object is detected by a predetermined number (an integer of 2 or more) or more of all the object detection units. The predetermined number may be fixed or variable. For example, the predetermined number may be varied according to a predetermined input from a user (operator or monitor) received through the input device 52 or the communication device 60.

[0260] Further, the detection determination unit 307 may switch the above two methods according to a predetermined input from a user (operator or monitor) received through the input device 52 or the communication device 60 (both are examples of reception devices).

[0261] In addition, for example, the detection determination unit 307 may detect the monitoring object based on the object detection unit 302 and the object detection unit 402X by using the importance defined by the respective detection results of the object detection unit 302 and the object detection unit 402X. Specifically, the detection determination unit 307 detects the monitoring object when the monitoring object is detected by the one with a relatively higher importance of the detection results between the object detection unit 302 and the object detection unit 402X.

[0262] The importance level may be set (variable) according to predetermined inputs from the user (operator or monitor) received, for example, through the input device 52 or the communication device 60. This allows the user to compare, for example, the surrounding image displayed on the display device 50A or the remote control display device with the detection status of the monitored object by the actual object detection units 302 and 402X, and set the importance level of the detection result that is considered to be more accurate. The importance level may also be set (variable) automatically according to the surrounding conditions of the shovel 100. For example, if there is a difference in the detection accuracy of the monitored object between the object detection units 302 and 402X depending on the environmental conditions in which the shovel 100 is placed, the importance level of the detection result with higher detection accuracy may be set higher than the importance level of the other detection result. The environmental conditions in which the shovel 100 is placed may include the condition of objects around the shovel 100 that are captured as background images by the camera 40X (for example, the type of soil, asphalt, etc. on the ground). Furthermore, the environmental conditions in which the shovel 100 is placed may include, for example, the time of day when work is being performed, the lighting conditions during nighttime work, the weather conditions, and the positional relationship between the light source and the imaging range of the camera 40X. This allows the controller 30 to set the importance of either the object detection unit 302 or 402X, which is assumed to have higher detection accuracy, to a relatively higher level, according to the situation in which the shovel 100 is placed.

[0263] Furthermore, as described above, if the shovel 100 is equipped with three or more object detection units, the detection and judgment unit 307 may detect the monitored object in such a way that the higher the relative importance of the detection result of each object detection unit, the more likely it is that the detection result will be reflected in the final judgment. For example, if a monitored object has been detected by some of the object detection units, the detection and judgment unit 307 may detect the monitored object if the sum of the importance of the detection results of those some object detection units is greater than the sum of the importance of the detection results of the remaining object detection units. Also, for example, if a monitored object has been detected by some of the object detection units, the detection and judgment unit 307 may detect the monitored object if the sum of the importance of the detection results of those some object detection units is greater than or equal to a predetermined threshold.

[0264] [Transformation / Modification] Although embodiments have been described in detail above, this disclosure is not limited to these specific embodiments, and various modifications and changes are possible within the scope of the gist described in the claims.

[0265] For example, in the embodiment described above, a method for detecting a monitored object based on images captured by an imaging device 40 mounted on a shovel 100 was described, but a similar method may be applied to a method for detecting a monitored object based on images captured by an imaging device mounted on other work machines. Other work machines include, for example, a lifting magnet machine in which a lifting magnet is attached to the tip of the attachment AT in place of a bucket 6, a bulldozer, forestry machinery (e.g., a harvester), road machinery (e.g., an asphalt finisher), etc. [Explanation of Symbols]

[0266] 1. Lower running body 1C Crawler 1ML Hydraulic Motor for Travel 1MR Hydraulic Motor for Travel 2. Swivel mechanism 2M Swivel Hydraulic Motor 3. Upper rotating body 4 Boom 5 Arms 6 buckets 7 Boom Cylinder 8 Arm Cylinder 9 Bucket Cylinder 10 cabins 11 Engine 13 Regulator 14 Main pump 15 Pilot pump 17 Control valve 25 Pilot Line 25A Pilot Line 25B Pilot Line 26 Operating device 27 Pilot Line 27A Pilot Line 27B Pilot Line 29 Operating Pressure Sensor 30 Controller (Control Device) 30A Auxiliary Memory Device 30B Memory Device 30C CPU 30D Interface Device 31 Hydraulic Control Valve 32 Shuttle Valve 33 Hydraulic Control Valve 40 Imaging Device 40B Camera 40F Camera 40L Camera 40R Camera 50 Output Device 50A Display Device 50B Sound Output Device 52 Input Device (Reception Device) 60 Communication Device (Reception Device) 70 Irradiation Device 100 Excavator 200 Management Device 201 External Interface 201A Recording Medium 202 Auxiliary Memory Device 203 Memory Device 204 CPU 205 High-Speed Arithmetic Unit 206 Communication Interface 207 Input Device 208 Display Device 301 Display Processing Unit 302 Object Detection Unit (Detection Unit) 303 Position Estimation Unit 304 Safety Control Unit (Control Unit) 305 Irradiation Control Unit 306 Image Correction Unit 307 Detection Judgment Unit (Final Detection Unit) 402,402B,402F,402L,402R Object Detection Unit (Detection Unit) 404,404B,404F,404L,404R Image Correction Unit AT attachment HA Hydraulic Actuator HMT Helmet LM pre-trained model Network communication line RV High-Visibility Safety Clothing S1 Boom Angle Sensor S2 Arm Angle Sensor S3 Bucket Angle Sensor S4 Aircraft attitude sensor S5 Swivel Angle Sensor SYS Shovel Management System W, W1~W3 Workers

Claims

1. Lower running body and An upper slewing body is mounted on the lower traveling body so as to be rotatable, An imaging device mounted on the upper rotating body for imaging the area around the upper rotating body, Each unit comprises a plurality of detection units that detect predetermined objects around the shovel using different algorithms based on the captured image from the imaging device, In the algorithms of one of the plurality of detection units and the other detection units, the predetermined object is detected using machine learning models that include different neural networks and are based on training data images in which the predetermined object is depicted. The aforementioned distinct neural networks are neural networks with different structures. Shovel.

2. Lower running body and An upper slewing body is mounted on the lower traveling body so as to be rotatable, An imaging device mounted on the upper rotating body for imaging the area around the upper rotating body, Each of the following detection units detects a predetermined object around the shovel using a different algorithm based on the image captured by the imaging device: The system includes a final detection unit that detects a predetermined object in the vicinity of the shovel based on the detection results of the plurality of detection units, The importance level is defined for each of the detection results from the aforementioned plurality of detection units. The final detection unit detects the predetermined object when, in the event that the predetermined object has been detected by some of the plurality of detection units, the sum of the importance values ​​of the detection results of the some detection units is greater than the sum of the importance values ​​of the detection results of the remaining detection units, or when the sum of the importance values ​​of the some detection units is equal to or greater than a predetermined threshold. Shovel.

3. Lower traveling body and An upper slewing body is mounted on the lower traveling body so as to be rotatable, An imaging device mounted on the upper rotating body for imaging the area around the upper rotating body, Each of the following detection units detects a predetermined object around the shovel using a different algorithm based on the image captured by the imaging device: The system includes a final detection unit that detects a predetermined object in the vicinity of the shovel based on the detection results of the plurality of detection units, In the algorithms of one of the plurality of detection units and the other detection units, the predetermined object is detected using machine learning models that include different neural networks and are based on training data images in which the predetermined object is depicted. Shovel.

4. Based on the detection results of the plurality of detection units, a final detection unit is provided to detect the predetermined object in the vicinity of the shovel. The shovel according to claim 1.

5. The final detection unit detects the predetermined object when the same predetermined object has been detected by two or more predetermined detection units among the plurality of detection units. A shovel according to any one of claims 2 to 4.

6. The final detection unit, if the predetermined object has been detected by at least one of the plurality of detection units, will detect the predetermined object. A shovel according to any one of claims 2 to 4.

7. Equipped with a reception device that accepts input from users, The final detection unit switches between a state in which it detects the predetermined object when the same predetermined object is detected by two or more predetermined detection units among the plurality of detection units, and a state in which it detects the predetermined object when the predetermined object is detected by at least one of the plurality of detection units, in response to a predetermined input received through the receiving device. A shovel according to any one of claims 2 to 4.

8. The importance level is defined for each of the detection results from the aforementioned plurality of detection units. The final detection unit detects the predetermined object when, in the event that the predetermined object has been detected by some of the plurality of detection units, the sum of the importance values ​​of the detection results of the some detection units is greater than the sum of the importance values ​​of the detection results of the remaining detection units, or when the sum of the importance values ​​of the some detection units is equal to or greater than a predetermined threshold. The shovel according to claim 3 or 4.

9. The aforementioned specified object is a person. A shovel according to any one of claims 1 to 8.

10. In the algorithm of one of the multiple detection units, a person is detected by recognizing the overall characteristics, including the shape of the person. In the algorithm of one of the multiple detection units, a person is detected by recognizing the person's clothing. The shovel according to claim 9.

11. The aforementioned equipment is a helmet or high-visibility safety clothing. The shovel according to claim 10.

12. In the algorithms of one of the plurality of detection units and the other detection units, the predetermined object is detected using a machine learning model based on training data images in which the predetermined object has at least one of different orientations and orientations. A shovel according to any one of claims 1 to 8.

13. In the algorithms of one of the plurality of detection units and the other detection units, the predetermined object is detected using a machine-learned model based on training data images in which the predetermined object is depicted, where at least one of the following is different: the type of background behind the predetermined object, the time of day when the image was taken, the nighttime lighting conditions when the image was taken, and the weather conditions when the image was taken. A shovel according to any one of claims 1 to 8.

14. The aforementioned distinct neural networks are neural networks with different structures. The shovel according to claim 1, 3, or 4.

15. The aforementioned distinct neural networks are neural networks that have been machine-trained using the aforementioned distinct training data. The shovel according to claim 1, 3, 4, or 14.

16. When the final detection unit detects the predetermined object, the unit includes a control unit that slows down or stops the movement of the shovel. A shovel according to any one of claims 2 to 8.

17. A control device that controls the shovel, The system includes a display device that displays the captured image or a processed image generated based on the captured image, The plurality of detection units are distributed and mounted on at least two of the imaging device, the control device, and the display device. A shovel according to any one of claims 1 to 16.

18. The control device includes a control device that performs image processing related to the captured image. The shovel according to claim 17.

Images