Shovel and shovel control system

Abstract

To improve safety.SOLUTION: A shovel comprises: a lower traveling body; an upper revolving body that is rotatably mounted on the lower traveling body; a space recognition device fitted to the upper revolving body; a display device; and a control device constituted so as to display positional information indicating a positional relation between a human detected by the space recognition device and the shovel on the display device, and continue to display the position of the human who is no longer detected, in the positional information, if prescribed conditions are satisfied when the position of the human indicated in the positional information is no longer detected by the space recognition device.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an excavator and a control system for an excavator.

Background Art

[0002] Conventionally, a technique has been proposed for detecting an object existing around an excavator and monitoring the surroundings of the excavator (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Patent Document 1 describes a technique for detecting an object existing around a work machine using a stereo camera. However, the detection of an object using a detection device such as a stereo camera may not be appropriately performed depending on the surrounding environment and the like.

[0005] One aspect of the present invention proposes a technique for improving safety by continuing to display the position of a person when a person is no longer detected by a space recognition device and a predetermined condition is satisfied.

Means for Solving the Problems

[0006] An excavator according to one aspect of the present invention includes a lower traveling body, an upper revolving body rotatably mounted on the lower traveling body, a space recognition device attached to the upper revolving body, a display device, and a control device configured to display on the display device position information representing the relationship between the positions of each person and the excavator detected by the space recognition device, and to continue displaying on the position information of the position of the person who has disappeared from detection by the space recognition device when a predetermined condition is satisfied.

Advantages of the Invention

[0007] According to one aspect of the present invention, by displaying that there is a person around the excavator, improvement in safety is realized.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, the embodiments described below are examples and do not limit the invention, and not all features and combinations thereof described in the embodiments are necessarily essential to the invention. In each drawing, the same or corresponding components are denoted by the same or corresponding reference numerals, and the description thereof may be omitted.

[0010] Hereinafter, in the embodiments of the present invention, an example in which an excavator is used as an example of a working machine will be described, but the present invention is not limited to excavators. It may be applied to construction machines, standard machines, application machines, forestry machines, or transport machines based on hydraulic excavators.

[0011] (First Embodiment) First, with reference to FIG. 1, an overview of the excavator 100 according to the present embodiment will be described. FIG. 1 is a side view of the excavator 100 according to the first embodiment. FIG. 2 is a top view of the excavator 100 according to the first embodiment.

[0012] The upper revolving body 3 is rotatably mounted on the lower traveling body 1 of the excavator 100 via a slewing mechanism 2. A boom 4 is attached to the upper revolving body 3. An arm 5 is attached to the tip of the boom 4, and a bucket 6 as an end attachment is attached to the tip of the arm 5. The end attachment may be a slope bucket, a dredging bucket, or the like.

[0013] The boom 4, arm 5, and bucket 6 constitute an excavation attachment which is an example of the attachment AT, and are respectively hydraulically driven by the boom cylinder 7, arm cylinder 8, and bucket cylinder 9. A boom angle sensor S1 is attached to the boom 4, an arm angle sensor S2 is attached to the arm 5, and a bucket angle sensor S3 is attached to the bucket 6. The excavation attachment may be provided with a bucket tilt mechanism.

[0014] The boom angle sensor S1 detects the rotation angle of the boom 4. In the present embodiment, the boom angle sensor S1 is an acceleration sensor and can detect the boom angle which is the rotation angle of the boom 4 with respect to the upper swing body 3. The boom angle becomes the minimum angle when the boom 4 is lowered most, and increases as the boom 4 is raised.

[0015] The boom angle sensor S1 may include, for example, a rotary encoder, an acceleration sensor, a six-axis sensor, an IMU (Inertial Measurement Unit), etc. Further, the boom angle sensor S1 may include a potentiometer using a variable resistor, a cylinder stroke sensor that detects the stroke amount of a hydraulic cylinder (boom cylinder 7) corresponding to the boom angle, etc. The same applies to the arm angle sensor S2, bucket angle sensor S3, and body tilt sensor S4 below. The detection signal corresponding to the boom angle by the boom angle sensor S1 is taken into the controller 30.

[0016] The arm angle sensor S2 detects the rotation angle of the arm 5. In the present embodiment, the arm angle sensor S2 is an acceleration sensor and can detect the arm angle which is the rotation angle of the arm 5 with respect to the boom 4. The arm angle becomes the minimum angle when the arm 5 is closed most, and increases as the arm 5 is opened.

[0017] The bucket angle sensor S3 detects the rotation angle of the bucket 6. In this embodiment, the bucket angle sensor S3 is an acceleration sensor and can detect the bucket angle, which is the rotation angle of the bucket 6 with respect to the arm 5. The bucket angle, for example, becomes the minimum angle when the bucket 6 is closed most, and increases as the bucket 6 is opened.

[0018] The boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3 may be a potentiometer using a variable resistor, a stroke sensor that detects the stroke amount of the corresponding hydraulic cylinder, a rotary encoder that detects the rotation angle around the connecting pin, or the like. The boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3 constitute an attitude sensor that detects the attitude of the excavation attachment.

[0019] The upper swing body 3 is provided with a cabin 10 which is a driver's cab and is equipped with a power source such as an engine 11. In addition, a machine body tilt sensor S4, a swing angle sensor S5, and an imaging device S6 are attached to the upper swing body 3. Further, a communication device T1 and a positioning device PS are attached to the upper swing body 3.

[0020] The machine body tilt sensor S4 is configured to detect the tilt of the upper swing body 3 with respect to a predetermined plane. In this embodiment, the machine body tilt sensor S4 is an acceleration sensor that detects the tilt angle around the front-rear axis and the tilt angle around the left-right axis of the upper swing body 3 with respect to the horizontal plane. The front-rear axis and the left-right axis of the upper swing body 3, for example, are orthogonal to each other and pass through the excavator center point which is a point on the swing axis of the excavator 100.

[0021] The swing angle sensor S5 is configured to detect the swing angular velocity of the upper swing body 3. In this embodiment, the swing angle sensor S5 is a gyro sensor. The swing angle sensor S5 may be a resolver or a rotary encoder or the like. The swing angle sensor S5 may detect the swing speed. The swing speed may be calculated from the swing angular velocity.

[0022] In addition, when the body tilt sensor S4 includes a gyro sensor, a 6-axis sensor, an IMU, etc. that can detect the angular velocity around three axes, the turning state (for example, the turning angular velocity) of the upper swing body 3 may be detected based on the detection signal of the body tilt sensor S4. In this case, the turning angle sensor S5 may be omitted.

[0023] The imaging device S6 is an example of a space recognition device and is configured to acquire an image of the periphery of the excavator 100. In the present embodiment, the imaging device S6 includes a front camera S6F that images the space in front of the excavator 100, a left camera S6L that images the space to the left of the excavator 100, a right camera S6R that images the space to the right of the excavator 100, and a rear camera S6B that images the space behind the excavator 100.

[0024] The imaging device S6 is, for example, a monocular camera having an imaging element such as a CCD or a CMOS, and outputs the captured image to the display device D3 via the controller 30.

[0025] The input device D2 receives an operation input from the operator and outputs it to the controller 30. The input device D2 includes, for example, any hardware operation means such as a touch panel, a touch pad, buttons, toggles, rotary knobs, etc. Further, the input device D2 may include software operation means that can be operated through hardware operation means such as virtual button icons on an operation screen displayed on the display device D3 or the like.

[0026] As shown in FIG. 2, the front camera S6F is, for example, attached to the roof of the cab 10. The left camera S6L is attached to the left end of the upper surface of the upper swing body 3. The right camera S6R is attached to the right end of the upper surface of the upper swing body 3. The rear camera S6B is attached to the rear end of the upper surface of the upper swing body 3.

[0027] The front camera S6F, the rear camera S6B, the left camera S6L, and the right camera S6R are all attached to the upper swing body 3 such that their optical axes face obliquely downward and a part of the upper swing body 3 is included in the imaging range. Therefore, each imaging range of the front camera S6F, the rear camera S6B, the left camera S6L, and the right camera S6R has a viewing angle of, for example, approximately 180 degrees in a top view. In the example of FIG. 2, the imaging range AF represents an example of the imaging range of the front camera S6F, the imaging range AB represents an example of the imaging range of the rear camera S6B, the imaging range AL represents an example of the imaging range of the left camera S6L, and the imaging range AR represents an example of the imaging range of the right camera S6R. The four monocular cameras are preferably attached to the upper swing body 3 so as not to protrude from the upper surface of the upper swing body 3 as shown in FIG. 2.

[0028] In this embodiment, by providing the imaging device S6 in the above-described arrangement, an object existing around the excavator 100 can be imaged.

[0029] The positioning device PS is configured to acquire information regarding the position of the excavator 100. In this embodiment, the positioning device PS is configured to measure the position and orientation of the excavator 100. Specifically, the positioning device PS is a GNSS receiver incorporating an electronic compass, which measures the latitude, longitude, and altitude of the current position of the excavator 100 and also measures the orientation of the excavator 100.

[0030] FIG. 3 is a diagram showing a configuration example of the drive control system of the excavator 100 in FIG. 1. In FIG. 3, the mechanical power transmission system is shown by a double line, the hydraulic oil line is shown by a thick solid line, the pilot line is shown by a broken line, and the electric drive / control system is shown by a thin solid line, respectively.

[0031] The engine 11 is a power source of the excavator 100. In this embodiment, the engine 11 is a diesel engine that employs isochronous control to maintain a constant engine speed regardless of the increase or decrease of the engine load. The fuel injection amount, fuel injection timing, boost pressure, etc. in the engine 11 are controlled by an engine control unit (ECU) D7.

[0032] The rotating shafts of the main pump 14 and the pilot pump 15, each serving as a hydraulic pump, are connected to the rotating shaft of the engine 11. The control valve unit 17 is connected to the main pump 14 via a hydraulic oil line.

[0033] The control valve unit 17 is a hydraulic control device that controls the hydraulic system of the excavator 100. Hydraulic actuators such as the left and right travel hydraulic motors, the boom cylinder 7, the arm cylinder 8, the bucket cylinder 9, and the swing hydraulic motor are connected to the control valve unit 17 via hydraulic oil lines. The swing hydraulic motor may be a swing electric generator.

[0034] Figure 3 shows the connection relationship between the controller 30 and the display device D3. In this embodiment, the display device D3 is connected to the controller 30. The display device D3 and the controller 30 may be connected via a communication network such as CAN. In this embodiment, an example in which one display device D3 is provided will be described, but the number of display devices provided in the cab 10 is not limited, and a plurality of them may be provided.

[0035] The display device D3 includes a control unit D3a that generates an image. In this embodiment, the control unit D3a generates a display camera image based on the output of the camera as the imaging device S6. The imaging device S6 is connected to the display device D3 via a dedicated line, for example.

[0036] The control unit D3a generates a display image based on the output of the controller 30. In this embodiment, the control unit D3a converts various information output by the controller 30 into an image signal. The information output by the controller 30 includes, for example, data indicating the temperature of the engine coolant, data indicating the temperature of the hydraulic oil, data indicating the remaining fuel amount, data indicating the remaining amount of urea water, data indicating the position of the working part of the bucket 6, data indicating the orientation of the working surface of the work target, data indicating the orientation of the excavator 100, data indicating the operation direction for aligning the excavator 100 with the working surface, etc.

[0037] The control unit D3a may be implemented as a function of the controller 30 instead of a function of the display device D3. In this case, the imaging device S6 is connected to the controller 30 instead of the display device D3.

[0038] The display device D3 operates by receiving power supply from the storage battery 70. The storage battery 70 is charged with the power generated by the alternator 11a (generator) of the engine 11. The power of the storage battery 70 is supplied to the electrical components 72 of the excavator 100 etc. in addition to the controller 30 and the display device D3. The starter 11b of the engine 11 is driven by the power from the storage battery 70 to start the engine 11.

[0039] The engine 11 is controlled by the engine controller unit D7. Various data indicating the state of the engine 11 are constantly transmitted from the engine controller unit D7 to the controller 30. The various data indicating the state of the engine 11 are an example of the operation information of the excavator 100, and include, for example, data indicating the cooling water temperature detected by the water temperature sensor 11c as an operation information acquisition unit. The controller 30 accumulates this data in the temporary storage unit (memory) 30a and can transmit it to the display device D3 when necessary.

[0040] The following various data are supplied to the controller 30 as the operation information of the excavator 100 and stored in the temporary storage unit 30a of the controller 30.

[0041] For example, data indicating the swash plate tilt angle is supplied from the regulator 13 of the main pump 14, which is a variable displacement hydraulic pump, to the controller 30. Also, data indicating the discharge pressure of the main pump 14 is supplied from the discharge pressure sensor 14b to the controller 30. These data are stored in the temporary storage unit 30a. Further, an oil temperature sensor 14c is provided in the pipeline between the tank storing the hydraulic oil sucked by the main pump 14 and the main pump 14, and data representing the temperature of the hydraulic oil flowing through the pipeline is supplied from the oil temperature sensor 14c to the controller 30. The regulator 13, the discharge pressure sensor 14b, and the oil temperature sensor 14c are examples of the operation information acquisition unit.

[0042] The operating device 26 is provided near the driver's seat in the cabin 10 and is used for the operator to operate various driven elements. Specifically, the operating device 26 is used for the operator to operate hydraulic actuators such as the left and right traveling hydraulic motors, the boom cylinder 7, the arm cylinder 8, the bucket cylinder 9, and the slewing hydraulic motor. As a result, the operator can operate the driven elements driven by the hydraulic actuators. The operating device 26 includes a pedal device and a lever device for operating the respective driven elements.

[0043] The operation sensor 29 is configured to detect the operation content of the operator using the operating device 26. In the present embodiment, the operation sensor 29 detects the operation direction and operation amount of the operating device 26 corresponding to each hydraulic actuator, and outputs an electrical signal (hereinafter also referred to as an operation signal) corresponding to the detected value to the controller 30. In the present embodiment, the controller 30 controls the opening area of the proportional valve 31 according to the output of the operation sensor 29. Then, the controller 30 supplies the hydraulic oil discharged by the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17. The pressure (pilot pressure) of the hydraulic oil supplied to each pilot port is, in principle, a pressure corresponding to the operation direction and operation amount of the operating device 26 corresponding to each hydraulic actuator. In this way, the operating device 26 is configured to be able to supply the hydraulic oil discharged by the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17. Thereby, the hydraulic actuator can be driven.

[0044] Also, the direction switching valve for driving each hydraulic actuator built in the control valve unit 17 may be an electromagnetic solenoid type. In this case, the operation signal output from the operating device 26 may be directly input to the control valve unit 17 (that is, to the electromagnetic solenoid type direction switching valve).

[0045] Note that the operating device 26 may be a hydraulic pilot type. Specifically, the operating device 26 uses the hydraulic oil supplied from the pilot pump 15 through the pilot line, and outputs a pilot pressure corresponding to the operation content to the secondary pilot line. Then, the secondary pilot line is connected to the control valve unit 17. Thereby, a pilot pressure corresponding to the operation content regarding various driven elements (hydraulic actuators) in the operating device 26 can be input to the control valve unit 17. Therefore, the control valve unit 17 can drive each hydraulic actuator according to the operation content of the operating device 26 by an operator or the like. In this case, an operation sensor 29 capable of acquiring information regarding the operation state of the operating device 26 is provided, and the output of the operation sensor 29 is taken into the controller 30. Thereby, the controller 30 can grasp the operation state of the operating device 26. The operation sensor 29 is, for example, a pressure sensor that acquires information regarding the pilot pressure (operation pressure) of the secondary pilot line of the operating device 26.

[0046] Further, part or all of the hydraulic actuators may be replaced with electric actuators. In this case, for example, the controller 30 may output an operation command corresponding to the operation content of the operating device 26 or the content of the remote operation defined by the remote operation signal to the electric actuator or a driver or the like that drives the electric actuator. Also, when an operation signal is input from the operating device 26 to the electric actuator or a driver or the like, the electric actuator may be configured to be operable by the operating device 26.

[0047] Also, when the excavator 100 is exclusively remotely operated or exclusively operates by a fully automatic operation function, the operating device 26 may be omitted.

[0048] The proportional valve 31 functions as a control valve for machine control, and is provided for each driven element (hydraulic actuator) to be operated by the operating device 26 and for each operating direction (for example, the raising direction and the lowering direction of the boom 4) of the driven element (hydraulic actuator). For example, two proportional valves 31 are provided for each double-acting hydraulic actuator for driving the lower traveling body 1, the upper swing body 3, the boom 4, the arm 5, the bucket 6, and the like. The proportional valve 31 is provided, for example, in a pilot line between the pilot pump 15 and the control valve unit 17, and may be configured to be able to change its flow passage area (that is, the cross-sectional area through which the hydraulic oil can flow). Thereby, the proportional valve 31 can output a predetermined pilot pressure to the secondary pilot line by using the hydraulic oil of the pilot pump 15 supplied through the primary pilot line. Therefore, the proportional valve 31 can apply a predetermined pilot pressure according to an operation command from the controller 30 to the control valve unit 17. Thus, for example, the controller 30 can directly supply a pilot pressure corresponding to the operation content (operation signal) of the operating device 26 from the proportional valve 31 to the control valve unit 17, and realize the operation of the excavator 100 based on the operation of the operator.

[0049] Further, the controller 30 may control the proportional valve 31 to realize the automatic operation function of the excavator 100. Specifically, the controller 30 outputs an operation command corresponding to the automatic operation function to the proportional valve 31 from the proportional valve 31. Thereby, the controller 30 can realize the operation of the excavator 100 by the automatic operation function.

[0050] Further, the controller 30 controls the proportional valve 31 to realize the remote operation of the excavator 100. Specifically, the controller 30 outputs an operation command corresponding to the content of the operation specified by the operation signal received from the remote operation room RC to the proportional valve 31 by the communication device T1. Thereby, the controller 30 can supply a pilot pressure corresponding to the content of the remote operation from the proportional valve 31 to the control valve unit 17, and realize the operation of the excavator 100 based on the remote operation of the operator.

[0051] In addition, when the operating device 26 is a hydraulic pilot type, a shuttle valve may be provided in the pilot line between the operating device 26 and the proportional valve 31 and the control valve unit 17. The shuttle valve has two inlet ports and one outlet port, and outputs the hydraulic oil having the higher pilot pressure among the pilot pressures input to the two inlet ports to the outlet port. The shuttle valve is provided for each driven element (hydraulic actuator) to be operated by the operating device 26 and for each operating direction of the driven element (hydraulic actuator), similar to the proportional valve 31. For example, two shuttle valves are provided for each double-acting hydraulic actuator for driving the lower traveling body 1, the upper swing body 3, the boom 4, the arm 5, the bucket 6, and the like. One of the two inlet ports of the shuttle valve is connected to the secondary pilot line of the operating device 26 (specifically, the above-described lever device or pedal device included in the operating device 26), and the other is connected to the secondary pilot line of the proportional valve 31. The outlet port of the shuttle valve is connected to the pilot port of the corresponding direction switching valve of the control valve unit 17 through the pilot line. The corresponding direction switching valve is the direction switching valve that drives the hydraulic actuator that is the operation target of the above-described lever device or pedal device connected to one inlet port of the shuttle valve. Therefore, each of these shuttle valves can apply the higher one of the pilot pressure in the secondary pilot line of the operating device 26 and the pilot pressure in the secondary pilot line of the proportional valve 31 to the pilot port of the corresponding direction switching valve. That is, the controller 30 can control the corresponding direction switching valve without depending on the operation of the operator on the operating device 26 by outputting a pilot pressure higher than the pilot pressure on the secondary side of the operating device 26 from the proportional valve 31. Therefore, the controller 30 can control the operation of the driven elements (lower traveling body 1, upper swing body 3, boom 4, arm 5, bucket 6) without depending on the operation state of the operator on the operating device 26, and can realize an automatic operation function and a remote operation function.

[0052] Also, when the operating device 26 is a hydraulic pilot type, in addition to the shuttle valve, a pressure reducing valve may be provided in the pilot line between the operating device 26 and the shuttle valve. The pressure reducing valve operates, for example, in response to a control signal input from the controller 30 and is configured to be able to change its flow passage area. Thereby, when the operating device 26 is being operated by the operator, the controller 30 can forcibly reduce the pilot pressure output from the operating device 26. Therefore, even when the operating device 26 is being operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator corresponding to the operation of the operating device 26. Further, the controller 30 can, for example, even when the operating device 26 is being operated, reduce the pilot pressure output from the operating device 26 with the pressure reducing valve to be lower than the pilot pressure output from the proportional valve 31. Therefore, by controlling the proportional valve 31 and the pressure reducing valve, the controller 30 can surely cause a desired pilot pressure to act on the pilot port of the direction switching valve in the control valve unit 17, for example, regardless of the operation content of the operating device 26. Thus, the controller 30 can more appropriately realize the automatic operation function and the remote operation function of the excavator 100, for example, by controlling the pressure reducing valve in addition to the proportional valve 31.

[0053] The communication system of the excavator 100 according to the present embodiment includes a communication device T1.

[0054] The communication device T1 is connected to an external communication line and communicates with a device provided separately from the excavator 100. The device provided separately from the excavator 100 may include, in addition to the devices outside the excavator 100, a portable terminal device (portable terminal) brought into the cab 10 by the user of the excavator 100. The communication device T1 may include, for example, a mobile communication module compliant with standards such as 4G (4th Generation) and 5G (5th Generation). Further, the communication device T1 may include, for example, a satellite communication module. Further, the communication device T1 may include, for example, a Wi-Fi communication module, a Bluetooth (registered trademark) communication module, or the like. Also, when there are a plurality of connectable communication lines, the communication device T1 may include a plurality of communication devices T1 according to the type of the communication line.

[0055] For example, the communication device T1 communicates with an external device such as a remote control room in the work site through a local communication line constructed in the work site. The local communication line is, for example, a mobile communication line by local 5G (so-called local 5G) constructed in the work site or a local network by Wi-Fi.

[0056] Further, the communication device T1 is configured to transmit and receive information to and from a communication device installed in the remote control room through a wide-area communication line including the work site, that is, a wide-area network.

[0057] In the present embodiment, a case will be described in which the engine 11 is used as a drive source and the hydraulic pump is operated by the driving force generated by the engine 11 to perform the operation of the attachment AT, the turning operation of the upper swing body 3, and the traveling. However, the present embodiment does not limit the drive source to the engine 11, and a motor may be used as the drive source. That is, the control described in the present embodiment may be applied to a so-called electric excavator in which the motor as the drive source is driven by the electric power supplied from the battery, or may be applied to an excavator equipped with a plurality of drive sources.

[0058] <Outline of processing performed by the controller> The controller 30 according to the present embodiment displays information regarding the position of a person detected from the imaging image information captured by the imaging device S6. For this reason, the controller 30 according to the present embodiment performs a detection process to determine whether a person exists around the excavator 100 by using the imaging image information captured by the imaging device S6.

[0059] However, there are situations where a person cannot be detected from the imaging image information even though a person exists around the excavator 100. For example, since the person is too close to the excavator 100, there is a situation where no person exists within the imaging range of the imaging device S6, and thus a person cannot be detected from the imaging image information. Furthermore, even though there is a person within the imaging range of the imaging device S6, due to light reflection or the setting sun, etc., white blooming occurs, and there is a situation where the person does not appear to be captured. Furthermore, even though there is a person within the imaging range of the imaging device S6, since the light amount is low and it becomes blacked out, there is a situation where the person does not appear to be captured.

[0060] In such a situation, although the person is not detected, there is a high possibility that a person exists. Therefore, when a person is no longer detected by the imaging device S6 (an example of a space recognition device), the controller 30 according to the present embodiment continues to display the position of the person who has become undetected if a predetermined condition is satisfied.

[0061] <Block Configuration of the Excavator Controller> FIG. 4 is a functional block diagram showing a configuration example of the controller 30 of the excavator 100 according to the present embodiment. In the example shown in FIG. 4, the block configuration of the controller 30 of the excavator 100 is shown.

[0062] The controller 30 receives information output by the boom angle sensor S1, arm angle sensor S2, bucket angle sensor S3, machine body inclination sensor S4, slewing angle sensor S5, imaging device S6, input device D2, communication device T1, positioning device PS, etc. Then, based on the received information and the information stored in the auxiliary storage device D4, various calculations are executed, and the calculation results are output to the display device D3, proportional valve 31, etc.

[0063] Note that in this embodiment, an example in which the controller 30 controls the excavator 100 will be described. However, a part of the functions of the controller 30 may be realized by another controller (control device). That is, the functions of the controller 30 may be distributed and realized by a plurality of controllers mounted on the excavator 100.

[0064] The excavator 100 operates an actuator (for example, a hydraulic actuator) according to the operation of an operator boarding the cab 10, and drives operating elements (hereinafter, "driven elements") such as the lower traveling body 1, upper slewing body 3, boom 4, arm 5, and bucket 6.

[0065] Further, instead of being configured to be operable by an operator in the cab 10, or in addition thereto, the excavator 100 may be configured to be remotely operable from the outside of the excavator 100. When the excavator 100 is remotely operated, the inside of the cab 10 may be unmanned.

[0066] The auxiliary storage device D4 stores the learned model LM.

[0067] When the imaging image information captured by the imaging device S6 is input from the input layer, the learned model LM outputs, from the output layer, image information indicating an image in which a person represented in the imaging image information is surrounded by a rectangle. Note that in this embodiment, an example of the output mode of the learned model LM will be described. However, this embodiment is not limited to a method of outputting image information in which a person is surrounded by a rectangle, and it is sufficient that the result of detecting a person represented in the imaging image information is shown.

[0068] As the machine learning used for generating the learned model LM, for example, a neural network may be applied, specifically, machine learning using a deep neural network (DNN), and deep learning (deep neural network) may be applied. As deep learning, for example, a convolutional neural network, RNN (Recurrent Neural Networks), or LSTM (Long Short Term Memory) may be applied.

[0069] The learned model LM is generated by performing machine learning based on a teacher dataset generated in advance in an information processing device (not shown).

[0070] Specifically, the learned model LM is generated by machine learning based on the captured image information in which a person appears and the image information in which the person appearing in the captured image information is surrounded by a rectangle, both of which are included in the teacher dataset.

[0071] Note that the learned model LM may be updated by additional learning of a new teacher dataset in an existing learned model LM.

[0072] The controller 30 includes an acquisition unit 301, a detection unit 302, a position estimation unit 303, an output control unit 304, and a determination unit 305.

[0073] The acquisition unit 301 acquires various information from various sensors. For example, the acquisition unit 301 acquires the captured image information captured by the imaging devices S6 (front camera S6F, left camera S6L, right camera S6R, and rear camera S6B).

[0074] The acquisition unit 301 acquires the detection information detected by each of the boom angle sensor S1, arm angle sensor S2, bucket angle sensor S3, body tilt sensor S4, and turning angle sensor S5. The acquisition unit 301 acquires the position and orientation of the excavator 100 from the positioning device PS.

[0075] The detection unit 302 performs a detection process of a person existing around the excavator 100 from the captured image information acquired by the acquisition unit 301. The detection unit 302 according to the present embodiment inputs the captured image information into the learned model LM, and receives from the learned model LM the image information in which the person represented in the captured image information is surrounded by a rectangle. In the present embodiment, a method for detecting a person using the learned model LM will be described. However, the present embodiment does not limit the method for detecting a person, and any method may be used regardless of well-known methods. For example, it may be determined whether or not a feature amount extracted from the captured image information is approximated by a predetermined value or more as compared with a feature amount indicating a person.

[0076] When there is a region surrounded by a rectangle in the image information received by the position estimation unit 303 from the detection unit 302, the position estimation unit 303 estimates the position of the person in the real space, for example, the direction and distance in which the person exists with respect to the excavator 100, from the position coordinates where the rectangle appears in the image information. As a specific estimation method, a conventionally used method may be used, and the description thereof will be omitted.

[0077] The position estimation unit 303 according to the present embodiment uses a method for estimating the direction and distance in which a person exists based on the position and size of the rectangle in which the person appears in the image information, but other methods may also be used. For example, a method in which the learned model LM outputs the direction and distance in which a person exists may be used.

[0078] Note that the present embodiment does not limit the method for detecting a person and estimating the position of the person by the controller 30. For example, it may be performed in the imaging device S6, or an external cloud service may be used.

[0079] The output control unit 304 outputs the turning angle, the image information in which the region where the person appears is surrounded by a rectangle, the position coordinate information of the person existing around the excavator 100 (information indicating the direction and distance in which the detected person exists), and the detection results of various sensors to the control unit D3a of the display device D3. Thereby, the display device D3 displays a screen showing the periphery of the excavator 100.

[0080] Next, with reference to FIG. 5, an example of the display screen displayed on the display device D3 will be described. FIG. 5 is a diagram showing an example of the display screen 42 displayed by the display device D3 according to the present embodiment.

[0081] The control unit D3a according to the present embodiment generates a display screen based on the image information input from the imaging device S6 and various information received from the controller 30. The information received from the controller 30 includes the turning angle, image information surrounding the area where a person is shown in a rectangle, position coordinate information of a person existing around the excavator (information indicating the direction and distance where the detected person exists), and detection results of various sensors.

[0082] On the display screen 42, according to the control from the control unit D3a, a date and time display area 42a, a travel mode display area 42b, an attachment display area 42c, a fuel consumption display area 42d, an engine control state display area 42e, an engine operation time display area, a coolant water temperature display area 42g, a fuel remaining amount display area 42h, a rotation speed level display area 42i, a urea water remaining amount display area 42j, an operating oil temperature display area 42k, a person detection map display area 421, a first image display area 422, and a second image display area 423 are displayed. The display screen 42 may include other display areas.

[0083] The travel mode display area 42b, the attachment display area 42c, the engine control state display area 42e, and the rotation speed level display area 42i are areas for displaying setting state information, which is information regarding the setting state of the excavator 100. The fuel consumption display area 42d, the engine operation time display area, the coolant water temperature display area 42g, the fuel remaining amount display area 42h, the urea water remaining amount display area 42j, and the operating oil temperature display area 42k are areas for displaying operation state information, which is information representing the operation state of the excavator 100 based on the detection results of various sensors.

[0084] The date and time display area 42a is an area for displaying the current date and time. The traveling mode display area 42b is an area for displaying the current traveling mode. The attachment display area 42c is an area for displaying an image representing the currently attached attachment. The fuel consumption display area 42d is an area for displaying the fuel consumption information calculated by the controller 30. The fuel consumption display area 42d includes an average fuel consumption display area 42d1 for displaying the lifetime average fuel consumption or the interval average fuel consumption, and an instantaneous fuel consumption display area 42d2 for displaying the instantaneous fuel consumption.

[0085] The engine control state display area 42e is an area for displaying the control state of the engine 11. The coolant water temperature display area 42g is an area for displaying the current temperature state of the engine coolant water. The fuel remaining amount display area 42h is an area for displaying the remaining amount state of the fuel stored in the fuel tank.

[0086] The rotation speed level display area 42i is an area for displaying the current level set by the dial 32 as an image. Numbers indicating the selected level are displayed in the rotation speed level display area 42i. The "1" displayed in the rotation speed level display area 42i indicates that the selected rotation speed level is the "first level". The number "n" displayed in the rotation speed level display area 42i indicates that the selected rotation speed level is the "nth level". "n" is a natural number. When the operator rotates the dial 32, the number displayed in the rotation speed level display area 42i changes.

[0087] The urea water remaining amount display area 42j is an area for displaying, as an image, the remaining amount state of the urea water stored in the urea water tank. The operating oil temperature display area 42k is an area for displaying the temperature state of the operating oil in the operating oil tank.

[0088] The person detection map display area 421 is an area for displaying information representing the positional relationship between the excavator 100 and the person detected around the excavator 100.

[0089] The human detection map display area (an example of position information) 421 is a display area of a map representing the real space centered on the excavator 100 at a predetermined scale ratio. In the human detection map display area 421, an excavator icon 421b indicating the presence of the excavator 100 is arranged at the center of the area.

[0090] In the human detection map display area 421, in addition to the excavator icon 421b representing the excavator 100, an icon indicating the possible traveling direction of the excavator 100 (direction display icon 421a in the example of FIG. 5) and icons representing the persons detected from the periphery of the excavator 100 (human detection icons 421e, 421f in the example of FIG. 5) are simultaneously displayed. The areas in the human detection map display area 421 other than the excavator icon 421b, the direction display icon 421a, and the human detection icons 421e, 421f (in other words, the background) may be areas represented by, for example, a single color (for example, black).

[0091] The excavator icon 421b is an icon combining an image indicating the upper swing body 3 and an image indicating the lower traveling body 1 according to the positional relationship between the upper swing body 3 and the lower traveling body 1 based on the swing angle.

[0092] The direction display icon 421a indicates, in a triangular shape, the direction in which the excavator 100 travels when the travel lever is tilted forward. Note that this embodiment shows an example of an icon representing the direction in which the excavator 100 travels when the travel lever is tilted forward, and any shape may be used as long as it represents the possible traveling direction of the excavator 100.

[0093] The human detection icons (an example of display information) 421e, 421f are icons indicating the positions where persons are present, estimated by the position estimation unit 303. Specifically, the human detection icons 421e, 421f are arranged based on the direction and distance in which persons are present with respect to the excavator 100, estimated by the position estimation unit 303. Specifically, the human detection icons 421e, 421f are arranged at positions obtained by multiplying the direction and distance in which the detected persons are present with respect to the excavator 100 by a predetermined scale ratio.

[0094] In this way, the positional relationship between the shovel icon 421b and the human detection icons 421e and 421f represented in the human detection map display area 421 corresponds to the positional relationship between the shovel 100 in the real space and the humans existing around the shovel 100.

[0095] In the present embodiment, an example using the human detection map display area 421 will be described as position information representing the positional relationship between each of the humans detected by the imaging device S6 and the shovel 100 in a diagram. However, the present embodiment does not limit the position information representing the positional relationship between each of the humans detected by the imaging device S6 and the shovel 100 to the human detection map display area 421. For example, position information representing the positional relationship by superimposing the positions of each of the humans detected by the imaging device S6 and the shovel 100 on the bird's-eye view image information based on the imaging image information captured by the imaging device S6 may be used. In this way, the position information may be any information that can recognize the positional relationship between the shovel and the humans.

[0096] In the human detection map display area 421 according to the present embodiment, since the display of objects other than the humans existing around the shovel 100 is suppressed, the operator can recognize the current situation and the possible moving direction of the shovel 100, and the positional relationship with the humans existing around the shovel 100 by referring to the human detection map display area 421.

[0097] Also, by referring to the human detection map display area 421, the operator can infer how the positional relationship between the shovel 100 and the humans existing around it changes when the shovel 100 is moved. Furthermore, since the display of objects other than the shovel and humans is suppressed in the human detection map display area 421, it is possible to prevent the operator from being attracted by other objects and forgetting the presence of humans. Therefore, an improvement in safety can be achieved.

[0098] In addition, in the human detection map display area 421, a first circular area 421c and a second circular area 421d, which are determined according to the distance from the shovel 100 with the shovel 100 as a reference, are displayed.

[0099] As shown in FIG. 5, the first circular area 421c and the second circular area 421d are represented as circles for allowing an operator to recognize the relative distance from the shovel 100 with the shovel icon 421b as a reference.

[0100] The first circular area 421c is, for example, information indicating a range within 2 m around the shovel 100. The second circular area 421d is, for example, information indicating a range within 4 m around the shovel 100.

[0101] The display screen 42 according to the present embodiment can recognize the position of a person with the shovel 100 as a reference based on the positional relationship between the first circular area 421c and the second circular area 421d and the human detection icons 421e, 421f.

[0102] Furthermore, the control unit D3a makes the display mode different for the human detection icons 421e, 421f according to whether they are included in the first circular area 421c and the second circular area 421d.

[0103] For example, the human detection icon 421e existing within the first circular area 421c is displayed in red, for example. Also, the human detection icon 421f existing outside the first circular area 421c and within the second circular area 421d is displayed in yellow, for example. Also, a human detection icon (not shown) existing outside the second circular area 421d is displayed in green, for example.

[0104] The control unit D3a according to this embodiment varies the display mode according to whether it is included in the first circular area 421c and the second circular area 421d. In other words, it changes the color of the human detection icon according to the distance from the excavator 100. In this way, by displaying the human detection icon whose color changes according to the distance, the display device D3 can arouse the operator's attention according to the distance between the excavator 100 and the person. Therefore, the safety can be improved.

[0105] The color of the human detection icon in this embodiment is shown as an example and is not limited to this color. For example, the human detection icon may be displayed in grayscale. Further, the color change of the human detection icon in this embodiment shows an example of the display mode and is not limited to the change of this color. For example, according to whether it is included in the first circular area 421c and the second circular area 421d, the blinking period of the human detection icon may be varied, or the contrast or brightness may be varied, etc.

[0106] On the display screen 42, a human detection map display area 421 is displayed, and the image information captured by the imaging device S6 is also displayed. The operator can recognize the specific situation around the excavator 100 by checking the image information together with the human detection map display area 421. Therefore, the safety can be improved.

[0107] Among the display screens 42 shown in FIG. 5, the first image display area 422 and the second image display area 423 are areas for displaying the image information captured by the imaging device S6 and include an image 423c of the counterweight of the upper swing body 3. A right-side image is displayed in the first image display area 422. A rear image is displayed in the second image display area 423. The right-side image is an image showing the space on the right side of the excavator 100 and includes an image 422c at the right end of the upper surface of the upper swing body 3.

[0108] The right image is image information output from the learned model LM, and is image information in which the person represented in the actual viewpoint image captured by the right camera S6R is surrounded by a rectangle. The rear image is image information output from the learned model LM, and is image information in which the person represented in the actual viewpoint image captured by the rear camera S6B is surrounded by a rectangle.

[0109] As a result, a frame 422b is displayed in the right image of the first image display area 422 so as to surround the person 422a, and a frame 423b is displayed in the rear image of the second image display area 423 so as to surround the person 423a.

[0110] The first image display area 422 is displayed to the right with reference to the person detection map display area 421. Also, the second image display area 423 is displayed below with reference to the person detection map display area 421. In this embodiment, it is assumed that the upper part of the display screen 42 corresponds to the front of the upper rotating body 3. In other words, the second image display area 423 is displayed at a position corresponding to the rear with reference to the person detection map display area 421. That is, the display screen 42 displays the image information captured by the imaging device S6 in the direction in which the imaging device S6 captured the image, with reference to the person detection map display area 421. In this embodiment, since the image information captured in the imaging direction is displayed with reference to the person detection map display area 421, the operator can intuitively recognize which direction the situation represented by the image information is when referring to the image information. Therefore, an improvement in safety can be realized.

[0111] Note that this embodiment shows an example of the arrangement of the image information and is not limited to this arrangement. For example, the first image display area 422 and the second image display area 423 may be arranged regardless of the imaging direction.

[0112] The control unit D3a matches the color of the frame 422b of the right image in the first image display area 422 with the color of the person detection icon 421e, and also matches (an example of association) the color of the frame 423b of the rear image in the second image display area 423 with the color of the person detection icon 421f.

[0113] That is, on the display screen 42, a frame indicating the detected person is displayed for the right-side image and the rear image, and the correspondence between the person indicated by the frame and the person detection icon is displayed in a recognizable manner. Thereby, the operator can recognize the situation of the person indicated in the person detection map display area 421 by referring to the right-side image and the rear image. Therefore, the operator can operate the excavator 100 in consideration of the situation of the people present around the excavator 100. Therefore, the safety can be improved. In this embodiment, as an example of the display for recognizing the correspondence, an example in which the color of the frame and the color of the person detection icon are made to match has been described. However, this embodiment shows an example of the display for recognizing the correspondence, and does not limit the method of making the color of the frame and the color of the person detection icon match. For example, the correspondence may be made recognizable by the blinking period of the frame and the person detection icon.

[0114] As described above, the color of the person detection icon is changed according to the distance from the excavator 100. The color of the person detection icon in the person detection map display area 421 is made to match the color of the frame displayed in the right-side image and the rear image. Therefore, on the display screen 42, based on the distance between the excavator 100 and the person, the colors (an example of the display mode) of the frames represented in the right-side image and the rear image are displayed differently. In this embodiment, since the colors of the frames represented in the right-side image and the rear image are changed according to the distance from the excavator 100, the operator can recognize the distance from the excavator 100 when referring to the color of the frame. Therefore, since the operator can perform an operation according to the distance, the safety can be improved.

[0115] In this embodiment, by the control unit D3a displaying the display screen 42, the operator can grasp the situation around the excavator 100.

[0116] Returning to FIG. 4, the determination unit 305 compares the direction and distance of a person estimated from the image information received from the previously learned model LM with the direction and distance of a person estimated from the image information received from the current learned model LM, and determines whether the person detected previously has disappeared in this detection.

[0117] Then, when the determination unit 305 determines that the person detected previously has disappeared in this detection, the determination unit 305 determines whether the positional relationship between the detection range in which a person can be detected from the imaging image information of the imaging device S6 and the position of the person who has disappeared satisfies a predetermined condition. The conditions of this embodiment will be described.

[0118] FIG. 6 is a diagram for explaining a method for determining whether a person exists around the excavator 100 according to the positional relationship between the excavator 100 and the person in the determination unit 305 according to this embodiment. The detection range 1600 shown in FIG. 6 indicates a range around the excavator 100 within which a person can be detected by the front camera S6F, the rear camera S6B, the left camera S6L, and the right camera S6R with the excavator 100 as the center. In other words, the detection range 1600 is a part of the range obtained by combining the imaging ranges AF, AB, AL, and AR shown in FIG. 2, and is a range within which a person can be detected from the imaging image information of the imaging device S6.

[0119] In this embodiment, for the determination by the determination unit 305, the detection range 1600 is divided into three regions. In this embodiment, the detection range 1600 is divided into a first range 1601, a second range 1602, and a third range 1603 from the inside.

[0120] The first range 1601 is a range within 2 m around the excavator 100. The second range 1602 is a range outside the first range 1601 and within 4 m around the excavator 100. The third range 1603 is a range outside the second range 1602 and within 5 m around the excavator 100.

[0121] In the present embodiment, an example is given where the first range 1601 corresponds to the first circular region 421c of the human detection map display area 421, and the second range 1602 corresponds to the second circular region 421d of the human detection map display area 421. However, the division of the detection range 1600 described above is not limited to the method of corresponding to the display of the human detection map display area 421, and the division of the detection range 1600 may be used only for internal processing.

[0122] When the determination unit 305 determines that a person detected previously is not detected this time, the determination result is made different according to which range among the first range 1601 to the third range 1603 the person was present in.

[0123] The third range 1603 is the outer range of the detection range 1600. For example, when a person 1631 is present in the third range 1603, by moving to the arrow 1632, the person can immediately move outside the detection range 1600. Therefore, when the determination unit 305 determines that a person present in the third range 1603 is not detected this time, it can be determined that the person has moved outside the detection range 1600. That is, the determination unit 305 can determine that the person has actually disappeared.

[0124] The second range 1602 is not near the shovel 100 and is not an area outside the detection range 1600. For example, when a person 1621 is present in the second range 1602, it is difficult to move outside the detection range 1600 without being detected in the first range 1601 or the third range 1603. Therefore, when the determination unit 305 determines that a person present in the second range 1602 is not detected this time, it determines that the person is still present in the detection range 1600 and that the reason for not being detected this time is a detection error.

[0125] The first range 1601 is the range near the excavator 100. For example, when a person 1611 is present in the first range 1601, the person can move under the lower traveling body 1 or attach to the excavator 100. That is, although the person is not present in the detection range 1600, there is a high possibility that a person is present near the excavator 100. Also, the person is still present in the detection range 1600, and there is a possibility that the non-detection this time is due to a detection error. Therefore, when a person who was present in the first range 1601 is not detected this time, the determination unit 305 determines that at least the person is present near the excavator 100.

[0126] That is, when a person whose position was represented in the person detection map display area 421 is no longer detected by the imaging device S6, the determination unit 305 determines that the position of the last detected person, such as the first range 1601 or the second range 1602, is not a position where it can move outside the detection range 1600, or when the position of the last detected person is a position where it can move from the detection range 1600 toward the excavator side, the determination unit 305 determines that the person is present near the excavator 100.

[0127] Note that the determination by the determination unit 305 in the present embodiment shows an example using determination conditions based on the positional relationship between the detection range in which a person can be detected from the imaging image information of the imaging device S6 and the position of the person who is no longer detected, and is not limited to the determination based on the determination conditions. For example, the determination unit 305 may determine whether the position of the person who is no longer detected is near the boundary of the detection range in which a person can be detected from the imaging image information of the imaging device S6. Furthermore, after considering the moving speed of the person, it may be determined whether the person has moved outside the detection range. The predetermined conditions for determination are not limited to the above-described determination conditions, and may be determined according to implementation modes such as the brightness situation at the work site, the moving speed of the person, the performance of the imaging device S6, and the resolution of the imaging image information of the imaging device S6.

[0128] In the present embodiment, determination conditions based on the positional relationship between the detection range in which a person can be detected from the captured image information of the imaging device S6 and the position of the person who has disappeared from detection are used. Therefore, the determination by the determination unit 305 according to the present embodiment is a determination considering the positional relationship and is not a determination using a new sensor or the like. Thus, it is possible to improve the detection accuracy while suppressing the cost.

[0129] The output control unit 304 outputs the determination result by the determination unit 305 to the control unit D3a of the display device D3. When the person whose position was represented in the person detection map display area 421 disappears from detection in the captured image information of the imaging device S6 and it is determined by the determination unit 305 that the person is present in the vicinity of the excavator 100 (an example of a case where a predetermined condition is satisfied), the control unit D3a continues to display the position of the person who has disappeared from detection in the person detection map display area 421.

[0130] FIG. 7 is a diagram showing an example of a display screen 42A displayed by the display device D3 according to the present embodiment. Regarding the same configuration as the display screen 42 shown in FIG. 5 among the display screens 42A shown in FIG. 7, the same reference numerals are assigned and the description thereof is omitted.

[0131] In the display screen 42A shown in FIG. 7, the right image displayed in the first image display area 422A is not detected by the learned model LM because the person 1422a is moving.

[0132] Even when no person is detected in the right image, in other words, even when the display of the frame surrounding the person (an example of the area where the person is detected) in the first image display area 422A is suppressed, the display device D3 continues to display the person detection icon 1421e corresponding to the person 1411a in the person detection map display area 421A together with the first image display area 422A.

[0133] Therefore, when the operator refers to the display screen 42A, even if a rectangle surrounding a person is not displayed in the first image display area 422A, by referring to the person detection icon 1421e displayed in the person detection map display area 421A, it can be inferred that there is a person in the vicinity of the right side of the excavator 100. Thus, in this embodiment, since the person detection map display area 421A and the first image display area 422A are simultaneously displayed on the display screen 42A, the operator can recognize the surrounding situation.

[0134] And when the determination unit 305 according to this embodiment determines that a person exists in the vicinity of the excavator 100, the controller 30 continues to hold the direction and distance at which the person was last detected. Then, the determination unit 305 repeatedly determines whether the previously detected person is detected this time using the held information.

[0135] And when the detection unit 302 newly detects a person from the captured image information of the imaging device S6, the determination unit 305 determines that the previously detected person is detected again this time.

[0136] In this case, the display device D3 moves the person detection icon to a position corresponding to the direction and distance of the newly detected person without increasing the number of person detection icons displayed in the person detection map display area. And the controller 30 deletes the information regarding the direction and distance at which the person was last detected, which it has been holding.

[0137] The controller 30 according to this embodiment initializes the detection situation once when performing control to turn off the power. That is, the controller 30 deletes the information regarding the direction and distance at which the person was last detected, which it has been holding. Also, the initialization process according to this embodiment may be performed other than when turning off the power, for example, when the upper slewing body 3 slews or the lower traveling body 1 travels.

[0138] Next, the processing procedures executed by the controller 30 and the display device D3 according to the present embodiment will be described. FIG. 8 is a flowchart showing the processing procedures until the controller 30 and the display device D3 according to the present embodiment display a display screen. It is assumed that the processing procedures shown in the flowchart according to the present embodiment are repeatedly performed at predetermined intervals.

[0139] First, the acquisition unit 301 acquires the captured image information captured by the imaging device S6 (S1801).

[0140] The detection unit 302 performs a detection process for a person existing around the excavator 100 from the captured image information acquired by the acquisition unit 301 (S1802). In the present embodiment, the detection unit 302 inputs the captured image information into the learned model LM, and receives, from the learned model LM, image information in which a person represented in the captured image information is surrounded by a rectangle, thereby detecting a person existing around.

[0141] The position estimation unit 303 estimates the direction and distance in which the person exists with respect to the excavator 100 from the position coordinates where the person appears (where the rectangle appears) in the captured image information (S1803).

[0142] The determination unit 305 compares the direction and distance of the person estimated in the previous S1803 with the direction and distance of the person estimated in the current S1803, and determines whether a person detected previously is not detected this time (S1804).

[0143] When the determination unit 305 determines that a person detected previously is also detected this time (S1804: NO), the display device D3 displays a display screen including a person detection map display area representing the position of the detected person (S1805).

[0144] When the determination unit 305 determines that a person detected previously is not detected this time (S1804: YES), it determines whether the position of the person at the time of the previous detection is within the first range or the second range of the detection range (S1806). When the determination unit 305 determines that it is outside the first range and the second range, in other words, within the third range (S1806: NO), the display device D3 displays a display screen including a person detection map display area representing the position of the detected person (S1805).

[0145] On the other hand, when the determination unit 305 determines that the position of the person at the time of the previous detection is within the first range or the second range (S1806: YES), the display device D3 displays a display screen including a person detection map display area where a person detection icon is represented at the position of the person detected previously (S1807).

[0146] In the present embodiment, by performing the above-described control, even when a person is not detected from the captured image information, if a predetermined condition is satisfied, a person detection map display area where a person detection icon is represented at the position of the person detected previously is displayed. Therefore, even when a detection error occurs or when a person approaches the excavator 100 and is no longer included in the imaging range, the operator can be made to recognize that the person is present.

[0147] The display device D3 according to the present embodiment has been described with an example in which when a person detected previously is no longer detected, a display screen including a person detection map display area similar to the previous one is displayed. However, the present embodiment does not limit the display mode of the display screen including the person detection map display area, and various display modes may be used.

[0148] FIG. 9 is a diagram showing a modified example of the person detection map display area displayed by the display device D3.

[0149] FIG. 9(a) shows a modified example in which when the determination unit 305 determines that a person detected previously is not detected this time, a region where there may be a person who is not detected this time is displayed. In the person detection map display region 1901 shown in FIG. 9(a), together with the display of the person detection icon 1911a corresponding to the position of the person detected previously, a region 1911b where there may be such a person is displayed. The region 1911b may be displayed so as to expand with the passage of time in consideration of the movement speed of the person.

[0150] In the person detection map display region 1901 shown in FIG. 9(a), by presenting to the operator a region where there may be a person, the operator can recognize in which region there may be a person. Therefore, an improvement in safety can be realized.

[0151] FIG. 9(b) shows a modified example in which when the determination unit 305 determines that a person detected previously is not detected this time, the display mode of the person detection icon regarding the person who is not detected this time is changed. In the person detection map display region 1902 shown in FIG. 9(b), the person detection icon 1921 corresponding to the position of the person detected previously is displayed. The person detection icon 1921 has a diamond shape and is different in shape from the person detection icon of the detected person (for example, the person detection icon 1922).

[0152] Thus, when the determination unit 305 determines that a person detected previously is not detected this time, by changing the display mode of the person detection icon by the display device D3, the operator can be made to recognize that there is a person who is not detected this time. Note that the display information indicating the position of the person may be other than the person detection icon, and may be shown, for example, in a region as shown in FIG. 9(c).

[0153] FIG. 9(c) shows a modified example of the human detection map display area 1903 displayed by the display device D3. In the human detection map display area 1903 shown in FIG. 9(c), instead of displaying a human detection icon, a circular area 1930 representing the periphery of the shovel 100 is displayed together with a shovel icon 1933. Then, the circular area 1930 is divided into predetermined areas, and for each divided area, whether a person exists or not is represented by color.

[0154] Furthermore, in the human detection map display area 1903, when it is determined that a previously detected person is not detected this time, the area where the previously detected person existed is made to have a different color from the area where no person is detected.

[0155] For example, the area 1931 indicates an area where a previously detected person existed but was not detected this time.

[0156] By having the display device D3 represent the area 1931, the operator can be made aware of the existence of a person who was not detected this time.

[0157] In the human detection map display area 1903 shown in FIG. 9(c), furthermore, the color of the area where a previously detected person was not detected this time (for example, the area 1931) and the area where a person was detected this time (for example, the area 1932) may be made different. Therefore, when the operator refers to the human detection map display area 1903, the operator can recognize the current situation in detail due to the color difference.

[0158] The human detection map display area 1903 shown in FIG. 9(c) has been described in terms of the mode of displaying the human detection results for 360 degrees around with the shovel icon 1933 as a reference. However, this modified example is not limited to the mode of displaying the human detection results for 360 degrees around with the shovel icon 1933 as a reference. For example, it may be 180 degrees behind, or the human detection results may be displayed for 270 degrees behind and on the right side.

[0159] This embodiment does not limit the continuation of the display in the above-described human detection map display area when a person who was detected previously is not detected this time. For example, in addition to continuing the display in the human detection map display area, the controller 30 may display a pop-up screen prompting attention. The pop-up screen is expressed, for example, as "There may have been a detection failure, so please check the surroundings." The pop-up screen is displayed so as not to overlap with the first image display area 422 and the second image display area 423. Thereby, the operator can recognize the surrounding situation and grasp the fact that no person has been detected.

[0160] (Second Embodiment) In the above-described embodiment, an example of detecting a person from the imaging image information captured by the imaging device S6 has been described. However, the above-described embodiment does not limit the method of detecting a person from the imaging image information captured by the imaging device S6. Therefore, in the second embodiment, a case where an object detection device S7 is provided in addition to the imaging device S6 will be described.

[0161] FIG. 10 is a side view of the excavator 100A according to the second embodiment. FIG. 11 is a top view of the excavator 100A according to the second embodiment. In this embodiment, the same components as those in the first embodiment are assigned the same reference numerals and the description thereof is omitted.

[0162] The object detection device S7 is an example of a space recognition device, and detects the presence or absence of an object existing in the space around the excavator 100A and the distance to the object. The object detection device S7 outputs the result of measuring the space to the controller 30A as measurement information.

[0163] The object detection device S7 includes a rear object detection device S7B that detects the space behind the excavator 100A, a left object detection device S7L that detects the space to the left of the excavator 100A, a right object detection device S7R that detects the space to the right of the excavator 100A, and a front object detection device S7F that detects the space in front of the excavator 100A.

[0164] The object detection device S7 may use LIDAR to detect objects existing around the excavator 100A. LIDAR measures, for example, the distance between more than one million points within the monitoring range and the LIDAR. Note that this embodiment is not limited to the method using LIDAR, and any spatial recognition device capable of measuring the distance to an object may be used. For example, a stereo camera may be used, or a ranging device such as a distance image camera or a millimeter wave radar may be used. When a millimeter wave radar or the like is used as the object detection device S7, a large number of signals (such as laser light) may be transmitted from the object detection device S7 toward the object, and the distance and direction of the object may be derived from the reflected signal by receiving the reflected signal.

[0165] The rear object detection device S7B is attached to the rear end of the upper slewing body 3. The left object detection device S7L is attached to the left end of the upper slewing body 3. The right object detection device S7R is attached to the right end of the upper slewing body 3. The front object detection device S7F is attached to the front end of the upper surface of the cab 10.

[0166] The front object detection device S7F, the rear object detection device S7B, the left object detection device S7L, and the right object detection device S7R are all attached to the upper slewing body 3 such that the optical axis faces obliquely downward and a part of the upper slewing body 3 is included in the detection range. Therefore, each detection range of the front object detection device S7F, the rear object detection device S7B, the left object detection device S7L, and the right object detection device S7R has, for example, a viewing angle of about 180 degrees in a top view. In the example of FIG. 11, the detection range SF represents an example of the detection range of the front object detection device S7F, the detection range SB represents an example of the detection range of the rear object detection device S7B, the detection range SL represents an example of the detection range of the left object detection device S7L, and the detection range SR represents an example of the detection range of the right object detection device S7R.

[0167] As shown in FIG. 11, the detection ranges SF, SB, SR, and SL are assumed to be narrower than the imaging ranges AF, AB, AR, and AL, but the detection range and the imaging range may vary depending on the embodiment.

[0168] The controller 30A according to this embodiment can perform the same processing as the controller 30 of the above-described embodiment. Compared with the controller 30, it is only different in that the detection result of the object detection device S7 is used to estimate the position of a person in the real space, for example, the direction and distance in which the person exists with respect to the excavator 100A.

[0169] In the controller 30A according to this embodiment, based on the detection result of the object detection device S7, the position of a person in the real space, for example, the direction and distance in which the person exists with respect to the excavator 100A, is estimated. As for the method of estimating the direction and distance in which the person exists from the detection result, a well-known method may be used, for example, a learned model may be used. For example, the controller 30A may receive the direction and distance in which the person exists by inputting the detection result of the object detection device S7 into the learned model.

[0170] The display device D3 according to this embodiment displays a person detection map display area based on the estimation result of the direction and distance in which the person exists by the controller 30A on the display screen. The description of the information displayed in other display areas is omitted as it is the same as in the above-described embodiment.

[0171] That is, the display device D3 according to this embodiment is configured to display the imaging image information captured by the imaging device S6 provided on the upper swing body 3 separately from the object detection device S7, together with the person detection map display area.

[0172] Also, the determination unit 305 according to this embodiment is not limited to making a determination based on the detection result of the object detection device S7, and may make a determination by combining the detection result of the object detection device S7 and the imaging result of the imaging device S6. In this determination, the difference between the detection ranges SF, SB, SR, SL and the detection range in which a person can be detected from the imaging image information of the imaging device S6 may be considered. For example, when it is determined that a person who has disappeared from the detection by the object detection device S7 is detected by the imaging device S6, the display device D3 may display the position of the person in the person detection map display area.

[0173] (Third Embodiment) In the above-described embodiment, the case where the operator performs work with the excavator 100 on which the operator is riding has been described. However, the above-described embodiment is not limited to the method of performing work when the operator is riding on the excavator 100. For example, when the excavator 100 performs work according to remote control, the same display as in the above-described embodiment may be provided. Therefore, in the third embodiment, the case of remotely operating the excavator 100 will be described.

[0174] Therefore, with reference to FIG. 12, the outline of the remote operation system SYS according to the third embodiment will be described. FIG. 12 is a schematic diagram showing an example of the remote operation system SYS according to the third embodiment.

[0175] As shown in FIG. 12, the remote operation system SYS according to the third embodiment includes an excavator 100, a fixed-point camera 1201, and a remote operation room RC.

[0176] The excavator 100 and the remote operation room RC are connected so as to be able to transmit and receive data via a communication line NW.

[0177] The excavator 100 can perform wireless communication by using a communication device T1. Then, the excavator 100 can transmit and receive data to and from devices (for example, the remote operation room RC) connected to the communication line NW.

[0178] And the excavator 100 can transmit information about the work site to the remote operation room RC. Thereby, the remote operation room RC can confirm the work site according to the information from the excavator 100. Note that this embodiment is not limited to the device for measuring the work site being the excavator 100, and other types of devices such as a drone flying over the work site, a fixed-point camera, or an imaging device that can be carried by the user may be used.

[0179] For example, the excavator 100 is provided with an imaging device S6. The excavator 100 transmits imaging image information showing the imaging result of the work site by the imaging device S6 to the remote operation room RC. Or, the fixed-point camera 1201 provided at the work site transmits imaging image information showing the imaging result of the work site to the remote operation room RC. Also, the device for monitoring the work site is not limited to the fixed-point camera 1201, and it may be a drone flying over the work site or an imaging device that can be held by a user. The drone or the imaging device may transmit imaging image information showing the imaging result of the work site to the remote operation room RC.

[0180] The excavator 100 included in the remote operation system SYS may be one or a plurality of them. Thereby, the remote operation system SYS can provide information regarding the work site to the remote operation room RC through a plurality of excavators 100.

[0181] <Configuration example of remote operation room> The remote operation room RC is equipped with a communication device T2, a remote controller R30, an operation device R26, an operation sensor R29, and a display device DR. Also, an operation seat DS on which an operator OP who remotely operates the excavator 100 sits is installed in the remote operation room RC.

[0182] The communication device T2 is configured to control communication with the communication device T1 attached to the excavator 100.

[0183] The remote controller (an example of a remote operation device) R30 is an arithmetic device that executes various operations. In the present embodiment, the remote controller R30 is composed of a microcomputer including a CPU and a memory. And various functions of the remote controller R30 are realized by the CPU executing a program stored in the memory.

[0184] The display device DR displays a screen based on the information transmitted from the excavator 100 so that the operator OP in the remote operation room RC can visually check the surroundings of the excavator 100. Even though the operator is in the remote operation room RC, the display device DR enables the operator to check the situation at the work site including the surroundings of the excavator 100.

[0185] Furthermore, similar to the first embodiment, the display device DR displays a display screen including a human detection map display area together with the captured image information.

[0186] The operating device R26 is provided with an operation sensor R29 for detecting the operation content of the operating device R26. The operation sensor R29 is, for example, an inclination sensor for detecting the inclination angle of the operation lever, or an angle sensor for detecting the swing angle around the swing axis of the operation lever, etc. The operation sensor R29 may be composed of other sensors such as a pressure sensor, a current sensor, a voltage sensor, or a distance sensor. The operation sensor R29 outputs information regarding the detected operation content of the operating device R26 to the remote controller R30. The remote controller R30 generates an operation signal based on the received information and transmits the generated operation signal toward the excavator 100. The operation sensor R29 may be configured to generate an operation signal. In this case, the operation sensor R29 may output the operation signal to the communication device T2 without going through the remote controller R30. Thereby, remote operation of the excavator 100 can be realized from the remote operation room RC.

[0187] Then, the communication device T1 of the excavator 100 receives the operation signal from the communication device T2 of the remote controller R30. The controller 30 of the excavator 100 performs various operations at the work site based on the received operation signal.

[0188] In this embodiment, the remote controller R30 causes the display device DR to display a display screen including a human detection map display area together with the captured image information. Further, when no human is detected from the captured image information of the imaging device S6, the remote controller R30 performs the same control as in the above-described embodiment. Thereby, this embodiment can obtain the same effects as the above-described embodiment. Note that this embodiment does not limit the display of the display screen including the human detection map display area together with the captured image information to the remote operation room RC, and for example, it may be displayed on a display device provided in a management center for managing the work site or the like.

[0189] <Operation> In the above-described embodiment, the controller 30, 30A, the display device D3, or the remote controller R30 can cause the operator to recognize the possibility that a person may be present when there may be a person around the excavator by displaying the display screen by the above-described control. Therefore, the operator can operate the excavator in consideration of the possibility that a person may be present around, so that an improvement in safety can be realized.

[0190] As described above, embodiments of the excavator and the control system of the excavator according to the present invention have been described, but the present invention is not limited to the above embodiments. Various changes, modifications, substitutions, additions, deletions, and combinations are possible within the scope described in the claims. Naturally, they also belong to the technical scope of the present invention.

Explanation of Reference Numerals

[0191] 100, 100A Excavator 1 Lower Traveling Body 2 Swing Mechanism 3 Upper Swing Body 4 Boom 5 Arm 6 Bucket S1 Boom Angle Sensor S2 Arm Angle Sensor S3 Bucket Angle Sensor S4 Machine Body Tilt Sensor S5 Swivel Angle Sensor S6 Imaging Device S7 Object Detection Device PS Positioning Device T1 Communication Device D3 Display Device D3a Control Unit D4 Auxiliary Memory Device LM Learned Model 30, 30A Controller 301 Acquisition Unit 302 Detection Unit 303 Position Estimation Unit 304 Output Control Unit 305 Judgment Unit RC Remote Operation Room R30 Remote Controller T2 Communication Device 1201 Fixed-Point Camera

Claims

1. A lower traveling body, an upper slewing body that is rotatably mounted on the lower traveling body, a space recognition device attached to the upper slewing body, a display device, a control device configured to display, on the display device, position information representing the relationship between the positions of each person detected by the space recognition device and the excavator, and when the person whose position is represented in the position information is no longer detected by the space recognition device, if a predetermined condition is satisfied, to continue displaying the position information of the person who has disappeared from the position information; An excavator comprising the same.

2. The predetermined condition is a condition based on the positional relationship between the detectable range of the space recognition device and the position of the person who has disappeared from detection. The excavator according to claim 1.

3. The predetermined condition is that the position of the person last detected is not a position where it can move outside the detection range, or the position of the person last detected is a position where it can move from the detection range to the excavator side. The excavator according to claim 2.

4. When the space recognition device is an imaging device, the control device is configured to display, on the display device together with the position information, an image captured by the imaging device or an image captured by an imaging device provided separately on the upper slewing body from the space recognition device. The excavator according to any one of claims 1 to 3.

5. The control device is configured to perform a display that represents the area where a person is detected in the image and associates the area with the position of the person represented in the position information. The excavator according to claim 4.

6. When the person whose position is represented in the position information is no longer detected by the space recognition device, if the predetermined condition is satisfied, the control device continues to display the position of the person who has disappeared from the position information and suppresses the display of the area. The excavator according to claim 5.

7. When the person whose position is represented in the position information is no longer detected by the space recognition device, if the predetermined condition is satisfied, the control device changes the display mode of the display information representing the position of the person who has disappeared from detection. The excavator according to claim 1.

8. An excavator comprising a lower traveling body, an upper slewing body that is rotatably mounted on the lower traveling body, and a space recognition device attached to the upper slewing body, a display device, The control device is configured to display, on the display device, position information representing the relationship between the positions of the person detected by the space recognition device and the excavator, and when the person whose position was represented in the position information is no longer detected by the space recognition device, to continue displaying on the position information the position of the person who is no longer detected when a predetermined condition is satisfied. An excavator control system comprising the same.

Images