EXCAVATORS AND CONTROLS FOR EXCAVATORS

Abstract

[Problem] Improve security. [Solution] An excavator according to one aspect of the present disclosure includes a lower traveling body; an upper revolving body rotatably mounted on the lower traveling body; a space detection device mounted on the upper revolving body; a display device; and a control device configured to display, on the display device, position information representing a relationship between positions of a person detected by the space detection device and the excavator, and, when the person whose position is represented in the position information is no longer detected by the space detection device, continue displaying the position of the person no longer detected in the position information when a predetermined condition is satisfied.

Description

TECHNICAL FIELD

[0001] The present invention relates to an excavator and a control system for an excavator. STATE OF THE ART

[0002] A conventional technique for detecting an object located around an excavator and monitoring the surroundings of the excavator has been proposed (see, for example, Patent Document 1). RELATED PRIOR ART PATENT DOCUMENT

[0003] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2014-224411 SUMMARY OF THE INVENTION PROBLEMS TO BE SOLVED BY THE INVENTION

[0004] Patent Document 1 describes a technique for detecting an object in the vicinity of a work machine using a stereo camera. However, there is a possibility that the detection of the object may not be adequately performed using a device such as a stereo camera depending on the environment or the like.

[0005] According to one aspect of an embodiment of the present invention, a technique for improving security by continuing to display a position of a person when the person is no longer detected by a space detection device and a predetermined condition is met is proposed. MEANS TO SOLVE THE PROBLEMS

[0006] According to one embodiment of the present disclosure, an excavator is provided. The excavator includes a lower traveling body; an upper revolving body rotatably mounted on the lower traveling body; a space detection device attached to the upper revolving body; a display device; and a control device configured to display, on the display device, position information representing a relationship between positions of a person detected by the space detection device and the excavator, and, when the person whose position is represented in the position information is no longer detected by the space detection device, further display the position of the person no longer detected in the position information in a case where a predetermined condition is satisfied.

[0007] Furthermore, according to another embodiment of the present disclosure, a control system for an excavator is provided. The control system includes an excavator having a lower traveling body, an upper revolving body rotatably mounted on the lower traveling body, and a space detection device attached to the upper revolving body; a display device; and a control device configured to display, on the display device, position information representing a relationship between positions of a person detected by the space detection device and the excavator, and, when the person whose position is represented in the position information is no longer detected by the space detection device, continue to display the position of the person no longer detected in the position information in a case where a predetermined condition is satisfied. EFFECTS OF THE INVENTION

[0008] According to one aspect of the invention, safety can be improved by indicating the presence of a person around the excavator. BRIEF DESCRIPTION OF THE DRAWINGS [ Fig. 1] Fig. 1 is a side view showing an example of an excavator according to a first embodiment. [ Fig. 2] Fig. 2 is a plan view showing an example of an excavator according to the first embodiment. [ Fig. 3] Fig. 3 is a diagram showing a configuration example of a drive control system of the excavator according to the first embodiment. [ Fig. 4] Fig. 4 is a functional block diagram showing a configuration example of a controller of the excavator according to the first embodiment; [ Fig. 5] Fig. 5 is a diagram showing an example of a display screen displayed by the display device according to the first embodiment; [ Fig. 6] Fig. 6 is a diagram illustrating a method of determining whether or not a person is present around the excavator according to a positional relationship between the excavator and the person in the determination unit according to the first embodiment; [ Fig. 7] Fig. 7 is a diagram showing an example of a display screen displayed by the display device according to the first embodiment; [ Fig. 8] Fig. 8 is a flowchart showing a processing procedure until the controller and the display device according to the first embodiment display a display screen; [ Fig. 9A] Fig. 9A is a diagram showing a modification of a display area of a human detection card displayed by the display device. [ Fig. 9B] Fig. 9B is a diagram showing a modification of the human detection display area displayed by the display device. [ Fig. 9C] Fig. 9C is a diagram showing a modification of the human detection display area displayed by the display device. [ Fig. 10] Fig. 10 is a side view of an excavator according to a second embodiment. [ Fig. 11] Fig. 11 is a plan view of the excavator according to the second embodiment. [ Fig. 12] Fig. 12 is a schematic diagram showing an example of a remote control system according to a third embodiment. EMBODIMENT OF THE INVENTION

[0009] Embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiments described below are not intended to limit the invention, but are merely examples, and all features and combinations thereof described in the embodiments are not necessarily essential to the invention. In the drawings, the same or corresponding components are designated by the same or corresponding reference numerals, and the description thereof may be omitted.

[0010] In the following embodiments of the present invention, an example using an excavator as an example of a work machine will be described, but the work machine is not limited to the excavator. The present invention can be applied to a construction machine, a standard machine, an application machine, a forestry machine, or a mining machine based on a hydraulic excavator. (First embodiment)

[0011] First, an overview of an excavator 100 according to the present embodiment will be described with reference to Fig. 1 described. Fig. 1 is a side view of the excavator 100 according to a first embodiment. Fig. 2 is a plan view of the excavator 100 according to a first embodiment.

[0012] An upper rotating body 3 is rotatably mounted on a lower traveling body 1 of the excavator 100 via a rotating mechanism 2. A boom 4 is attached to the upper rotating body 3. An arm 5 is attached to the distal end of the boom 4, and a bucket 6 is attached to the distal end of the arm 5 as an end attachment. The end attachment may be a shovel for an embankment or a dredging bucket.

[0013] The boom 4, the arm 5, and the bucket 6 constitute an excavation attachment, which is an example of the attachment AT, and are hydraulically driven by a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9, respectively. A boom angle sensor S1 is attached to the boom 4, an arm angle sensor S2 is attached to the arm 5, and a bucket angle sensor S3 is attached to the bucket 6. The excavation attachment may be provided with a bucket tilting mechanism.

[0014] The boom angle sensor S1 detects the rotation angle of the boom 4. In the present embodiment, the boom angle sensor S1 is an accelerometer and can detect a boom angle, which is a rotation angle of the boom 4 with respect to the upper rotating body 3. For example, the boom angle is a minimum angle when the boom 4 is lowered to the lowest position and increases as the boom 4 is raised.

[0015] The boom angle sensor S1 may include, for example, a rotary encoder, an accelerometer, a six-axis sensor, an inertial measurement unit (IMU), and the like. The boom angle sensor S1 may include potentiometers with a variable resistance, a cylinder stroke sensor that detects a stroke of a hydraulic cylinder (boom cylinder 7) corresponding to the boom angle, and the like. The same applies to the arm angle sensor S2, the bucket angle sensor S3, and a body inclination sensor S4. A detection signal of the boom angle sensor S1 corresponding to the boom angle is input to a controller 30.

[0016] The arm angle sensor S2 detects the rotation angle of the arm 5. In the present embodiment, the arm angle sensor S2 is an acceleration sensor and can detect an arm angle, which is a rotation angle of the arm 5 with respect to the boom 4. For example, the arm angle is a minimum angle when the arm 5 is closed to the maximum and increases when the arm 5 is opened.

[0017] The bucket angle sensor S3 detects the rotation angle of the bucket 6. In the present embodiment, the bucket angle sensor S3 is an accelerometer and can detect a bucket angle, which is a rotation angle of the bucket 6 with respect to the arm 5. For example, the bucket angle is a minimum angle when the bucket 6 is closed to the maximum and increases when the bucket 6 is opened.

[0018] The boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3 can be potentiometers using variable resistors, stroke sensors that detect the stroke amounts of the corresponding hydraulic cylinders, rotary encoders that detect the rotation angles around the coupling pins, or the like. The boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3 constitute a position sensor that detects the position of the attachment.

[0019] The upper rotating body 3 is provided with a cab 10 as an operator's cab and a power source such as a motor 11. Furthermore, a body inclination sensor S4, a rotation angle sensor S5, and an imaging device S6 are mounted on the upper rotating body 3. A communication device T1 and a positioning device PS are mounted on the upper rotating body 3.

[0020] The body tilt sensor S4 is configured to detect the tilt of the upper revolving body 3 with respect to a predetermined plane. In the present embodiment, the body tilt sensor S4 is an acceleration sensor that detects a tilt angle about the front-rear axis and a tilt angle about the left-right axis of the upper revolving body 3 with respect to the horizontal plane. For example, the longitudinal axis and the lateral axis of the upper revolving body 3 are orthogonal to each other and pass through the bucket center, which is a point on the rotation axis of the excavator 100.

[0021] The rotation angle sensor S5 is configured to detect a rotation angular velocity of the upper rotating body 3. In the present embodiment, the rotation angle sensor S5 is a gyro sensor. The rotation angle sensor S5 may be a resolver, a rotary encoder, or the like. The rotation angle sensor S5 can detect a rotational velocity. The rotational velocity can be calculated from the rotational angular velocity.

[0022] Note that when the body tilt sensor S4 includes a gyro sensor, a six-axis sensor, an IMU, or the like capable of detecting angular velocities around three axes, the rotation state (e.g., the rotation angular velocity) of the upper rotating body 3 can be detected based on the detection signal of the body tilt sensor S4. In this case, the rotation angle sensor S5 may be omitted.

[0023] The imaging device S6 is an example of a space recognition device and is configured to capture an image of the surroundings of the excavator 100. In the present embodiment, the imaging device S6 includes a front camera S6F that images a space in front of the excavator 100, a left camera S6L that images a space on the left side of the excavator 100, a right camera S6R that images a space on the right side of the excavator 100, and a rear camera S6B that images a space behind the excavator 100.

[0024] The imaging device S6 is, for example, a monocular camera with an imaging device such as a CCD or a CMOS, which outputs a captured image to the display device D3 via the controller 30.

[0025] The input device D2 receives an operation input from the user and outputs the operation input to the controller 30. The input device D2 includes any hardware operation unit, such as a touch panel, a touchpad, a button, a toggle switch, and a rotary knob. The input device D2 may include a software operation unit that can be operated via the hardware operation unit, such as a virtual button icon on an operation screen displayed on the display device D3, or the like.

[0026] As in Fig. For example, as shown in Figure 2, the front camera S6F is mounted on the roof of the cabin 10. The left camera S6L is mounted on a left attachment piece on the top of the upper rotating body 3. The right camera S6R is mounted on a right end of the top of the upper rotating body 3. The rear camera S6B is mounted on a rear end of the top of the upper rotating body 3.

[0027] The front camera S6F, the rear camera S6B, the left camera S6L, and the right camera S6R are all mounted on the upper rotating body 3 such that their optical axes are directed obliquely downward, and a part of the upper rotating body 3 is included in the imaging range. Therefore, the imaging range of each of the front camera S6F, the rear camera S6B, the left camera S6L, and the right camera S6R has, for example, a viewing angle of approximately 180 degrees in a plan view. In the example of Fig. 2, an imaging area AF represents an example of an imaging area of the front camera S6F, an imaging area AB represents an example of an imaging area of the rear camera S6B, an imaging area AL represents an example of an imaging area of the left camera S6L, and an imaging area AR represents an example of an imaging area of the right camera S6R. The four monocular cameras are preferably attached to the upper rotating body 3 such that they do not protrude beyond the upper surface of the upper rotating body 3, as in Fig. 2 shown.

[0028] In the present embodiment, the imaging device S6 is provided in the arrangement described above, and thus it is possible to capture an image of an object located around the excavator 100.

[0029] The positioning device PS is configured to acquire information about the position of the excavator 100. In the present embodiment, the positioning device PS is configured to measure the position and orientation of the excavator 100. Specifically, the positioning device PS is a GNSS receiver with an electronic compass that measures the latitude, longitude, and altitude of the current position of the excavator 100, as well as the orientation of the excavator 100.

[0030] Fig. 3 is a diagram showing an example of a configuration of a drive control system of the excavator 100 of Fig. 1 shows. In Fig. 3, the mechanical power transmission system is represented by a double line, the hydraulic oil line is represented by a thick solid line, the pilot line is represented by a dashed line, and the electric drive control system is represented by a thin solid line.

[0031] The engine 11 is a power source of the excavator 100. In the present embodiment, the engine 11 is a diesel engine that uses isochronous control to keep the engine speed constant regardless of an increase or decrease in the engine load. The fuel injection amount, fuel injection timing, boost pressure, and the like in the engine 11 are controlled by an engine control unit (ECU) D7.

[0032] A rotating shaft of the motor 11 is connected to the respective rotating shafts of a main pump 14 and a pilot pump 15 as hydraulic pumps. A control valve unit 17 is connected to the main pump 14 via a hydraulic oil line.

[0033] The control valve unit 17 is a hydraulic control device that controls a hydraulic system of the excavator 100. Hydraulic actuators such as the left and right travel hydraulic motors, the boom cylinder 7, the arm cylinder 8, the bucket cylinder 9, and the rotary hydraulic motor are connected to the control valve 17 via hydraulic oil lines. The rotary hydraulic motor may be a rotary motor generator.

[0034] Fig. 3 illustrates the connection between the controller 30 and the display device D3. In the present embodiment, the display device D3 is connected to the controller 30. The display device D3 and the controller 30 may be connected to each other via a communication network such as a CAN. In this embodiment, an example in which one display device D3 is provided is described, but the number of display devices provided in the cabin 10 is not limited, and multiple display devices may be provided.

[0035] The display device D3 includes a control unit D3a that generates an image. In the present embodiment, the control unit D3a generates a camera image for display based on an output from a camera as the imaging device S6. The imaging device S6 is connected to the display device D3, for example, via a dedicated line.

[0036] The control unit D3a generates an image for display based on the data output from the controller 30. In the present embodiment, the control unit D3a converts various types of information output from the controller 30 into an image signal. The information output from the controller 30 includes, for example, data indicating the temperature of the cooling water of the engine, data indicating the temperature of the hydraulic oil, data indicating the remaining amount of fuel, data indicating the remaining amount of urea water, data indicating the position of a work site of the bucket 6, data indicating the orientation of the slope of the work target, data indicating the orientation of the excavator 100, data indicating the operation direction for causing the excavator 100 to be aligned with the slope, and the like.

[0037] The control unit D3a may be implemented as a function of the controller 30 and not as a function of the display device D3. In this case, the imaging device S6 is connected to the controller 30 instead of the display device D3.

[0038] The display device D3 is powered by receiving power from a storage battery 70. The storage battery 70 is charged with electric power generated by a generator 11a (power generator) of the engine 11. The power of the storage battery 70 is supplied to the controller 30, the display device D3, and also to the electrical component 72 of the excavator 100. An engine starter 11b of the engine 11 is driven with electric power from the storage battery 70 to start the engine 11.

[0039] The engine 11 is controlled by an engine control unit D7. The engine control unit D7 constantly transmits various information indicating the state of the engine 11 to the controller 30. The various types of information indicating the state of the engine 11 are examples of the operating information of the excavator 100 and include, for example, information about the coolant temperature detected by the coolant temperature sensor 11c as an operating information detecting unit. The controller 30 stores the information in a temporary storage unit (memory) 30a and can transmit the information to the display device D3 as needed.

[0040] Various types of information are supplied to the controller 30 as operation information of the excavator 100 as described below and stored in the temporary storage unit 30a of the controller 30.

[0041] For example, data indicating the swash plate inclination angle is supplied from a controller 13 of the main pump 14, which is a variable displacement hydraulic pump, to the controller 30. A signal from a pressure sensor 14b indicating the discharge amount of the main pump 14 is also supplied to the controller 30. This information is stored in the temporary storage unit 30a. An oil temperature sensor 14c is provided in a pipe between the main pump 14 and a tank in which the hydraulic oil sucked by the main pump 14 is stored, and the oil temperature sensor 14c supplies the controller 30 with information indicating the temperature of the hydraulic oil flowing through the pipe. The controller 13, the pressure sensor 14b, and the oil temperature sensor 14c are examples of an operation information acquisition unit.

[0042] An operating device 26 is located near the operator's seat of the cab 10 and is used by the operator to operate various driven elements. Specifically, the operator can use the operating device 26 to operate hydraulic actuators such as the left and right travel hydraulic motors, the boom cylinder 7, the arm cylinder 8, the bucket cylinder 9, and the rotary hydraulic motor, allowing the operator to operate the drive elements to be driven by the hydraulic actuators. The operating device 26 includes a pedal device and a lever device for operating the respective driven elements.

[0043] The operation sensor 29 is configured to detect the content of an operation performed by the operator with the operation device 26. In the present embodiment, the operation sensor 29 detects the operation direction and operation amount of the operation device 26 corresponding to each of the hydraulic actuators and outputs an electrical signal (hereinafter also referred to as an operation signal) corresponding to the detected value to the controller 30. In the present embodiment, the controller 30 controls the opening area of a proportional valve 31 according to the output of the operation sensor 29. The controller 30 supplies the hydraulic oil discharged from the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17.The pressure of the hydraulic oil supplied to each of the pilot ports (pilot pressure) is, in principle, a pressure corresponding to the operating direction and amount of the operating device 26 for each of the hydraulic actuators. Thus, the operating device 26 is configured to supply the hydraulic oil discharged from the pilot pump 15 to the pilot port of the corresponding control valve in the control valve block 17. Thus, the hydraulic actuator can be driven.

[0044] Furthermore, the direction switching valve incorporated in the control valve unit 17 and controlling each hydraulic actuator may be an electromagnetic solenoid valve. In this case, the operation signal output from the operating device 26 can be directly input to the control valve unit 17 (i.e., the electromagnetic solenoid-type direction switching valve).

[0045] The operating device 26 may be a hydraulic pilot type. Specifically, the operating device 26 outputs a pilot pressure corresponding to the operation content to the secondary-side pilot line by using the hydraulic oil supplied from the pilot pump 15 via the pilot line. The secondary-side pilot line is connected to the control valve unit 17. Thus, the pilot pressure corresponding to the operation content of the various driven elements (hydraulic actuators) in the operating device 26 can be input to the control valve unit 17. Therefore, the control valve unit 17 can drive each hydraulic actuator in accordance with the operation content of the operating device 26 by the operator or the like.In this case, an operation sensor 29 capable of detecting information about an operating state of the operating device 26 is provided, and an output of the operation sensor 29 is input to the controller 30. Thus, the controller 30 can identify the operating state of the operating device 26. The operation sensor 29 is, for example, a pressure sensor that detects information related to a pilot pressure (operating pressure) of a pilot line on the secondary side of the operating device 26.

[0046] Furthermore, some or all of the hydraulic actuators may be replaced with electric actuators. In this case, the controller 30 may, for example, issue an operation command corresponding to the operation content of the operation device 26 or the remote control content of the electric actuator or a driver or the like that drives the electric actuator, as defined by the remote control signal. Furthermore, the electric actuator may be configured to be operated by the operation device 26 by inputting an operation signal from the operation device 26 to the electric actuator, the driver, or the like.

[0047] Furthermore, the operating device 26 may be omitted if the excavator 100 is operated exclusively by remote control or if the excavator 100 is operated exclusively with a fully automatic drive function.

[0048] The proportional valve 31 functions as a control valve for machine control and is provided for each driven element (hydraulic actuator) to be operated by the operating device 26 and for each operating direction of the driven element (hydraulic actuator) (for example, the raising direction and the lowering direction of the boom 4). For example, two proportional valves 31 are provided for each of the double-acting hydraulic actuators for driving the lower traveling body 1, the upper rotating body 3, the boom 4, the arm 5, the bucket 6, and the like. The proportional valve 31 may be provided, for example, in a pilot line between the pilot pump 15 and the control valve unit 17 and configured to change the flow path area (i.e., the cross-sectional area through which the hydraulic oil can flow).Accordingly, the proportional valve 31 can output a predetermined pilot pressure to the secondary-side pilot line by using the hydraulic oil of the pilot pump 15 supplied via the primary-side pilot line. Therefore, the proportional valve 31 can apply a predetermined pilot pressure to the control valve unit 17 according to the operation command from the controller 30. Therefore, for example, the controller 30 can cause the proportional valve 31 to directly supply the pilot pressure corresponding to the operation content (operation signal) of the operation device 26 to the control valve unit 17, and can perform the operation of the excavator 100 based on the operator's operation.

[0049] The controller 30 can control the proportional valve 31 to implement an automatic operation function of the excavator 100. Specifically, the controller 30 outputs an operation command corresponding to the automatic drive function from the proportional valve 31 to the proportional valve 31. Thus, the controller 30 can implement the operation of the excavator 100 through the automatic operation function.

[0050] The controller 30 controls the proportional valve 31 to implement the remote control of the excavator 100. Specifically, the controller 30 outputs an operation command corresponding to the operation content determined by the operation signal received from the remote control room RC to the proportional valves 31 via the communication device T1. Thus, the controller 30 causes the proportional valve 31 to supply the pilot pressure corresponding to the remote control content to the control valve unit 17 and can perform the operation of the excavator 100 based on the remote control by the operator.

[0051] In the case where the operating device 26 is a hydraulic pilot type, a shuttle valve may be provided in a pilot line between the operating device 26 and the proportional valve 31 and the control valve unit 17. The shuttle valve has two inlet ports and one outlet port, and outputs the hydraulic oil with the higher pilot pressure of the pilot pressures input to the two inlet ports to the outlet port. The shuttle valve is provided for each driven element (hydraulic actuator) to be operated by the operating device 26 and for each operating direction of the driven element (hydraulic actuator), as with the proportional valve 31. For example, two shuttle valves are provided for each double-acting hydraulic actuator for driving the lower traveling body 1, the upper rotating body 3, the boom 4, the arm 5, the bucket 6, and the like.One of the two inlet ports of the shuttle valve is connected to a pilot line on the secondary side of the operating device 26 (specifically, the above-described lever device or pedal device included in the operating device 26), and the other is connected to a pilot line on the secondary side of the proportional valve 31. The outlet port of the shuttle valve is connected via a pilot line to the pilot port of the corresponding directional control valve of the control valve unit 17. The corresponding direction switching valve is a direction switching valve that drives a hydraulic actuator, which is an actuation target of the above-described lever device or pedal device connected to an inlet port of the shuttle valve.Therefore, each of these shuttle valves can apply the higher of the pilot pressures of the pilot line on the secondary side of the operating device 26 and the pilot pressure of the pilot line on the secondary side of the proportional valve 31 to the pilot port of the corresponding direction switching valve. That is, the controller 30 can control the corresponding direction switching valve without depending on the operator's operation of the operating device 26 by outputting the pilot pressure higher than the pilot pressure on the secondary side of the operating device 26 from the proportional valve 31. Therefore, the controller 30 can control the operation of the driven elements (the lower traveling body 1, the upper slewing body 3, the boom 4, the arm 5, and the bucket 6) regardless of the operator's operation state on the operating device 26, and implement the automatic operation function and the remote control function.

[0052] Furthermore, in a case where the operating device 26 is a hydraulic pilot type, in addition to the shuttle valve, a pressure reducing valve may be provided in a pilot line between the operating device 26 and the shuttle valve. The pressure reducing valve is configured to be operated in response to a control signal input from, for example, the controller 30 and to be capable of changing its flow area. Thus, the controller 30 can forcibly reduce the pilot pressure output from the operating device 26 when the operating device 26 is operated by the operator. Therefore, the controller 30 can forcibly prohibit or stop the operation of the hydraulic actuator corresponding to the operation of the operating device 26 even when the operating device 26 is operated.Furthermore, even when the operating device 26 is operating, the controller 30 can, for example, reduce the pilot pressure output from the operating device 26 to a lower value than the pilot pressure output from the proportional valve 31 through the pressure reducing valve. Therefore, the controller 30 can reliably apply a desired pilot pressure to the pilot port of the direction changeover valve in the control valve unit 17, for example, regardless of the operation content of the operating device 26, by controlling the proportional valve 31 and the pressure reducing valve. Therefore, the controller 30 can more effectively realize the automatic operation function and the remote control function of the excavator 100 by controlling, for example, the pressure reducing valve in addition to the proportional valve 31.

[0053] The communication system of the excavator 100 according to the present embodiment includes the communication device T1.

[0054] The communication device T1 is connected to an external communication line and communicates with a device provided separately from the excavator 100. The device provided separately from the excavator 100 may include, in addition to the device outside the excavator 100, a portable terminal device (portable terminal) brought into the cab 10 by the user of the excavator 100. The communication device T1 may include a mobile communication module conforming to a standard such as 4G (4th generation) or 5G (5th generation). The communication device T1 may include, for example, a satellite communication module. The communication device T1 may include, for example, a Wi-Fi communication module or a Bluetooth (registered trademark) communication module.In addition, the communication device T1 may comprise a plurality of communication devices T1 depending on the type of communication line if there are a plurality of connectable communication lines.

[0055] For example, the communication device T1 communicates with an external device, such as a remote control center on the construction site, via a local communication line established at the construction site. The local communication line is, for example, a mobile communication line via a local 5G network (so-called local 5G) established at the construction site or a local network via Wi-Fi.

[0056] The communication device T1 is configured to send and receive information to and from a communication device installed in the remote control center via a communication line in a wide area encompassing the construction site, i.e. a wide area network.

[0057] In the present embodiment, a case will be described where the engine 11 is used as a drive source, and the hydraulic pump is driven by the drive force generated by the engine 11, thereby performing the operation of the attachment AT, the rotation operation of the upper revolving body 3, and traveling. However, in the present embodiment, the drive source is not limited to the engine 11, but a motor may be used as the drive source. That is, the control described in the present embodiment can be applied to a so-called electric excavator in which a motor serving as a drive source is driven by electric power supplied from a battery, or it can be applied to an excavator on which multiple drive sources are mounted. <Overview of the operation to be performed by the controller>

[0058] The controller 30 according to the present embodiment displays information regarding a position of a person detected from the image information acquired by the imaging device S6. Therefore, the controller 30 according to the present embodiment performs a process of detecting whether or not a person is near the excavator 100 using the image information acquired by the imaging device S6.

[0059] However, there is a situation where a person cannot be detected from the captured image information even though the person is near the excavator 100. For example, there is a situation where a person is too close to the excavator 100 and the person does not enter the imaging range of the imaging device S6 and thus cannot be detected from the captured image information. Furthermore, even if a person is present in the imaging range of the imaging device S6, the person may not appear to be detected due to light reflections, the setting sun, or the like. Furthermore, there is a situation where, even though a person is present in the imaging range of the imaging device S6, the person cannot be seen as if the person is detected because the amount of light is small and the image is blackened.

[0060] In such a situation, a person is not detected, but it is highly likely that the person is present. Therefore, when a person is no longer detected by the imaging device S6 (an example of a space recognition device), the controller 30 according to the present embodiment continues to display the position of the no longer detected person if a predetermined condition is met. <Blockkonfiguration von Steuerung von Bagger>

[0061] Fig. 4 is a functional block diagram showing an example of the configuration of the controller 30 of the excavator 100 according to the present embodiment. Fig. The example shown in Figure 4 shows a block configuration of the controller 30 of the excavator 100.

[0062] The controller 30 receives information output from the boom angle sensor S1, the arm angle sensor S2, the bucket angle sensor S3, the body inclination sensor S4, the rotation angle sensor S5, the imaging device S6, the input device D2, the communication device T1, the positioning device PS, and the like. Various calculations are then performed based on the received information and the information stored in the auxiliary storage device D4, and the calculation results are output to the display device D3, the proportional valve 31, and the like.

[0063] Note that in the present embodiment, an example is described in which the controller 30 controls the excavator 100, but part of the functions of the controller 30 may be implemented by another controller (control device). That is, the function of the controller 30 may be performed by a plurality of controllers mounted on the excavator 100 in a distributed manner.

[0064] The excavator 100 operates an actuator (for example, a hydraulic actuator) in response to the operation of an operator entering the cab 10, and drives operating members (hereinafter "driven members") such as the lower traveling body 1, the upper rotating body 3, the boom 4, the arm 5, and the bucket 6.

[0065] Furthermore, the excavator 100 may be configured to be remotely controlled from the outside, instead of or in addition to being configured to be operable by the operator of the cabin 10. When the excavator 100 is remotely controlled, the interior of the cabin 10 may be unmanned.

[0066] The auxiliary storage device D4 stores a trained model LM.

[0067] When the image information acquired by the imaging device S6 is input from the input layer, the trained model LM outputs image information indicating an image in which a person depicted in the acquired image information is surrounded by a rectangle. In the present embodiment, an example of an output mode of the trained model LM will be described, but the present embodiment is not limited to the mode of outputting image information in which a person is surrounded by a rectangle, and may be any output mode insofar as a detection result of a person depicted in the acquired image information can be displayed.

[0068] For example, a neural network can be used as the machine learning method used to generate the trained model (LM). In particular, deep learning, i.e., machine learning using a deep neural network (DNN), can be used. For example, a convolutional neural network, a recurrent neural network (RNN), or a long-short-term memory (LSTM) can be used as deep learning.

[0069] The trained model LM is generated by machine learning based on a training data set generated in advance in an information processing device (not shown).

[0070] In particular, the trained model LM is generated by machine learning based on captured image information depicting a person and image information in which the person depicted in the captured image information is surrounded by a rectangle, which comprise the training dataset.

[0071] The trained model LM can be updated by additionally training the existing trained model LM with a new training dataset.

[0072] The controller 30 includes a sensing unit 301, a detection unit 302, a position estimation unit 303, an output control unit 304, and a determination unit 305.

[0073] The acquisition unit 301 acquires various types of information from various sensors. For example, the acquisition unit 301 acquires image information captured by the imaging device S6 (the front camera S6F, the left camera S6L, the right camera S6R, and the rear camera S6B).

[0074] The detection unit 301 detects detection information detected by the boom angle sensor S1, the arm angle sensor S2, the bucket angle sensor S3, the body inclination sensor S4, and the rotation angle sensor S5. The detection unit 301 detects the position and orientation of the excavator 100 from the positioning device PS.

[0075] The detection unit 302 performs a process of detecting a person located around the excavator 100 from the image information acquired by the acquisition unit 301. The detection unit 302 according to the present embodiment inputs the acquired image information into the trained model LM, thereby obtaining image information in which the person depicted in the acquired image information is surrounded by a rectangle of the trained model LM. In the present embodiment, a method for detecting a person using the trained model LM will be described. However, the present embodiment does not limit the method for detecting a person, and any known methods can be used.For example, it may be determined whether or not the feature extracted from the acquired image information resembles a predetermined feature indicative of a person by a predetermined value or more.

[0076] When an area surrounded by a rectangle exists in the image information received by the detection unit 302, the position estimation unit 303 estimates the position of the person in the real space, for example, the direction and distance of the person relative to the excavator 100, based on the position coordinates where the rectangle is displayed in the image information. A specific estimation method may be a conventionally used method, and thus, the description thereof may be omitted.

[0077] The position estimation unit 303 according to the present embodiment uses a method of estimating the direction and distance of a person based on the position and size of a rectangle in which the person appears in the image information, but another method may also be used. For example, a method in which the trained model LM outputs the direction and distance of the person may be used.

[0078] It should be noted that the present embodiment is not limited to the method of detecting a person and estimating the position of the person by the controller 30, and that, for example, the detection and estimation may be performed in the imaging device S6 or an external cloud service may be used.

[0079] The output control unit 304 outputs the rotation angle, image information about an area where a person surrounded by a rectangle is detected, position coordinates of a person around the excavator 100 (information about the direction in which the detected person is located and the distance to the person), and the detection results of the various sensors to the control unit D3a of the display device D3. Thus, the display device D3 displays a screen showing the surroundings of the excavator 100.

[0080] Next, an example of a screen displayed on the display device D3 will be described with reference to Fig. 5 described. Fig. 5 is a diagram showing an example of a display screen 42 displayed by the display device D3 according to the present embodiment.

[0081] The control unit D3a according to the present embodiment generates a display screen based on the image information input from the imaging device S6 and various types of information received from the controller 30. The information received by the controller 30 includes the rotation angle, image information in which an area where a person is photographed is enclosed by a rectangle, position coordinates of a person present around the excavator (information indicating a direction and distance where the detected person is located), and detection results from various sensors.

[0082] The display screen 42 displays, under the control of the control unit D3a, a date and time display area 42a, a travel mode display area 42b, an attachment display area 42c, a fuel consumption display area 42d, an engine control state display area 42e, an engine operating time display area, a coolant temperature display area 42g, a remaining fuel amount display area 42h, a rotational speed display area 42i, a remaining urea water amount display area 42j, a hydraulic oil temperature display area 42k, a human detection map display area 421, a first image display area 422, and a second image display area 423. The display screen 42 may include other display areas.

[0083] The travel mode display area 42b, the attachment display area 42c, the engine control state display area 42e, and the speed level display area 42i are areas for displaying setting state information, that is, information about the setting state of the excavator 100. The fuel consumption display area 42d, the engine operating time display area 42g, the remaining fuel amount display area 42h, the remaining urea-water amount display area 42j, and the hydraulic oil temperature display area 42k are areas for displaying operating state information, that is, information indicating the operating state of the excavator 100 based on the detection results of the various sensors.

[0084] The date and time display area 42a is an area for displaying the current date and time. The drive mode display area 42b is an area for displaying the current drive mode. The attachment display area 42c is an area for displaying an image representing the currently attached attachment. The fuel consumption display area 42d is an area for displaying fuel consumption information calculated by the controller 30. The fuel consumption display area 42d includes an average fuel consumption display area 42d1 for displaying an average lifetime fuel consumption or an average interval fuel consumption, and an instantaneous fuel consumption display area 42d2 for displaying an instantaneous fuel consumption.

[0085] The engine control state display area 42e is an area for displaying the control state of the engine 11. The coolant temperature display area 42g is an area for displaying the current temperature state of the engine coolant. The remaining fuel amount display area 42h is an area for displaying the remaining fuel amount in the fuel tank.

[0086] The speed level display area 42i is an area in which the current level set with a rotary knob 32 (not shown) is displayed as an image. A number indicating the selected level is displayed in the speed display area 42i. The number "1" displayed in the speed display area 42i means that the selected speed level is the "first level." The number "n" displayed in the speed display area 42i means that the selected speed level is the "nth level." Note that "n" is a natural number. When the user rotates the rotary knob 32, the number displayed in the speed display area 42i changes.

[0087] The remaining urea water amount display area 42j is an area where the remaining amount of urea water stored in the urea water tank is displayed as an image. The hydraulic oil temperature display area 42k is an area for displaying the temperature state of the hydraulic oil in the hydraulic oil tank.

[0088] Human detection map display area 421 is an area for displaying information indicating a positional relationship between the excavator 100 and a human (person) detected around the excavator 100.

[0089] The human detection map display area (an example of position information) 421 is a map display area in which a real space around the excavator 100 is displayed at a predetermined scale. In the human detection map display area 421, an excavator icon 421b indicating the presence of the excavator 100 is arranged at the center of the area.

[0090] In the human detection map display area 421, in addition to the excavator icon 421b representing the excavator 100, a symbol (direction display icon 421a in the example of Fig. 5), which indicates the direction in which the excavator 100 can move, and symbols (human detection symbols 421e and 421f in the example of Fig. 5) representing people detected in the vicinity of the excavator 100 are displayed simultaneously. In the human detection map display area 421, an area other than the excavator icon 421b, the direction indicator icon 421a, and the human detection icons 421e and 421f may be an area represented by a single color (for example, black).

[0091] The excavator symbol 421b is a symbol determined by combining an image indicating the upper rotary body 3 and an image indicating the lower traveling body 1 according to the positional relationship between the upper rotary body 3 and the lower traveling body 1 based on the rotation angle.

[0092] The direction indicator symbol 421a indicates the direction in which the excavator 100 moves when the travel lever is tilted forward with a triangular shape. Note that the present embodiment illustrates an example of the symbol indicating the direction in which the excavator 100 moves when the travel lever is tilted forward, and the symbol can have any shape as long as the shape indicates the direction in which the excavator 100 can move.

[0093] The human detection symbols (an example of display information) 421e and 421f are symbols indicating the positions where people are located, which are estimated by the position estimation unit 303. Specifically, the human detection symbols 421e and 421f are arranged based on the directions and distances of the people with respect to the excavator 100, which are estimated by the position estimation unit 303. Specifically, the human detection symbols 421e and 421f are arranged at positions determined by multiplying the directions and distances of the detected people with respect to the excavator 100 by a predetermined scale factor.

[0094] In this way, the positional relationship between the excavator icon 421b and the human detection icons 421e and 421f displayed in the human detection map display area 421 corresponds to the positional relationship between the excavator 100 in the real space and the people around the excavator 100.

[0095] In the present embodiment, an example will be described in which the human detection map display area 421 is used as the positional information representing the positional relationship between the person detected by the imaging device S6 and the excavator 100 as a diagram. However, in the present embodiment, the positional information representing the relationship between the positions of the person detected by the imaging device S6 and the excavator 100 is not limited to the human detection map display area 421. For example, positional information representing a positional relationship can be obtained by superimposing the positions of the person and the excavator 100 detected by the imaging device S6 on the bird's-eye view image information based on the image information acquired by the imaging device S6.As described above, the position information may be information that can recognize the positional relationship between the excavator and the person.

[0096] Since the human detection map display area 421 according to the present embodiment prohibits the display of an object other than the person present around the excavator 100, the operator can recognize the current situation and the direction in which the excavator 100 can travel, as well as the positional relationship with the person present around the excavator 100 by referring to the human detection map display area 421.

[0097] Furthermore, by referring to the human detection map display area 421, the operator can estimate how the positional relationship between the excavator 100 and the person present in the surrounding area changes as the excavator 100 moves. Since the display of the excavator and an object other than the person is prevented in the human detection map display area 421, the operator can be prevented from focusing on other objects and forgetting the presence of the person. Therefore, safety can be improved.

[0098] Furthermore, in the human detection map display area 421, a first circular area 421c and a second circular area 421d are displayed, which are determined based on distances to the excavator 100, with the excavator 100 serving as a reference.

[0099] The first circular region 421c and the second circular region 421d, which are in Fig. 5 are shown as circles to allow the user to identify relative distances from the excavator 100 with reference to the excavator symbol 421b.

[0100] The first circular area 421c is, for example, information indicating an area within 2 m of the excavator 100. The second circular area 421d is, for example, information indicating an area of 4 m around the excavator 100.

[0101] The display screen 42 according to the present embodiment can recognize the position of the person with respect to the excavator 100 based on the positional relationship between the first circular area 421c and the second circular area 421d and the human detection icons 421e and 421f.

[0102] In addition, the control unit D3a changes the display mode of the human detection symbols 421e and 421f depending on whether the human detection symbols 421e and 421f include the first circular area 421c and the second circular area 421d.

[0103] For example, the human detection symbol 421e within the first circular area 421c is displayed in red. The human detection symbol 421f, which is located outside the first circular area 421c and within the second circular area 421d, is displayed in yellow. The human detection symbol (not shown) located outside the second circular area 421d is displayed in green, for example.

[0104] The control unit D3a according to the present embodiment changes the display mode depending on whether the human detection symbol includes the first circular area 421c and the second circular area 421d. Thus, the display device D3 displays the human detection symbols whose colors change according to the distances, thereby attracting the operator's attention according to the distances between the excavator 100 and the people. This can improve safety.

[0105] The color of the human detection icon in the present embodiment is merely an example, and there is no limitation on using colors as above. For example, the human detection icon may be displayed in grayscale. The color change of the human detection icon in the present embodiment is an example of the display mode, and the present invention is not limited to color change. For example, the blinking cycle of the human detection icon may be changed, or the contrast or brightness may be changed depending on whether the human detection icon includes the first circular area 421c and the second circular area 421d.

[0106] On the display screen 42, the human detection map display area 421 is displayed, and the image information captured by the imaging device S6 is displayed. The operator can recognize a specific situation around the excavator 100 by checking the image information together with the human detection map display area 421. Therefore, safety can be improved.

[0107] In the Fig. In the display screen 42 shown in FIG. 5, the first image display area 422 and the second image display area 423 are areas for displaying image information acquired by the imaging device S6, and include an image 423c of the counterweight of the upper revolving body 3. The first image display area 422 displays a right image. A rear image is displayed in the second image display area 423. The right image is an image showing a space on the right side of the excavator 100 and includes an image 422c of the right end of the upper surface of the upper revolving body 3.

[0108] The right image is image information output by the trained model LM, and is image information in which a person depicted in the real viewpoint image captured by the right camera S6R is surrounded by a rectangle. The rear image is image information output by the trained model LM, and is image information in which a person depicted in the real viewpoint image captured by the rear camera S6B is surrounded by a rectangle.

[0109] As a result, a frame 422b is displayed around the person 422a in the right image in the first image display area 422 and a frame 423b is displayed around the person 423a in the rear image in the second image display area 423.

[0110] The first image display area 422 is displayed on the right side with respect to the human detection map display area 421. The second image display area 423 is displayed below the human detection map display area 421. In the present embodiment, the top side of the display screen 42 corresponds to the front side of the upper rotating body 3. In other words, the second image display area 423 is displayed at a position corresponding to the back side with respect to the human detection map display area 421. That is, the display screen 42 displays the image information acquired by the imaging device S6 in the direction in which the imaging device S6 performs imaging with respect to the human detection map display area 421.In the present embodiment, since the acquired image information is displayed in the direction in which the image is acquired with respect to the human detection map display area 421, the operator can intuitively recognize the direction in which the image information represents the situation when referring to the image information. Therefore, safety can be improved.

[0111] Note that the present embodiment shows an example of the arrangement of image information, and the present invention is not limited to this arrangement. For example, the first image display area 422 and the second image display area 423 may be arranged regardless of the direction in which the image is captured.

[0112] The control unit D3a matches the color of the frame 422b of the right image of the first image display area 422 with the color of the human detection symbol 421e, and matches (an example of “associated”) the color of the frame 423b of the rear image of the second image display area 423 with the color of the human detection symbol 421f.

[0113] That is, on the display screen 42, the frame indicating the detected person is displayed with respect to the right image and the rear image, and the correspondence relationship between the person indicated by the frame and the human detection icon is displayed in a recognizable manner. Accordingly, the operator can recognize the situation of the person displayed in the human detection map display area 421 from the right image and the rear image. Therefore, the operator can operate the excavator 100 taking into account the situation of the person near the excavator 100. This can improve safety. In the present embodiment, the color of the frame is matched with the color of the human detection icon as an example of the display for recognizing the correspondence relationship.However, the present embodiment is an example of a display that prompts the user to recognize the correspondence relationship, and is not limited to the example of matching the color of the frame with the color of the human detection icon. For example, the correspondence relationship can be recognized by a blinking cycle of the frame and the human detection icon.

[0114] As described above, the color of the human detection icon changes depending on the distance from the excavator 100. The color of the human detection icon in the human detection map display area 421 is matched with the colors of the frames displayed in the right image and the rear image. Therefore, the display screen 42 displays the right image and the rear image with the frames in a different color (an example of a display mode) based on the distance between the excavator 100 and the person. In the present embodiment, the colors of the frames displayed in the right image and the rear image are changed according to the distance from the excavator 100, so that the operator can recognize the distance from the excavator 100 by referring to the color of the frame. Thus, the operator can perform an operation according to the distance, and thus, it is possible to improve safety.

[0115] In the present embodiment, the control unit D3a displays the screen 42, and thus the user can identify the situation around the excavator 100.

[0116] Back to Fig. 4, the determination unit 305 compares the direction and distance of the person estimated from the image information previously received by the trained model LM with the direction and distance of the person estimated from the image information currently received by the trained model LM, and determines whether or not the previously detected person is no longer detected at the current time.

[0117] Then, if it is determined that the previously detected person is no longer detected at the current time, the determination unit 305 determines whether or not the positional relationship between the detection area in which a person can be detected from the acquired image information by the imaging device S6 and the position of the no longer detected person satisfies a predetermined condition. The conditions of the present embodiment are described below.

[0118] Fig. 6 is a diagram illustrating a method for determining whether or not a person is around the excavator 100 according to the positional relationship between the excavator 100 and the person in the determination unit 305 according to the present embodiment. Fig. The detection area 1600 shown in Fig. 6 indicates an area in which a person can be detected around the excavator 100 by the front camera S6F, the rear camera S6B, the left camera S6L, and the right camera S6R, with the excavator 100 as the center. In other words, the detection area 1600 is a part of an area formed by the combination of the Fig. 2, the imaging area AF, the imaging area AB, the imaging area AL and the imaging area AR, and is an area in which a person can be detected from the acquired image information by the imaging device S6.

[0119] In the present embodiment, the detection area 1600 is divided into three areas for determination by the determination unit 305. In the present embodiment, the detection area 1600 is divided from the inside into a first area 1601, a second area 1602, and a third area 1603.

[0120] The first area 1601 is an area within a 2 m radius of the excavator 100. The second area 1602 is an area outside the first area 1601 and within 4 m of the excavator 100. The third area 1603 is an area outside the second area 1602 and within 5 m of the excavator 100.

[0121] In the present embodiment, an example will be described in which the first area 1601 corresponds to the first circular area 421c of the human detection map display area 421, and the second area 1602 corresponds to the second circular area 421d of the human detection map display area 421. However, the above-described division of the detection area 1600 is not limited to the method corresponding to the display of the human detection map display area 421, and the division of the detection area 1600 may be used only for internal processing.

[0122] If it is then determined that the previously detected person is no longer detected at the current time, the determination unit 305 causes the determination result to depend on which area of the first area 1601 to the third area 1603 the person is located in.

[0123] The third area 1603 lies outside the detection area 1600. For example, if the person 1631 is present in the third area 1603, a person 1631 can immediately move out of the detection area 1600 by moving in the direction of the arrow 1632. Therefore, if the person who was in the third area 1603 is no longer detected at the current time, the determination unit 305 can determine that the person has moved outside the detection area 1600. That is, the determination unit 305 can determine that the person is no longer present.

[0124] The second area 1602 is neither near the excavator 100 nor in an area outside the detection area 1600. For example, if the person 1621 is present in the second area 1602, it is difficult for the person 1621 to move out of the detection area 1600 without being detected in the first area 1601 or the third area 1603. Therefore, if a person who has been present in the second area 1602 is not detected at the current time, the determination unit 305 determines that the person is still present in the detection area 1600 and the person is not detected at the current time because a detection error has occurred.

[0125] The first area 1601 is an area near the excavator 100. For example, if the person 1611 is located in the first area 1601, the person 1611 may move under the lower traveling body 1 or get stuck on the excavator 100. That is, there is a high probability that a person is near the excavator 100 even though the person is not located in the detection area 1600. The reason why the person is still located in the detection area 1600 and is no longer detected at the current time is that a detection error may have occurred. Therefore, if the person who was located in the first area 1601 is no longer detected at the current time, the determination unit 305 determines that at least the person is near the excavator 100.

[0126] That is, when the person whose position is displayed in the human detection map display area 421 is no longer detected by the imaging device S6, the determination unit 305 determines that the person is near the excavator 100 if the last detected position of the person is not a position where the person can move outward from the detection area 1600, such as the first area 1601 or the second area 1602, or the last detected position of the person is a position where the person can move outward from the detection area 1600 toward the excavator side.

[0127] Note that the determination by the determination unit 305 in the present embodiment is an example that uses a determination condition based on the positional relationship between the detection range in which a person can be detected from the acquired image information by the imaging device S6 and the position of a person who is no longer detected, and is not limited to the determination based on the determination condition. For example, the determination unit 305 may determine whether or not the position of the person who is no longer detected is near the boundary of the detection range in which the person can be recognized by the imaging device S6 from the acquired image information. Furthermore, the determination unit 305 may determine whether or not the person has moved out of the detection range, taking into account the moving speed of the person.The predetermined condition for the determination is not limited to the determination condition described above, and the predetermined condition may be determined according to a situation regarding the brightness of the construction site, the moving speed of the person, the capability of the imaging device S6, the resolution of the acquired image information by the imaging device S6, and other implementation conditions.

[0128] In the present embodiment, a determination condition based on a positional relationship between a detection area in which a person can be detected from the image information acquired by the imaging device S6 and a position of a person no longer detected is used. Therefore, the determination by the determination unit 305 according to the present embodiment is a determination taking the positional relationship into account, rather than a determination using a new sensor or the like, and thus it is possible to improve detection accuracy while reducing costs.

[0129] The output control unit 304 outputs the determination result of the determination unit 305 to the control unit D3a of the display device D3. When the person whose position is displayed in the human detection map display area 421 is no longer detected by the imaging device S6 from the acquired image information, and the determination unit 305 determines that the person is near the excavator 100 (an example of a case where the predetermined condition is satisfied), the control unit D3a continues to control the position of the person who is no longer detected in the human detection map display area 421.

[0130] Fig. Fig. 7 is a diagram showing an example of a display screen 42A displayed by the display device D3 according to the present embodiment. Fig. The display screen 42A shown in Figure 7 is the same components as in the Fig. 5 are indicated by the same symbols, and the description thereof is omitted.

[0131] In the Fig. 7, the right image displayed in a first image display area 422A is not detected by the trained model LM because a person 1422a is moving.

[0132] Even if the person in the right image is no longer detected, in other words, even if the display of the frame surrounding the person (an example of an area where the person is detected) is inhibited in the first image display area 422A, the display device D3 continues to display the human detection icon 1421e corresponding to the person 1422a in the human detection map display area 421A together with the first image display area 422A.

[0133] Therefore, even if the rectangle surrounding the person is not displayed in the first image display area 422A when the user accesses the display screen 42A, the user can estimate that the person is in the correct proximity to the excavator 100 by referring to the human detection icon 1421e displayed in the human detection map display area 421A. As described above, in the present embodiment, the human detection map display area 421A and the first image display area 422A are simultaneously displayed on the display screen 42A.

[0134] Then, according to the present embodiment, when the determination unit 305 determines that a person is near the excavator 100, the controller 30 continues to retain the direction and distance at which the person was last detected. Subsequently, the determination unit 305 repeatedly determines whether the previously detected person is currently detected based on the retained information.

[0135] When the detection unit 302 then again detects a person from the image information acquired by the imaging device S6, the determination unit 305 determines that the previously detected person is recognized again at the current time.

[0136] In this case, the display device D3 moves the human detection icons to positions corresponding to the directions and distances of the newly detected people without increasing the number of human detection icons displayed in the display area of the person map. Subsequently, the controller 30 deletes the retained information about the last detected direction and distance.

[0137] In the controller 30 according to the present embodiment, the detection situation is initialized once upon power off. That is, the controller 30 clears the information about the direction and the last detected distance that have been retained. The initialization process according to the present embodiment can also be performed in cases other than power off, such as when the upper rotating body 3 is rotating or when the lower traveling body 1 is traveling.

[0138] Next, a processing operation performed by the controller 30 and the display device D3 according to the present embodiment will be described. Fig. 8 is a flowchart illustrating a processing operation until the controller 30 and the display device D3 according to the present embodiment display a display screen. The processing operation illustrated in the flowchart according to the present embodiment is repeatedly performed every predetermined cycle.

[0139] First, the acquisition unit 301 acquires the image information captured by the imaging device S6 (S1801).

[0140] The detection unit 302 detects a person present near the excavator 100 based on the image information acquired by the acquisition unit 301 (S1802). In the present embodiment, the detection unit 302 inputs the acquired image information to the trained model LM and receives image information in which the person depicted in the acquired image information is surrounded by a rectangle of the trained model LM, thereby detecting the person present around the excavator 100.

[0141] The position estimation unit 303 estimates the direction and distance of the present person with respect to the excavator 100 based on the position coordinates of the person (the rectangle) in the acquired image information (S1803).

[0142] The determination unit 305 compares the direction and distance of the person estimated in the previous step S1803 with the direction and distance of the person estimated in the current step S1803 and determines whether the previously detected person is not detected at the current time (S1804).

[0143] When the determination unit 305 determines that the previously detected person is also detected at the current time (S1804: NO), the display device D3 displays the display screen including the person detection map display area in which the position of the detected person is displayed (S1805).

[0144] If the determination unit 305 determines that the previously detected person is not currently detected (S1804: YES), the determination unit 305 determines whether the position of the person at the time of the previous detection is within the first area or the second area of the detection area (S1806). If the determination unit 305 determines that the position is outside the first area and the second area, that is, within the third area (S1806: NO), the display device D3 displays the screen including the person detection map display area, in which the position of the detected person is displayed (S1805).

[0145] On the other hand, when the determination unit 305 determines that the position of the person at the time of the previous detection is within the first area or the second area (S1806: YES), the display device D3 displays the display screen including the person detection map display area in which the person detection icon at the previously detected position of the person is displayed (S1807).

[0146] In the present embodiment, by performing the above-described control, even if a person is no longer detected from the acquired image information and a predetermined condition is met, an area is displayed on the person detection map in which the person detection symbol is displayed at the previously detected position of the person. Therefore, even if a detection error occurs and the person is not included in the image area because the person is approaching the excavator 100, the operator can recognize that the person is present.

[0147] The display device D3 according to the present embodiment has been described using an example in which, when a previously detected person is no longer detected, a display screen including the same display area of the person detection map as the previous one is displayed. However, in the present embodiment, the display mode of the screen including the display area of the person detection map is not limited, and various display modes can be used.

[0148] Fig. 9A to 9C are diagrams illustrating a change in the display area for human detection displayed by the display device D3.

[0149] Fig. 9A shows a modification in which, when the determination unit 305 determines that a previously detected person is currently no longer detected, an area is displayed in which the currently no longer detected person is likely to be present. Fig. In the person detection map display area 1901 shown in FIG. 9A, a person detection symbol 1911a corresponding to a previously detected position of the person is displayed, and an area 1911b in which the person is likely to be present is displayed. The area 1911b may be displayed such that the area 1911b expands over time, taking into account the speed at which the person in question is moving.

[0150] In the Fig. The human detection map display area 1901 shown in FIG. 9A displays the area where the person is likely to be located to the operator, and thus the operator can recognize the area in which the person is likely to be located. Therefore, safety can be improved.

[0151] Fig. 9B shows a modification example in which, when the determination unit 305 determines that a previously detected person is no longer detected at the current time, the display mode of the person detection icon is changed with respect to the person no longer detected at the current time. Fig. In the person detection map display area 1902 shown in FIG. 9B, a person detection icon 1921 corresponding to a previously detected position of the person is displayed. The person detection icon 1921 is diamond-shaped and differs in shape from the person detection icon of the detected person (for example, the person detection icon 1922).

[0152] Therefore, if the determination unit 305 determines that a previously detected person is no longer detected at the current time, the display device D3 changes the person detection icon display mode to allow the operator to recognize the presence of the person not currently detected. It should be noted that the display information representing the position of the person may be different from the person detection icon and may be represented, for example, by an area as shown in Fig. 9C can be displayed.

[0153] Fig. 9C illustrates a modification of the person detection card display area 1903 displayed by the display device D3. In the Fig. In the human detection map display area 1903 shown in FIG. 9C, a circular area 1930 representing the perimeter of the excavator 100 is displayed along with an excavator symbol 1933 instead of displaying the human detection symbol. The circular area 1930 is divided into predetermined areas, and whether or not a person is present in each divided area is indicated by a color.

[0154] Further, when the determination unit 305 determines that the previously detected person is not detected at the current time, the area where the previously detected person was present is displayed in a different color than the area where no person is detected.

[0155] For example, an area 1931 indicates an area of a person who was previously detected but is not detected at the current time.

[0156] The display device D3 shows the area 1931, and thus the operator can recognize that there is a person who is not detected at the current time.

[0157] In the Fig. In the human detection map display area 1903 shown in FIG. 9C, the area (for example, area 1931) where the previously detected person is not detected at the current time and the area (for example, area 1932) where the person is detected at the current time can be displayed in different colors. Therefore, the operator can recognize the current situation in detail based on the color difference when referring to the human detection map display area 1903.

[0158] The example in which the Fig. The human detection map display area 1903 shown in FIG. 9C displays the human detection result for 360 degrees around the excavator icon 1933 as a reference has already been described. However, the present modification is not limited to the example in which the human detection result is displayed within a radius of 360 degrees around the excavator icon 1933 as a reference. For example, the human detection result may also be displayed within a radius of 180 degrees to the rear or 270 degrees to the rear and to the right.

[0159] The present embodiment is not limited to continuing the display of the above-described person detection map display area in a case where a previously detected person is not detected at the current time. For example, in addition to continuing the display of the person detection map display area, the controller 30 may display a pop-up screen to attract attention. The pop-up screen is displayed, for example, as follows: "It may be that the information that the previously detected person has not been detected at the current time has been overlooked, so please check your surroundings." The pop-up screen is displayed so that it does not overlap the first image display area 422 and the second image display area 423. Thus, the operator can recognize the surrounding situation and then identify that no person has been detected. (Second embodiment)

[0160] In the above-described embodiment, the example in which a person is detected from the image information acquired by the imaging device S6 was described. However, the above-described embodiment is not limited to the example of detecting a person from the image information acquired by the imaging device S6. Therefore, in a second embodiment, a case in which a space recognition device S7 is provided in addition to the imaging device S6 will be described.

[0161] Fig. 10 is a side view of an excavator 100A according to the second embodiment. Fig. 11 is a plan view of the excavator 100A according to the second embodiment. In this embodiment, the same reference numerals are assigned to the same components as in the first embodiment, and the description thereof is omitted.

[0162] The space detection device S7 is an example of a space detection device and detects the presence or absence of an object in the space around the excavator 100A, the distances to the object, and the like. The space detection device S7 outputs a measurement result of the space to a controller 30A as measurement information.

[0163] The space detection device S7 includes a rear space detection device S7B that detects a space behind the excavator 100A, a left space detection device S7L that detects a space to the left of the excavator 100A, a right space detection device S7R that detects a space to the right of the excavator 100A, and a front space detection device S7F that detects a space in front of the excavator 100A.

[0164] The spatial detection device S7 may use a LIDAR to detect an object located near the excavator 100A. For example, the LIDAR measures the distances between the LIDAR and one million or more points within a monitoring area. Note that the present embodiment is not limited to a configuration using a LIDAR, and any spatial detection device capable of measuring the distance to an object may be used. For example, a stereo camera or a distance measuring device such as an imaging device or a millimeter-wave radar may be used.When the millimeter-wave radar or the like is used as the space detection device S7, a large number of signals (laser light or the like) can be transmitted from the space detection device S7 to the object, and the reflection signals can be received to derive the distances and directions of the object from the reflection signals.

[0165] The rear space detection device S7B is mounted on the rear end of the upper surface of the upper rotating body 3. The left space detection device S7L is mounted on the left end of the upper surface of the upper rotating body 3. The right space detection device S7R is mounted on the right end of the upper surface of the upper rotating body 3. The front space detection device S7F is mounted on a front end of a top surface of the cab 10.

[0166] The front space detection device S7F, the rear space detection device S7B, the left space detection device S7L, and the right space detection device S7R are all mounted on the upper rotating body 3 such that their optical axes are directed obliquely downward, and a part of the upper rotating body 3 is included in the detection range. Therefore, the detection range of each of the front space detection device S7F, the rear space detection device S7B, the left space detection device S7L, and the right space detection device S7R has a viewing angle of, for example, approximately 180 degrees in a plan view. In the example of Fig. 11, a detection area SF represents an example of a detection area of the front room detection device S7F, a detection area SB represents an example of a detection area of the rear room detection device S7B, a detection area SL represents an example of a detection area of the left room detection device S7L, and a detection area SR represents an example of a detection area of the right room detection device S7R.

[0167] As in Fig. As shown in Figure 11, the detection areas SF, SB, SR and SL are narrower than the imaging areas AF, AB, AR and AL, but the detection and imaging areas are different depending on the embodiment.

[0168] The controller 30A according to the present embodiment can perform the same processing as the controller 30 of the above-described embodiment, and differs from the controller 30 only in that the detection result of the space recognition device S7 is used to estimate the position of the person in the real space, for example, the direction and distance at which the person is located with respect to the excavator 100A.

[0169] The controller 30A according to the present embodiment estimates the position of a person in the real space, for example, the direction and length of the person relative to the excavator 100A, based on the detection result of the space recognition device S7. As a method for estimating the direction and distance of the person from the detection result, a known method, such as a trained model, can be used. For example, the controller 30A can obtain a direction and distance at which a person is located by inputting a detection result of the space recognition device S7 to the trained model.

[0170] The display device D3 according to the present embodiment displays on the screen a person map display area based on the result of estimation of the direction and distance to the person by the controller 30A. The information displayed in the other display area is the same as in the above-described embodiment, and thus the description thereof is omitted.

[0171] That is, the display device D3 according to the present embodiment is configured to display the image information acquired by the imaging device S6 provided in the upper rotating body 3 separately from the space recognition device S7 together with the person detection map display area.

[0172] The determination unit 305 according to the present embodiment is not limited to the determination based on the detection result by the space recognition device S7, and may perform the determination by combining the detection result by the space recognition device S7 and the image result by the imaging device S6. In the determination, a difference between the detection areas SF, SB, SR, and SL and a detection area in which a person can be detected from the acquired image information by the imaging device S6 may be taken into account. For example, when the determination unit 305 determines that a person who is no longer detected by the space recognition device S7 is detected by the imaging device S6, the display device D3 may display the position of the person in the person detection map display area. (Third embodiment)

[0173] In the above-described embodiment, the case where the work is performed by the excavator 100 with an operator on board was described. However, the above-described embodiment is not limited to the example of performing the work when the operator is on board the excavator 100. For example, when the excavator 100 performs work according to remote control, the same display as in the above-described embodiment can be performed. Therefore, in a third embodiment, a case where the excavator 100 is remotely controlled will be described.

[0174] Therefore, an outline of a remote control system SYS according to the third embodiment will be described with reference to Fig. 12 described. Fig. 12 is a schematic diagram showing an example of a remote control system SYS according to the third embodiment.

[0175] As in Fig.12, the remote control system SYS according to the third embodiment includes the excavator 100, a fixed point camera 1201, and a remote control room RC.

[0176] The excavator 100 and the remote control room RC are connected via a communication line NW so that they can send and receive data.

[0177] The excavator 100 can perform wireless communication using the communication device T1. The excavator 100 can send and receive data to and from a device connected to the communication line NW (for example, the remote control center RC).

[0178] Then, the excavator 100 can transmit information about a construction site to the remote control room RC. Thus, the remote control room RC can inspect the construction site according to the information from the excavator 100. In the present embodiment, a device that measures the construction site is not limited to the excavator 100 and may be a device of another aspect, such as a drone that flies over the construction site, a fixed-point camera, or an imaging device that can be worn by the user.

[0179] The excavator 100 is equipped with, for example, an imaging device S6. The excavator 100 transmits image information, which is acquired by the imaging device S6 and indicates an imaging result of the construction site, to the remote control room RC. Alternatively, the fixed-point camera 1201 provided at the construction site transmits image information, which is acquired, indicating an imaging result of the construction site to the remote control room RC. The device that monitors the construction site is not limited to the fixed-point camera 1201, but may be a drone that flies over the construction site or an imaging device that can be worn by the user. The drone or the imaging device can transmit the acquired image information, which indicates an imaging result of the construction site, to the remote control room RC.

[0180] The number of excavators 100 included in the remote control system SYS may be one or more. Accordingly, the remote control system SYS may provide information about the construction site to the remote control room RC via the plurality of excavators 100. < Remote control room configuration example>

[0181] The remote control room RC includes a communication device T2, a control device R30, an operating device R26, an operation sensor R29, and a display device DR. In addition, an operator seat DS is installed in the remote control room RC, on which an operator OP sits to remotely control the excavator 100.

[0182] The communication device T2 is configured to control communication with the communication device T1 mounted on the excavator 100.

[0183] The remote controller (an example of a control device) R30 is a computing device that performs various calculations. In the present embodiment, the remote controller R30 is configured by a microcomputer including a circuit or central processing unit (CPU) and a memory. The various functions of the remote controller R30 are performed by the CPU, which executes the programs stored in the memory.

[0184] The display device DR displays a screen based on the information transmitted from the excavator 100, allowing the operator OP in the remote control room RC to visually recognize the surroundings of the excavator 100. The display device DR can confirm the situation at the construction site, including the surroundings of the excavator 100, even when the operator is in the remote control room RC.

[0185] Further, as in the first embodiment, the display device DR displays the screen including the person detection map display area together with the acquired image information.

[0186] The operating device R26 is provided with the operation sensor R29 for detecting the operation content of the operating device R26. The operation sensor R29 is, for example, an inclination sensor that detects an inclination angle of the operating lever, an angle sensor that detects a rotation angle of the operating lever around a rotary shaft, or the like. The operation sensor R29 may be configured with other sensors, such as a force sensor, a current sensor, a voltage sensor, or a distance sensor. The operation sensor R29 outputs information about the detected operation content of the operating device R26 to the remote controller R30. The remote controller R30 generates an operation signal based on the received information and transmits the generated operation signal to the excavator 100. The operation sensor R29 may be configured to generate an operation signal.In this case, the operation sensor R29 can output the operation signal to the communication device T2 without passing through the remote controller R30. Thus, the excavator 100 can be remotely controlled from the remote control room RC.

[0187] The communication device T1 of the excavator 100 receives the operation signal from the communication device T2 of the remote controller R30. The controller 30 of the excavator 100 performs various work at the construction site based on the received operation signal.

[0188] In the present embodiment, the remote controller R30 displays a screen including the person detection map display area together with the acquired image information on the display device DR. When the imaging device S6 no longer detects a person from the acquired image information, the remote controller R30 performs the same control as in the above-described embodiment. Thus, the present embodiment can achieve the same effects as the above-described embodiment. In the present embodiment, the display of the display screen including the person detection map display area together with the acquired image information is not limited to the display device in the remote control room RC, and the display screen can be displayed, for example, on a display device or the like provided in a management center for managing the construction site. <wirkung>

[0189] In the above-described embodiments, the controller 30, 30A, the display device D3, or the remote controller R30 displays the screen through the above-described control, and thus, when there is a possibility of a person being near the excavator, the operator can recognize that there is a possibility of a person being near the excavator. Thus, the operator can operate the excavator taking into account the possibility of a person being near the excavator, thus achieving safety improvement.

[0190] Although the embodiments of the excavator and the controller for the excavator according to the present invention have been described above, the present invention is not limited to the above-described embodiments and the like. Various changes, modifications, substitutions, additions, deletions, and combinations are possible within the scope of the claims. Such changes are also included in the technical scope of the present invention. DESCRIPTION OF REFERENCE SYMBOLS 100,100A excavator 1 lower chassis 2 rotating mechanism 3 Upper rotating body 4 booms 5 arms 6 shovels S1 boom angle sensor S2 arm angle sensor S3 blade angle sensor S4 Body Tilt Sensor S5 angle sensor S6 imaging device S7 room detection device PS positioning device T1 communication device D3 Display device D3a control unit D4 Auxiliary storage device LM trained model 30.30A control 301 registration unit 302 Detection unit 303 Position estimation unit 304 Output control unit 305 Unit of determination RC remote control room R30 remote control T2 communication device 1201 fixed-point camera QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] JP 2014-224411

[0003] < / wirkung>

Claims

[1] Excavator, comprising: a lower chassis; an upper rotating body rotatably mounted on the lower traveling body; a space detection device mounted on the upper rotating body; a display device; and a control device configured to display, on the display device, position information representing a relationship between positions of a person detected by the space detection device and the excavators, and, when the person whose position is represented in the position information is no longer detected by the space detection device, continue to display the position of the person who is no longer detected in the position information in a case where a predetermined condition is satisfied. [2] The excavator according to claim 1, wherein the predetermined condition is a condition based on a positional relationship between a detection range in which the person can be detected by the space recognition device and a position of the person which is no longer detected. [3] The excavator according to claim 2, wherein the predetermined condition is a condition in which a last detected position of the person is not a position in which the person is able to move to an outside of the detection range, or the last detected position of the person is a position in which the person is able to move out of the detection range to the excavator side. [4] An excavator according to any one of claims 1 to 3, wherein, when the space recognition device is an imaging device, the control device is configured to display on the display device an image captured by the imaging device or an image captured by another imaging device provided in the upper rotary body separately from the imaging device, together with the position information. [5] The excavator according to claim 4, wherein the control device is configured to display an area in which a person is detected on the image and to connect the area to a position of the person shown in the position information. [6] The excavator according to claim 5, wherein the control device, when the person whose position is shown in the position information is no longer detected by the space detection device, continues to display the position of the person who is no longer detected in the position information and prohibits the display of the area in a case where the predetermined condition is satisfied. [7] The excavator according to claim 1, wherein the control device, when the person whose position is represented in the position information is no longer detected by the space recognition device, changes a display mode of the display information representing the position of the person no longer detected in a case where the predetermined condition is satisfied. [8] Control system for an excavator, the control system comprising: an excavator comprising a lower traveling body, an upper rotating body rotatably mounted on the lower traveling body, and a space detection device mounted on the upper rotating body; a display device; and a control device configured to display, on the display device, position information representing a relationship between positions of a person detected by the space detection device and the excavator, and, when the person whose position is represented in the position information is no longer detected by the space detection device, continue to display the position of the person who is no longer detected in the position information in a case where a predetermined condition is satisfied.

Images