Shovel and shovel control system

Abstract

The present invention addresses the problem of improving safety. A shovel according to one embodiment of the present invention is provided with: a lower traveling body; an upper rotating body rotatably mounted on the lower traveling body; the space recognition device is mounted on the upper revolving body; a display device; and a control device configured to display, on the display device, position information indicating a relationship between the person detected by the space recognition device and each position of the shovel, and to control the position information to be displayed on the display device when a predetermined condition is satisfied when a person showing a position in the position information is not detected by the space recognition device. And continuing to display the position of the person that has not been detected to the position information.

Description

Technical Field

[0001] This application claims priority based on Japanese Patent Application No. 2023-222736 filed on December 28, 2023. The entire content of the Japanese application is incorporated herein by reference.

[0002] The present invention relates to an excavator and a control system for an excavator. Background Art

[0003] Conventionally, a technique for detecting an object existing around an excavator and monitoring the surroundings of the excavator has been proposed (for example, refer to Patent Document 1).

[0004] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2014-224411

[0005] Patent Document 1 describes a technique for detecting an object existing around a construction machine using a stereo camera. However, object detection using a detection device such as a stereo camera may sometimes not be accurately performed due to the surrounding environment or the like. Summary of the Invention

[0006] One aspect of the present invention proposes a technique that, when a person is not detected by a space recognition device, continues to display the position of the person under certain conditions, thereby improving safety.

[0007] An excavator according to one aspect of the present invention includes: a lower traveling body; an upper revolving body rotatably mounted on the lower traveling body; a space recognition device mounted on the upper revolving body; a display device; and a control device configured to display position information indicating the relationship between a person detected by the space recognition device and each position of the excavator on the display device, and when the person whose position is displayed in the position information is not detected by the space recognition device, continue to display the position of the person who could not be detected in the position information under certain conditions.

[0008] Advantages of the Invention

[0009] According to one aspect of the present invention, safety is improved by displaying the presence of a person around the excavator. Brief Description of the Drawings

[0010] Figure 1 It is a side view showing an example of an excavator according to the first embodiment.

[0011] Figure 2 It is a plan view showing an example of an excavator according to the first embodiment.

[0012] Figure 3It is a diagram showing a structural example of a drive control system of an excavator according to the first embodiment.

[0013] Figure 4 It is a functional block diagram showing a structural example of a controller of an excavator according to the first embodiment.

[0014] Figure 5 It is a diagram showing an example of a display screen displayed on a display device according to the first embodiment.

[0015] Figure 6 It is a diagram for explaining a determination method for determining whether a person exists around the excavator based on the positional relationship between the excavator and the person in the determination unit according to the first embodiment.

[0016] Figure 7 It is a diagram showing an example of a display screen displayed on a display device according to the first embodiment.

[0017] Figure 8 It is a flowchart showing the processing steps until a display screen is displayed on the controller and the display device according to the first embodiment.

[0018] Figure 9 It is a diagram showing a modified example of a person detection map display area displayed on a display device.

[0019] Figure 10 It is a side view of an excavator according to the second embodiment.

[0020] Figure 11 It is a top view of an excavator according to the second embodiment.

[0021] Figure 12 It is a schematic diagram showing an example of a remote operation system according to the third embodiment.

[0022] In the figure: 100, 100A - excavator, 1 - lower traveling body, 2 - slewing mechanism, 3 - upper slewing body, 4 - boom, 5 - arm, 6 - bucket, S1 - boom angle sensor, S2 - arm angle sensor, S3 - bucket angle sensor, S4 - body tilt sensor, S5 - slewing angle sensor, S6 - imaging device, S7 - object detection device, PS - positioning device, T1 - communication device, D3 - display device, D3a - control unit, D4 - auxiliary storage device, LM - learned model, 30, 30A - controller, 301 - acquisition unit, 302 - detection unit, 303 - position inference unit, 304 - output control unit, 305 - determination unit, RC - remote operation room, R30 - remote controller, T2 - communication device, 1201 - fixed-point camera. Detailed Embodiments

[0023] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Note that the embodiments described below are merely examples and do not limit the invention. All features and combinations thereof described in the embodiments are not necessarily essential to the invention. In addition, the same or corresponding reference numerals are assigned to the same or corresponding structures in the respective drawings, and the description may sometimes be omitted.

[0024] Hereinafter, in an embodiment of the present invention, as an example of construction machinery, an example of using an excavator will be described, but it is not limited to an excavator. It can also be applied to construction machinery, standard machines, application machines, forestry machinery, or conveying machinery based on hydraulic excavators.

[0025] (First Embodiment)

[0026] First, Figure 1 a summary of the excavator 100 according to the present embodiment will be described. Figure 1 FIG. 13 is a side view of the excavator 100 according to the first embodiment. Figure 2 FIG. 14 is a top view of the excavator 100 according to the first embodiment.

[0027] The upper swing body 3 is rotatably mounted on the lower traveling body 1 of the excavator 100 via a slewing mechanism 2. An arm 4 is mounted on the upper swing body 3. A boom 5 is mounted at the front end of the arm 4, and a bucket 6 as an end attachment is mounted at the front end of the boom 5. The end attachment can be a bucket for slopes or a bucket for dredging.

[0028] The arm 4, the boom 5, and the bucket 6 constitute an excavation attachment as an example of the attachment AT, and are respectively hydraulically driven by an arm cylinder 7, a boom cylinder 8, and a bucket cylinder 9. An arm angle sensor S1 is mounted on the arm 4, a boom angle sensor S2 is mounted on the boom 5, and a bucket angle sensor S3 is mounted on the bucket 6. A bucket tilt mechanism may be provided in the excavation attachment.

[0029] The arm angle sensor S1 detects the rotation angle of the arm 4. In the present embodiment, the arm angle sensor S1 is an acceleration sensor and can detect the rotation angle of the arm 4 with respect to the upper swing body 3, that is, the arm angle. The arm angle becomes the minimum angle when the arm 4 is lowered to the maximum extent, and increases as the arm 4 is lifted.

[0030] The boom angle sensor S1 can include, for example, a rotary encoder, an acceleration sensor, a 6-axis sensor, an IMU (Inertial Measurement Unit), etc. Also, the boom angle sensor S1 can include a potentiometer using a variable resistor, a cylinder stroke sensor for detecting the stroke amount of a hydraulic cylinder (boom cylinder 7) corresponding to the boom angle, etc. The same applies to the arm angle sensor S2, the bucket angle sensor S3, and the body tilt sensor S4 below. The detection signal corresponding to the boom angle based on the boom angle sensor S1 is input to the controller 30.

[0031] The arm angle sensor S2 detects the rotation angle of the arm 5. In the present embodiment, the arm angle sensor S2 is an acceleration sensor and can detect the rotation angle of the arm 5 relative to the boom 4, that is, the arm angle. The arm angle becomes the minimum angle, for example, when the arm 5 is closed to the maximum extent, and increases as the arm 5 is opened.

[0032] The bucket angle sensor S3 detects the rotation angle of the bucket 6. In the present embodiment, the bucket angle sensor S3 is an acceleration sensor and can detect the rotation angle of the bucket 6 relative to the arm 5, that is, the bucket angle. The bucket angle becomes the minimum angle, for example, when the bucket 6 is closed to the maximum extent, and increases as the bucket 6 is opened.

[0033] The boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3 can be a potentiometer using a variable resistor, a stroke sensor for detecting the stroke amount of the corresponding hydraulic cylinder, or a rotary encoder for detecting the rotation angle around the connecting pin, etc. The boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3 constitute a posture sensor for detecting the posture of the excavation attachment.

[0034] On the upper swing body 3, a cab 10 serving as a driver's cab is provided and a power source such as an engine 11 is mounted. Also, a body tilt sensor S4, a swing angle sensor S5, and a camera device S6 are installed on the upper swing body 3. And a communication device T1 and a positioning device PS are installed in the upper swing body 3.

[0035] The body tilt sensor S4 is configured to detect the tilt of the upper swing body 3 relative to a specified plane. In the present embodiment, the body tilt sensor S4 is an acceleration sensor that detects the tilt angle of the upper swing body 3 around the front-rear axis and the tilt angle around the left-right axis related to the horizontal plane. The front-rear axis and the left-right axis of the upper swing body 3 are orthogonal to each other, for example, and pass through the excavator center point that is a point on the rotation axis of the excavator 100.

[0036] The rotational angle sensor S5 is configured to detect the rotational angular velocity of the upper slewing body 3. In the present embodiment, the rotational angle sensor S5 is a gyro sensor. The rotational angle sensor S5 may also be a resolver, a rotary encoder, or the like. The rotational angle sensor S5 can detect the rotational speed. The rotational speed can be calculated based on the rotational angular velocity.

[0037] In addition, when the body tilt sensor S4 includes a gyro sensor, a six-axis sensor, an IMU, etc. that can detect the angular velocity around three axes, the rotational state (e.g., rotational angular velocity) of the upper slewing body 3 can also be detected based on the detection signal of the body tilt sensor S4. In this case, the rotational angle sensor S5 can be omitted.

[0038] The imaging device S6 is an example of a space recognition device and is configured to acquire an image of the periphery of the excavator 100. In the present embodiment, the imaging device S6 includes a front camera S6F that captures the front space of the excavator 100, a left camera S6L that captures the left space of the excavator 100, a right camera S6R that captures the right space of the excavator 100, and a rear camera S6B that captures the rear space of the excavator 100.

[0039] The imaging device S6 is, for example, a monocular camera having an imaging element such as a CCD or a CMOS, and outputs the captured image to the display device D3 via the controller 30.

[0040] The input device D2 receives an operation input from the operator and outputs it to the controller 30. The input device D2 includes, for example, any hardware operation mechanism such as a touch panel, a touchpad, buttons, a toggle key, a rotary knob, etc. And, the input device D2 may also include, for example, software operation mechanisms such as virtual button icons displayed on an operation screen of the display device D3 and the like that can be operated by the hardware operation mechanism.

[0041] As Figure 2 shown, the front camera S6F is installed, for example, on the ceiling of the cab 10. The left camera S6L is installed at the left end of the upper surface of the upper slewing body 3. The right camera S6R is installed at the right end of the upper surface of the upper slewing body 3. The rear camera S6B is installed at the rear end of the upper surface of the upper slewing body 3.

[0042] The front camera S6F, the rear camera S6B, the left camera S6L, and the right camera S6R are all installed on the upper slewing body 3 so that the optical axis faces obliquely downward and a part of the upper slewing body 3 is included in the imaging range. Therefore, for example, each of the imaging ranges of the front camera S6F, the rear camera S6B, the left camera S6L, and the right camera S6R has a field of view angle of approximately 180 degrees when viewed from above. In Figure 2In the example, the imaging range AF represents an example of the imaging range of the front camera S6F, the imaging range AB represents an example of the imaging range of the rear camera S6B, the imaging range AL represents an example of the imaging range of the left camera S6L, and the imaging range AR represents an example of the imaging range of the right camera S6R. As Figure 2 shown, it is preferable to mount four monocular cameras on the upper rotating body 3 in such a manner that they do not protrude from the upper surface of the upper rotating body 3.

[0043] In the present embodiment, by configuring the imaging device S6 as described above, it is possible to image an object existing around the excavator 100.

[0044] The positioning device PS is configured to acquire information related to the position of the excavator 100. In the present embodiment, the positioning device PS is configured to measure the position and orientation of the excavator 100. Specifically, the positioning device PS is a GNSS receiver equipped with an electronic compass, which measures the latitude, longitude, and altitude of the current position of the excavator 100 and measures the orientation of the excavator 100.

[0045] Figure 3 is a diagram showing Figure 1 a structural example of the drive control system of the excavator 100. Figure 3 In, the mechanical power transmission system is represented by a double line, the working oil pipeline is represented by a thick solid line, the pilot pipeline is represented by a dotted line, and the electric drive / control system is represented by a thin solid line.

[0046] The engine 11 is the power source of the excavator 100. In the present embodiment, the engine 11 is a diesel engine that employs isochronous control to constantly maintain the engine speed regardless of the increase or decrease of the engine load. The fuel injection amount, fuel injection timing, supercharging pressure, etc. of the engine 11 are controlled by the engine control unit (ECU) D7.

[0047] The rotating shafts of the main pump 14 and the pilot pump 15, which are hydraulic pumps, are connected to the rotating shaft of the engine 11. The main pump 14 is connected to the control valve unit 17 via a working oil pipeline.

[0048] The control valve unit 17 is a hydraulic control device that controls the hydraulic system of the excavator 100. Hydraulic actuators such as the left and right travel hydraulic motors, the boom cylinder 7, the arm cylinder 8, the bucket cylinder 9, and the swing hydraulic motor are connected to the control valve unit 17 via a working oil pipeline. In addition, the swing hydraulic motor may be a swing motor generator.

[0049] Figure 3Shows the connection relationship between the controller 30 and the display device D3. In the present embodiment, the display device D3 is connected to the controller 30. The display device D3 and the controller 30 can be connected via a communication network such as CAN. In the present embodiment, an example in which one display device D3 is provided is described, but the number of display devices provided in the cab 10 is not limited, and a plurality of them can also be provided.

[0050] The display device D3 includes a control unit D3a that generates an image. In the present embodiment, the control unit D3a generates a display camera image based on the output of the camera of the imaging device S6. The imaging device S6 is connected to the display device D3 via a dedicated line, for example.

[0051] The control unit D3a generates a display image based on the output of the controller 30. In the present embodiment, the control unit D3a converts various information output by the controller 30 into an image signal. The information output by the controller 30 includes, for example, data representing the engine coolant temperature, data representing the working oil temperature, data representing the fuel remaining amount, data representing the urea water remaining amount, data representing the position of the construction part of the bucket 6, data representing the slope orientation of the construction object, data representing the orientation of the excavator 100, and data representing the operation direction for making the excavator 100 face the slope directly, etc.

[0052] The control unit D3a can be implemented as a function of the controller 30 rather than as a function of the display device D3. At this time, the imaging device S6 is connected to the controller 30 instead of the display device D3.

[0053] The display device D3 operates by receiving power supply from the battery 70. The battery 70 is charged using the power generated by the alternator 11a (generator) of the engine 11. The power of the battery 70 is supplied to the electrical fittings 72 of the excavator 100 in addition to the controller 30 and the display device D3. The starter 11b of the engine 11 is driven by the power from the battery 70 and starts the engine 11.

[0054] The engine 11 is controlled by the engine control unit D7. The engine control unit D7 always sends various data representing the state of the engine 11 to the controller 30. The various data representing the state of the engine 11 is an example of the operation information of the excavator 100, and includes, for example, data representing the coolant temperature detected by the water temperature sensor 11c as the operation information acquisition unit. The controller 30 can store this data in the temporary storage unit (memory) 30a and send it to the display device D3 when needed.

[0055] The following supplies various data as the operation information of the excavator 100 to the controller 30 and stores it in the temporary storage unit 30a of the controller 30.

[0056] For example, data representing the swash plate deflection angle is supplied from the regulator 13 of the variable displacement hydraulic pump, i.e., the main pump 14, to the controller 30. Also, data representing the discharge pressure of the main pump 14 is supplied from the discharge pressure sensor 14b to the controller 30. These data are stored in the temporary storage section 30a. Further, an oil temperature sensor 14c is provided in the pipeline between the oil tank storing the hydraulic oil sucked by the main pump 14 and the main pump 14, and data representing the temperature of the hydraulic oil flowing through this pipeline is supplied from the oil temperature sensor 14c to the controller 30. The regulator 13, the discharge pressure sensor 14b, and the oil temperature sensor 14c are examples of the operation information acquisition section.

[0057] The operation device 26 is provided near the driver's seat in the cab 10 and is used by the operator to perform operations on various driven components. Specifically, the operation device 26 is used by the operator to operate hydraulic actuators such as the left and right travel hydraulic motors, the boom cylinder 7, the arm cylinder 8, the bucket cylinder 9, and the swing hydraulic motor. As a result, the operator can operate the driven components that are the driving objects of the hydraulic actuators. The operation device 26 includes a pedal device and a rocker device for operating each driven component.

[0058] The operation sensor 29 is configured to detect the operation content of the operator using the operation device 26. In the present embodiment, the operation sensor 29 detects the operation direction and operation amount of the operation device 26 corresponding to each hydraulic actuator, and outputs an electric signal (hereinafter, also referred to as an operation signal) corresponding to the detected value to the controller 30. In the present embodiment, the controller 30 controls the opening area of the proportional valve 31 based on the output of the operation sensor 29. Moreover, the controller 30 supplies the hydraulic oil discharged by the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17. The pressure of the hydraulic oil (pilot pressure) supplied to each pilot port is, in principle, a pressure corresponding to the operation direction and operation amount of the operation device 26 corresponding to each hydraulic actuator. In this way, the operation device 26 is configured to be able to supply the hydraulic oil discharged by the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17. Thereby, the hydraulic actuator can be driven.

[0059] Also, the directional control valve for driving each hydraulic actuator built in the control valve unit 17 can be an electromagnetic solenoid type. In this case, the operation signal output from the operation device 26 can be directly input to the control valve unit 17 (i.e., the electromagnetic solenoid type directional control valve).

[0060] In addition, the operating device 26 can also be a hydraulic pilot type. Specifically, the operating device 26 uses the working oil supplied from the pilot pump 15 through the pilot pipeline to output the pilot pressure corresponding to the operation content to the secondary-side pilot pipeline. Moreover, the secondary-side pilot pipeline is connected to the control valve unit 17. Thus, a pilot pressure corresponding to the operation content related to various driven components (hydraulic actuators) in the operating device 26 can be input to the control valve unit 17. Therefore, the control valve unit 17 can drive each hydraulic actuator according to the operation content of the operator or the like on the operating device 26. At this time, an operation sensor 29 capable of acquiring information related to the operation state of the operating device 26 is provided, and the output of the operation sensor 29 is input to the controller 30. Thus, the controller 30 can grasp the operation state of the operating device 26. The operation sensor 29 is, for example, a pressure sensor that acquires information related to the pilot pressure (operation pressure) of the secondary-side pilot pipeline of the operating device 26.

[0061] Moreover, part or all of the hydraulic actuators can be replaced by electric actuators. At this time, for example, the controller 30 can output an operation command corresponding to the operation content of the operating device 26 or the content of the remote operation specified by the remote operation signal to the electric actuator or the driver that drives the electric actuator, etc. Also, it can be configured such that the electric actuator can be operated by the operating device 26 by inputting an operation signal from the operating device 26 to the electric actuator or the driver, etc.

[0062] Moreover, in the case where the excavator 100 is only remotely operated or only operates through the fully automatic driving function, the operating device 26 can be omitted.

[0063] The proportional valve 31 functions as a control valve for mechanical control, and is provided for each driven component (hydraulic actuator) of the operation object of the operating device 26 and for each movement direction (for example, the raising direction and the lowering direction of the boom 4) of the driven component (hydraulic actuator). For example, two proportional valves 31 are provided for each double-acting hydraulic actuator for driving the lower traveling body 1, the upper swing body 3, the boom 4, the arm 5, the bucket 6, etc. The proportional valve 31 can be provided, for example, on the pilot pipeline between the pilot pump 15 and the control valve unit 17, and is configured to be able to change its flow path area (that is, the cross-sectional area through which the working oil can flow). Thus, the proportional valve 31 can output a specified pilot pressure to the secondary-side pilot pipeline by using the working oil of the pilot pump 15 supplied through the primary-side pilot pipeline. Therefore, the proportional valve 31 can make a specified pilot pressure corresponding to the operation command from the controller 30 act on the control valve unit 17. Therefore, for example, the controller 30 can directly supply a pilot pressure corresponding to the operation content (operation signal) of the operating device 26 from the proportional valve 31 to the control valve unit 17, and realize the operation of the excavator 100 based on the operation of the operator.

[0064] Moreover, the controller 30 can control the proportional valve 31 to implement the autonomous driving function of the excavator 100. Specifically, the controller 30 outputs an operation instruction corresponding to the autonomous driving function to the proportional valve 31. Thereby, the controller 30 can implement the operation of the excavator 100 based on the autonomous driving function.

[0065] Moreover, the controller 30 controls the proportional valve 31 to implement the remote operation of the excavator 100. Specifically, the controller 30 outputs an operation instruction corresponding to the operation content specified by the operation signal received from the remote operation room RC to the proportional valve 31 through the communication device T1. Thereby, the controller 30 supplies the pilot pressure corresponding to the remote operation content from the proportional valve 31 to the control valve unit 17, and can implement the operation of the excavator 100 based on the remote operation of the operator.

[0066] In addition, when the operating device 26 is a hydraulic pilot type, shuttle valves can be provided in the pilot pipelines between the operating device 26 and the proportional valve 31 and the control valve unit 17. The shuttle valve has 2 inlet ports and 1 outlet port, and outputs the working oil with the higher pilot pressure among the pilot pressures input to the 2 inlet ports to the outlet port. Similarly to the proportional valve 31, the shuttle valves are provided for each driven component (hydraulic actuator) of the operation object of the operating device 26 and for each action direction of the driven component (hydraulic actuator). For example, 2 shuttle valves are provided for each double-acting hydraulic actuator for driving the lower traveling body 1, the upper swing body 3, the boom 4, the arm 5, and the bucket 6, etc. One of the 2 inlet ports of the shuttle valve is connected to the pilot pipeline on the secondary side of the operating device 26 (specifically, the above-mentioned rocker device and pedal device included in the operating device 26), and the other is connected to the pilot pipeline on the secondary side of the proportional valve 31. The outlet port of the shuttle valve is connected to the pilot port of the corresponding switching valve of the control valve unit 17 through a pilot pipeline. The corresponding switching valve refers to the switching valve that drives the hydraulic actuator that is the operation object of the above-mentioned rocker device or pedal device connected to one inlet port of the shuttle valve. Therefore, these shuttle valves can respectively apply the higher pilot pressure among the pilot pressure of the pilot pipeline on the secondary side of the operating device 26 and the pilot pressure of the pilot pipeline on the secondary side of the proportional valve 31 to the pilot port of the corresponding switching valve. That is, the controller 30 can control the corresponding switching valve without depending on the operation of the operator on the operating device 26 by outputting a pilot pressure higher than the pilot pressure on the secondary side of the operating device 26 from the proportional valve 31. Thereby, the controller 30 can control the operation of the driven components (the lower traveling body 1, the upper swing body 3, the boom 4, the arm 5, and the bucket 6) without depending on the operation state of the operator on the operating device 26, and implement the autonomous driving function or the remote operation function.

[0067] Also, when the operating device 26 is hydraulically pilot-operated, a pressure reducing valve may be provided in the pilot line between the operating device 26 and the shuttle valve in addition to the shuttle valve. The pressure reducing valve is configured, for example, to operate according to a control signal input from the controller 30 and to be able to change its flow path area. Thus, when the operating device 26 is operated by the operator, the controller 30 can forcibly reduce the pilot pressure output from the operating device 26. Therefore, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator corresponding to the operation of the operating device 26 even when the operating device 26 is operated. And, for example, even when the operating device 26 is operated, the controller 30 can reduce the pilot pressure output from the operating device 26 by the pressure reducing valve to be lower than the pilot pressure output from the proportional valve 31. Therefore, the controller 30 can reliably apply a desired pilot pressure to the pilot port of the switching valve in the control valve unit 17 regardless of the operation content of the operating device 26 by controlling the proportional valve 31 and the pressure reducing valve. Therefore, the controller 30 can more accurately implement the autonomous driving function or the remote operation function of the excavator 100, for example, by controlling the pressure reducing valve in addition to the proportional valve 31.

[0068] The communication system of the excavator 100 according to the present embodiment includes a communication device T1.

[0069] The communication device T1 is connected to an external communication line and communicates with a device separately provided from the excavator 100. Among the devices separately provided from the excavator 100, in addition to the devices located outside the excavator 100, it may also include a portable terminal device (mobile terminal) brought into the cab 10 by the user of the excavator 100. The communication device T1 may include, for example, a mobile communication module compliant with standards such as 4G (4th Generation) and 5G (5th Generation). And, the communication device T1 may also include a satellite communication module, for example. And, the communication device T1 may also include a WiFi communication module, a Bluetooth (registered trademark) communication module, etc. And, when there are multiple connectable communication lines, it may include multiple communication devices T1 according to the type of the communication line.

[0070] For example, the communication device T1 communicates with an external device such as a remote operation room in the construction site through a local communication line built in the construction site. The local communication line is, for example, a mobile communication line based on the local 5G (so-called local 5G) built in the construction site or a local area network based on WiFi.

[0071] And, the communication device T1 is configured to transmit and receive information with a communication device provided in the remote operation room through a wide area communication line including the construction site, that is, a wide area network.

[0072] In the present embodiment, a case where the attachment AT is operated, the upper swing body 3 is swung, and traveling are performed by driving a hydraulic pump with the driving force generated in the engine 11 using the engine 11 as a driving source will be described. However, in the present embodiment, the driving source is not limited to the engine 11, and a motor may be used as the driving source. That is, the control described in the present embodiment can be applied to a so-called electric excavator that drives a motor as a driving source with electric power supplied from a battery, and can also be applied to an excavator equipped with a plurality of driving sources.

[0073] <Outline of processing performed by the controller>

[0074] The controller 30 according to the present embodiment displays information related to the position of a person detected from the captured image information captured by the imaging device S6. Therefore, the controller 30 according to the present embodiment uses the captured image information captured by the imaging device S6 to perform a detection process of whether there is a person around the excavator 100.

[0075] However, there are cases where a person cannot be detected from the captured image information even though there is a person around the excavator 100. For example, there are cases where a person is too close to the excavator 100 and there is no person within the imaging range of the imaging device S6, and a person cannot be detected from the captured image information. Moreover, even when there is a person within the imaging range of the imaging device S6, overexposure occurs due to light reflection or sunset, etc., and there are cases where it cannot be seen that a person has been photographed. Moreover, even when there is a person within the imaging range of the imaging device S6, underexposure occurs due to insufficient light quantity, and there are cases where it cannot be seen that a person has been photographed.

[0076] In such a case, although a person is not detected, the possibility of a person being present is high. Therefore, when a person is not detected by the imaging device S6 (an example of a space recognition device), the controller 30 according to the present embodiment continues to display the position of the person who could not be detected when a predetermined condition is satisfied.

[0077] <Functional block structure of the excavator controller>

[0078] Figure 4 It is a functional block diagram showing a structural example of the controller 30 of the excavator 100 according to the present embodiment. In Figure 4 the example shown, the functional block structure of the controller 30 of the excavator 100 is shown.

[0079] The controller 30 receives the information output by the boom angle sensor S1, the arm angle sensor S2, the bucket angle sensor S3, the body tilt sensor S4, the slewing angle sensor S5, the imaging device S6, the input device D2, the communication device T1, the positioning device PS, etc. Moreover, based on the received information and the information stored in the auxiliary storage device D4, various operations are performed, and the operation results are output to the display device D3, the proportional valve 31, etc.

[0080] In addition, in this embodiment, an example of the controller 30 controlling the excavator 100 is described, but some functions of the controller 30 can also be implemented by other controllers (control devices). That is, the functions of the controller 30 can also be dispersedly implemented by a plurality of controllers mounted on the excavator 100.

[0081] The excavator 100 operates actuators (for example, hydraulic actuators) according to the operations of the operator riding in the cab 10 to drive the lower traveling body 1, the upper slewing body 3, the boom 4, the arm 5, the bucket 6, and other moving elements (hereinafter referred to as "driven elements").

[0082] Moreover, the excavator 100 can be configured to be remotely operable (remotely controlled) from outside the excavator 100 instead of or in addition to being operable by the operator in the cab 10. When the excavator 100 is remotely operated, the inside of the cab 10 can be in an unmanned state.

[0083] The auxiliary storage device D4 stores the learned model LM.

[0084] When the captured image information captured by the imaging device S6 is input from the input layer, the learned model LM outputs, from the output layer, the image information representing the image of the person shown in the captured image information surrounded by a rectangle. In addition, in this embodiment, an example of the output method of the learned model LM is described, but this embodiment is not limited to the method of outputting the image information of the person surrounded by a rectangle, as long as the result of detecting the person shown in the captured image information is displayed.

[0085] As the machine learning for generating the learned model LM, for example, a neural network can be applied. Specifically, machine learning using a deep neural network (DNN: Deep Neural Network), that is, deep learning (DeepLearning), can be applied. As deep learning, for example, a convolutional neural network, an RNN (Recurrent Neural Networks: recurrent neural network), and an LSTM (Long Short Term Memory: long short-term memory) can be applied.

[0086] The learned model LM is generated in an information processing apparatus (not shown) by performing machine learning based on a pre-generated training dataset.

[0087] Specifically, the learned model LM is generated by performing machine learning based on the captured image information of a person shown in the training dataset and the image information of the person shown in the captured image information surrounded by a rectangle.

[0088] In addition, the learned model LM can be updated by causing an existing learned model LM to additionally learn a new training dataset.

[0089] The controller 30 includes an acquisition unit 301, a detection unit 302, a position inference unit 303, an output control unit 304, and a determination unit 305.

[0090] The acquisition unit 301 acquires various information from various sensors. For example, the acquisition unit 301 acquires the captured image information captured by the imaging devices S6 (front camera S6F, left camera S6L, right camera S6R, and rear camera S6B).

[0091] The acquisition unit 301 acquires the detection information detected by the boom angle sensor S1, the arm angle sensor S2, the bucket angle sensor S3, the body tilt sensor S4, and the slewing angle sensor S5 respectively. The acquisition unit 301 acquires the position and orientation of the excavator 100 from the positioning device PS.

[0092] Based on the captured image information acquired by the acquisition unit 301, the detection unit 302 performs a detection process of a person existing around the excavator 100. The detection unit 302 according to the present embodiment inputs the captured image information into the learned model LM, and receives from the learned model LM the image information of the person shown in the captured image information surrounded by a rectangle. In the present embodiment, a method of detecting a person using the learned model LM is described. However, the present embodiment does not limit the method of detecting a person, and all methods can be used regardless of known methods. For example, it is possible to make a determination as to whether a feature amount extracted from the captured image information is approximately equal to or greater than a predetermined feature amount representing a person.

[0093] When there is a region surrounded by a rectangle in the image information received from the detection unit 302, the position inference unit 303 infers the position of the person in the actual space from the position coordinates of the rectangle shown in the image information, for example, infers the direction and distance where the person exists based on the excavator 100. As a specific inference method, a method used in the past can be used, and the description is omitted.

[0094] The position inference unit 303 according to this embodiment uses a method of inferring the direction and distance where a person exists based on the position and size of the rectangle showing the person in the image information, but other methods can also be used. For example, a method of using the learned model LM to output the direction and distance where a person exists can be used.

[0095] In addition, this embodiment is not limited to the method of detecting a person and inferring the position of the person in the controller 30. For example, it can also be performed in the imaging device S6, or an external cloud service can be used.

[0096] The output control unit 304 outputs the rotation angle, the image information of the area where the person is captured in a rectangle, the position coordinate information of the person existing around the excavator 100 (information indicating the detected direction and distance where the person exists), and the detection results of various sensors to the control unit D3a of the display device D3. As a result, the display device D3 displays a screen showing the surroundings of the excavator 100.

[0097] Next, referring to Figure 5 , an example of the display screen displayed on the display device D3 will be described. Figure 5 FIG. is an example of the display screen 42 displayed on the display device D3 according to this embodiment.

[0098] The control unit D3a according to this embodiment generates a display screen based on the image information input from the imaging device S6 and various information received from the controller 30. The information received from the controller 30 includes the rotation angle, the image information of the area where the person is captured in a rectangle, the position coordinate information of the person existing around the excavator (information indicating the detected direction and distance where the person exists), and the detection results of various sensors.

[0099] In the display screen 42, according to the control from the control unit D3a, a date and time display area 42a, a travel mode display area 42b, an attachment display area 42c, a fuel consumption rate display area 42d, an engine control state display area 42e, an engine operation time display area, a coolant water temperature display area 42g, a fuel remaining amount display area 42h, a rotational speed level display area 42i, a urea water remaining amount display area 42j, a working oil temperature display area 42k, a person detection map display area 421, a first image display area 422, and a second image display area 423 are displayed. The display screen 42 may include other display areas.

[0100] The travel mode display area 42b, the attachment display area 42c, the engine control status display area 42e, and the rotational speed level display area 42i are areas for displaying information related to the set state of the excavator 100, that is, the set state information. The fuel consumption rate display area 42d, the engine operation time display area, the coolant water temperature display area 42g, the fuel remaining amount display area 42h, the urea water remaining amount display area 42j, and the working oil temperature display area 42k are areas for displaying information indicating the operation state of the excavator 100, that is, the operation state information, based on the detection results of various sensors.

[0101] The date and time display area 42a is an area for displaying the current date and time. The travel mode display area 42b is an area for displaying the current travel mode. The attachment display area 42c is an area for displaying an image indicating the currently installed attachment. The fuel consumption rate display area 42d is an area for displaying the fuel consumption rate information calculated by the controller 30. The fuel consumption rate display area 42d includes an average fuel consumption rate display area 42d1 for displaying the average fuel consumption rate over the entire life period or the average fuel consumption rate for an interval, and an instantaneous fuel consumption rate display area 42d2 for displaying the instantaneous fuel consumption rate.

[0102] The engine control status display area 42e is an area for displaying the control status of the engine 11. The coolant water temperature display area 42g is an area for displaying the temperature status of the current engine coolant water. The fuel remaining amount display area 42h is an area for displaying the remaining amount status of the fuel stored in the fuel tank.

[0103] The rotational speed level display area 42i is an area for graphically displaying the current level set by the control panel 32. Numbers indicating the selected level are displayed in the rotational speed level display area 42i. The "1" displayed in the rotational speed level display area 42i indicates that the selected rotational speed level is the "first level". The number "n" displayed in the rotational speed level display area 42i indicates that the selected rotational speed level is the "nth level". "n" is a natural number. When the operator rotates the turntable 32, the numbers displayed in the rotational speed level display area 42i change.

[0104] The urea water remaining amount display area 42j is an area for graphically displaying the remaining amount status of the urea water stored in the urea water tank. The working oil temperature display area 42k is an area for displaying the temperature status of the working oil in the working oil tank.

[0105] The person detection mapping display area 421 is an area for displaying information indicating the positional relationship between the excavator 100 and the person detected around the excavator 100.

[0106] The human detection map display area (an example of position information) 421 is set as a display area of a map representing the actual space centered on the excavator 100 at a specified scale rate. In the human detection map display area 421, an excavator icon 421b indicating the presence of the excavator 100 is arranged at the center of the area.

[0107] In the human detection map display area 421, in addition to the excavator icon 421b indicating the excavator 100, an icon indicating the traveling direction of the excavator 100 ( Figure 5 in the example of Figure 5 is the direction display icon 421a) and icons indicating the humans detected around the excavator 100 (

[0108] in the example of Figure 5 are the human detection icons 421e, 421f) are simultaneously displayed. The area other than the excavator icon 421b, the direction display icon 421a, and the human detection icons 421e, 421f in the human detection map display area 421 (in other words, the background) can be, for example, an area represented by a single color (for example, black).

[0108] The excavator icon 421b is an icon formed by combining an image representing the upper swing body 3 and an image representing the lower traveling body 1 according to the positional relationship between the upper swing body 3 and the lower traveling body 1 based on the swing angle.

[0109] When the traveling lever is laid forward, the direction display icon 421a represents the traveling direction of the excavator 100 in a triangular shape. In addition, this embodiment shows an example of an icon representing the traveling direction of the excavator 100 when the traveling lever is laid forward, and any shape can be used as long as it represents the possible traveling direction of the excavator 100.

[0110] The human detection icons (an example of display information) 421e, 421f are icons representing the positions where the presence of humans is inferred by the position inference unit 303. Specifically, the human detection icons 421e, 421f are arranged according to the direction and distance of the presence of humans based on the excavator 100 inferred by the position inference unit 303. Specifically, the human detection icons 421e, 421f are arranged at positions obtained by multiplying the direction and distance of the detected presence of humans by a specified scale rate with the excavator 100 as a reference.

[0111] In this way, the positional relationship between the excavator icon 421b and the human detection icons 421e, 421f displayed in the human detection map display area 421 corresponds to the positional relationship between the excavator 100 in the actual space and the humans existing around the excavator 100.

[0112] In the present embodiment, an example in which the human detection map display area 421 is used as the position information indicating the positional relationship between the person detected by the imaging device S6 and each of the excavators 100 is described. However, the present embodiment does not limit the position information indicating the positional relationship between the person detected by the imaging device S6 and each of the excavators 100 to the human detection map display area 421. For example, position information indicating the positional relationship can be used by superimposing the position of the person detected by the imaging device S6 and each of the excavators 100 on the bird's-eye view image information based on the captured image information captured by the imaging device S6. Thus, the position information only needs to be information that can identify the positional relationship between the excavator and the person.

[0113] In the human detection map display area 421 according to the present embodiment, the display of objects other than the people existing around the excavator 100 is suppressed. Therefore, by referring to the human detection map display area 421, the operator can identify the current situation of the excavator 100, the feasible traveling direction, and the positional relationship with the people existing around the excavator 100.

[0114] Moreover, by referring to the human detection map display area 421, the operator can infer how the positional relationship between the excavator 100 and the people existing around it changes when the excavator 100 is moved. Also, in the human detection map display area 421, the display of objects other than the excavator and people is suppressed, so that the operator can be prevented from being distracted by other objects and forgetting the existence of people. Therefore, an improvement in safety can be achieved.

[0115] Moreover, a first circular area 421c and a second circular area 421d determined according to the distance from the excavator 100 are displayed in the human detection map display area 421.

[0116] Figure 5 The first circular area 421c and the second circular area 421d shown are displayed as circles for the operator to recognize the relative distance from the excavator 100 based on the excavator icon 421b.

[0117] The first circular area 421c represents, for example, information within a range of 2 m around the excavator 100. The second circular area 421d represents, for example, information within a range of 4 m around the excavator 100.

[0118] The display screen 42 according to the present embodiment can identify the position of the person based on the positional relationship between the first circular area 421c and the second circular area 421d and the human detection icons 421e, 421f with respect to the excavator 100.

[0119] Moreover, the control unit D3a makes the display modes of the human detection icons 421e and 421f different according to whether they are included in the first circular region 421c and the second circular region 421d.

[0120] For example, the human detection icon 421e existing in the first circular region 421c is displayed in red, for example. And the human detection icon 421f existing outside the first circular region 421c and inside the second circular region 421d is displayed in yellow, for example. And the human detection icon (not shown) existing outside the second circular region 421d is displayed in green, for example.

[0121] The control unit D3a according to the present embodiment makes the display modes different according to whether they are included in the first circular region 421c and the second circular region 421d. In other words, the color of the human detection icon is changed according to the distance from the excavator 100. Thus, by displaying the human detection icon whose color changes according to the distance, the display device D3 can give an attention reminder to the operator corresponding to the distance between the excavator 100 and the person. Therefore, an improvement in safety can be achieved.

[0122] The color of the human detection icon in the present embodiment is an example shown, and is not limited to this color. For example, the human detection icon can be displayed in grayscale. And the color change of the human detection icon in the present embodiment is an example of the display mode shown, and is not limited to this color change. For example, the blinking period of the human detection icon can be made different according to whether it is included in the first circular region 421c and the second circular region 421d, or the contrast or brightness can be made different, etc.

[0123] While the human detection map display area 421 is displayed on the display screen 42, the image information captured by the imaging device S6 is also displayed. By the operator confirming the image information together with the human detection map display area 421, the specific situation around the excavator 100 can be recognized. Therefore, an improvement in safety can be achieved.

[0124] In Figure 5 In the shown display screen 42, the first image display area 422 and the second image display area 423 are areas for displaying the image information captured by the imaging device S6, and include an image 423c of the counter weight of the upper revolving structure 3. The right image is displayed in the first image display area 422. The rear image is displayed in the second image display area 423. The right image is an image showing the right space of the excavator 100 and includes an image 422c of the right end of the upper surface of the upper revolving structure 3.

[0125] The right image is the image information output from the learned model LM, that is, the image information of the person shown in the actual viewpoint image captured by the right camera S6R and surrounded by a rectangle. The rear image is the image information output from the learned model LM, that is, the image information of the person shown in the actual viewpoint image captured by the rear camera S6B and surrounded by a rectangle.

[0126] Thus, a frame 422b is displayed in the right image of the first image display area 422 so as to surround the person 422a, and a frame 423b is displayed in the rear image of the second image display area 423 so as to surround the person 423a.

[0127] The first image display area 422 is displayed on the right with reference to the person detection map display area 421. And the second image display area 423 is displayed below with reference to the person detection map display area 421. In addition, this embodiment is an example where the upper side of the display screen 42 corresponds to the front of the upper rotating body 3. In other words, the second image display area 423 is displayed at a position corresponding to the rear with reference to the person detection map display area 421. That is, the display screen 42 displays the image information captured by the imaging device S6 in the direction of the imaging device S6 with reference to the person detection map display area 421. In this embodiment, the image information captured along the shooting direction is displayed with reference to the person detection map display area 421. Therefore, when the operator refers to the image information, the operator can intuitively recognize which direction the image information represents. Therefore, an improvement in safety can be achieved.

[0128] In addition, this embodiment shows an example of the configuration of the image information, but is not limited to this configuration. For example, the first image display area 422 and the second image display area 423 can be configured regardless of the shooting direction.

[0129] The control unit D3a makes the color of the frame 422b of the right image of the first image display area 422 consistent with the color of the person detection icon 421e and at the same time makes the color of the frame 423b of the rear image of the second image display area 423 consistent with the color of the person detection icon 421f (an example of establishing an association).

[0130] That is, in the display screen 42, for the right-side image and the rear image, while displaying the frame indicating the detected person, the frame is displayed in such a manner that the correspondence between the person displayed in the frame and the person detection icon can be recognized. Thus, by referring to the right-side image and the rear image, the operator can recognize the situation of the person shown in the person detection map display area 421. Therefore, the operator can operate the excavator 100 considering the situation of the people existing around the excavator 100. Therefore, an improvement in safety can be achieved. In addition, as an example of the display for recognizing the correspondence, this embodiment describes an example in which the color of the frame is made the same as the color of the person detection icon. However, although this embodiment shows an example of the display for recognizing the correspondence, it is not limited to the method of making the color of the frame the same as the color of the person detection icon. For example, it may be configured such that the correspondence can be recognized by the blinking cycle of the frame and the person detection icon.

[0131] As described above, the color of the person detection icon changes according to the distance from the excavator 100. The color of the person detection icon in the person detection map display area 421 is made the same as the color of the frame displayed in the right-side image and the rear image. Therefore, in the display screen 42, according to the distance between the excavator 100 and the person, the color of the frame (an example of the display method) displayed in the right-side image and the rear image is made different. In this embodiment, since the color of the frame displayed in the right-side image and the rear image changes according to the distance from the excavator 100, when the operator refers to the color of the frame, the operator can recognize the distance from the excavator 100. Therefore, the operator can perform an operation corresponding to the distance, and thus an improvement in safety can be achieved.

[0132] In this embodiment, the control unit D3a displays the display screen 42, and the operator can grasp the situation around the excavator 100.

[0133] Return Figure 4 , the determination unit 305 compares the direction and distance of the person inferred from the image information received from the previous learned model LM with the direction and distance of the person inferred from the image information received from the current learned model LM to determine whether the person detected previously cannot be detected this time.

[0134] Then, when the determination unit 305 determines that the person detected previously cannot be detected this time, it is determined whether the positional relationship between the detection range in which a person can be detected from the captured image information of the imaging device S6 and the position of the person who cannot be detected satisfies a specified condition. The conditions of this embodiment will be described.

[0135] Figure 6 FIG. is a diagram for explaining the determination method in the determination unit 305 according to the present embodiment for determining whether a person exists around the excavator 100 based on the positional relationship between the excavator 100 and the person.Figure 6 The detection range 1600 shown represents the range within which a person can be detected around the excavator 100 centered on the excavator 100 by the front camera S6F, the rear camera S6B, the left camera S6L, and the right camera S6R. In other words, the detection range 1600 is a part of the range formed by combining Figure 2 the shown imaging ranges AF, AB, AL, and AR, and is the range within which a person can be detected from the imaging image information of the imaging device S6.

[0136] In the present embodiment, in order to perform determination by the determination unit 305, the detection range 1600 is divided into three regions. In the present embodiment, the detection range 1600 is divided from the inside into a first range 1601, a second range 1602, and a third range 1603.

[0137] The first range 1601 is set as the range within 2 m around the excavator 100. The second range 1602 is set as the range that is more outward than the first range 1601 and within 4 m around the excavator 100. The third range 1603 is set as the range that is more outward than the second range 1602 and within 5 m around the excavator 100.

[0138] In the present embodiment, an example is given where the first range 1601 corresponds to the first circular region 421c of the human detection map display region 421, and the second range 1602 corresponds to the second circular region 421d of the human detection map display region 421. However, the division of the detection range 1600 is not limited to the method corresponding to the display of the human detection map display region 421, and the division of the detection range 1600 can also be used only for internal processing.

[0139] Moreover, when the determination unit 305 determines that a person detected previously is not detected this time, the determination result varies depending on which of the first range 1601 to the third range 1603 the person is present in.

[0140] The third range 1603 is the outer range in the detection range 1600. For example, if a person 1631 is present in the third range 1603 and moves along the arrow 1632, the person can immediately move outside the detection range 1600. Therefore, when a person present in the third range 1603 is not detected this time, the determination unit 305 can determine that the person has moved outside the detection range 1600. That is, the determination unit 305 can determine that there is actually no person.

[0141] The second range 1602 is not near the excavator 100 nor the outer area of the detection range 1600. For example, if a person 1621 is present in the second range 1602, it is difficult to move outside the detection range 1600 without being detected in the first range 1601 or the third range 1603. Therefore, when a person present in the second range 1602 is not detected this time, the determination unit 305 determines that the person still exists within the detection range 1600 and that the failure to detect this time is due to a detection error.

[0142] The first range 1601 is the range near the excavator 100. For example, when a person 1611 is present in the first range 1601, they can move under the lower traveling body 1 or close to the excavator 100. That is, although not within the detection range 1600, there is a high possibility that a person is near the excavator 100. And it is possible that the person still exists within the detection range 1600 and the failure to detect this time is due to a detection error. Therefore, when a person present in the first range 1601 is not detected this time, the determination unit 305 determines that there is at least a person near the excavator 100.

[0143] That is, when a person whose position is displayed in the human detection map display area 421 is not detected by the imaging device S6, and the position of the last detected person, such as in the first range 1601 or the second range 1602, is not a position where they can move outside the detection range 1600 or the position of the last detected person is a position where they can move from the detection range 1600 towards the excavator side, the determination unit 305 determines that a person exists near the excavator 100.

[0144] In addition, the determination performed by the determination unit 305 in the above embodiment shows an example of using determination conditions based on the positional relationship between the detection range in which a person can be detected from the imaging image information of the imaging device S6 and the position of the person who has not been detected, but is not limited to the determination based on this determination condition. For example, the determination unit 305 can determine whether the position of the person who has not been detected is near the boundary of the detection range in which a person can be detected from the imaging image information of the imaging device S6. Moreover, it is possible to determine whether they have moved outside the detection range taking into account the moving speed of the person. The specified conditions for determination are not limited to the above determination conditions and can be determined according to the implementation, such as the brightness of the construction site, the moving speed of the person, the performance of the imaging device S6, and the resolution of the imaging image information of the imaging device S6.

[0145] In the present embodiment, determination conditions based on the positional relationship between the detection range in which a person can be detected from the captured image information of the imaging device S6 and the position of a person who cannot be detected are used. Therefore, the determination by the determination unit 305 according to the present embodiment is a determination that takes into account the positional relationship, rather than a determination using a new sensor or the like, and thus it is possible to improve the detection accuracy while suppressing costs.

[0146] The output control unit 304 outputs the determination result of the determination unit 305 to the control unit D3a of the display device D3. When a person whose position is displayed in the person detection map display area 421 cannot be detected from the captured image information of the imaging device S6, and when the determination unit 305 determines that the person exists near the excavator 100 (an example of a case where a specified condition is satisfied), the control unit D3a continues to display the position of the person who cannot be detected in the person detection map display area 421.

[0147] Figure 7 FIG. is an example of a display screen 42A displayed by the display device D3 according to the present embodiment. Figure 7 In the displayed display screen 42A, Figure 5 The same reference numerals are assigned to the same structures as those in the displayed display screen 42, and the description thereof is omitted.

[0148] In Figure 7 In the displayed display screen 42A, since the person 1422a is moving, the right image displayed in the first image display area 422A is not detected by the learned model LM.

[0149] Even when a person cannot be detected in the right image, in other words, even when the display of the frame (an example of the area where a person is detected) surrounding the person in the first image display area 422A is suppressed, the display device D3 continues to display the person detection icon 1421e corresponding to the person 1411a in the first image display area 422A and the person detection map display area 421A.

[0150] Therefore, when the operator refers to the display screen 42A, even if a rectangle surrounding the person is not displayed in the first image display area 422A, the operator can infer that there is a person on the right side of the excavator 100 by referring to the person detection icon 1421e displayed in the person detection map display area 421A. Thus, in the present embodiment, the person detection map display area 421A and the first image display area 422A are simultaneously displayed in the display screen 42A, and thus the operator can recognize the surrounding situation.

[0151] Moreover, when the determination unit 305 according to the present embodiment determines that a person is present near the excavator 100, the controller 30 continuously holds the direction and distance at which the person was last detected. Moreover, the determination unit 305 uses the stored information to repeatedly determine whether the previously detected person has been detected this time.

[0152] Moreover, when the detection unit 302 newly detects a person from the captured image information of the imaging device S6, the determination unit 305 determines that the previously detected person has been detected again this time.

[0153] At this time, the display device D3 does not increase the number of person detection icons displayed in the person detection map display area, but moves the person detection icon to a position corresponding to the direction and distance of the newly detected person. Moreover, the controller 30 deletes the information related to the last detected direction and distance that has been continuously held.

[0154] When the controller 30 according to the present embodiment performs power-off control, the situations related to detection are initialized temporarily. That is, the controller 30 deletes the information related to the last detected direction and distance that has been continuously held. Moreover, the initialization process according to the present embodiment can also be performed in situations other than power-off, for example, it can be performed when the upper slewing body 3 slews or the lower traveling body 1 travels.

[0155] Next, the processing steps executed by the controller 30 and the display device D3 according to the present embodiment will be described. Figure 8 It is a flowchart showing the processing steps until the controller 30 and the display device D3 according to the present embodiment display a display screen. The processing steps shown in the flowchart according to the present embodiment are repeated at regular intervals.

[0156] First, the acquisition unit 301 acquires the captured image information captured by the imaging device S6 (step S1801).

[0157] The detection unit 302 performs a detection process for a person present around the excavator 100 based on the captured image information acquired by the acquisition unit 301 (S1802). In the present embodiment, the detection unit 302 inputs the captured image information into the learned model LM and receives, from the learned model LM, image information that encloses the person displayed in the captured image information with a rectangle, thereby detecting a person present around.

[0158] The position inference unit 303 infers the direction and distance at which the person exists with respect to the excavator 100 from the position coordinates of the person (displayed with a rectangle) shown in the captured image information (S1803).

[0159] The determination unit 305 compares the direction and distance of the person inferred in S1803 previously with the direction and distance of the person inferred in S1803 this time to determine whether the person detected previously cannot be detected this time (S1804).

[0160] When the determination unit 305 determines that the person detected previously is also detected this time (S1804: No), the display device D3 displays a display screen including a person detection map display area that displays the position of the detected person (S1805).

[0161] When the determination unit 305 determines that the person detected previously cannot be detected this time (S1804: Yes), it determines whether the position of the person detected previously is within the first range or the second range in the detection range (S1806). When the determination unit 305 determines that it is within the third range, in other words, outside the first range and the second range (step S1806: No), the display device D3 displays a display screen including a person detection map display area that displays the position of the detected person (step S1805).

[0162] On the other hand, when the determination unit 305 determines that the position of the person detected previously is within the first range or the second range (step S1806: Yes), the display device D3 displays a display screen including a person detection map display area that displays a person detection icon at the position of the person when detected previously (step S1807).

[0163] In the present embodiment, by performing the above control, even when a person cannot be detected from the captured image information, as long as the specified conditions are met, a person detection map display area that displays a person detection icon at the position of the person detected previously is displayed. Therefore, even in the case of a detection error and when a person approaches the excavator 100 and is not included in the captured range, the operator can recognize the presence of a person.

[0164] The display device D3 according to the present embodiment has been described by taking as an example a case where a display screen including the same person detection map display area as before is displayed when a person detected previously cannot be detected. However, the present embodiment does not limit the display method of the display screen including the person detection map display area, and various display methods can be used.

[0165] Figure 9 It is a diagram showing a modified example of the person detection map display area displayed on the display device D3.

[0166] Figure 9 (a) shows a modified example of a region where it is possible that the person not detected this time exists when the determination unit 305 determines that the person detected previously cannot be detected this time. In Figure 9In the human detection map display area 1901 shown in (a), while displaying the human detection icon 1911a corresponding to the position of the previously detected person, the area 1911b where the person may exist is also displayed. The area 1911b can be displayed in a manner that expands over time considering the moving speed of the person.

[0167] In Figure 9 In the human detection map display area 1901 shown in (a), by prompting the operator with the area where a person may exist, the operator can identify the area where a person may be present. Therefore, an improvement in safety can be achieved.

[0168] Figure 9 (b) shows a modified example in which the display device D3 changes the display mode of the human detection icon related to the person not detected this time when the determination unit 305 determines that the previously detected person has not been detected this time. In Figure 9 In the human detection map display area 1902 shown in (b), the human detection icon 1921 corresponding to the position of the previously detected person is displayed. The human detection icon 1921 is in a diamond shape and is different in shape from the human detection icon of the detected person (for example, the human detection icon 1922).

[0169] In this way, when the determination unit 305 determines that the previously detected person has not been detected this time, the display device D3 can make the operator recognize the existence of the person not detected this time by changing the display mode of the human detection icon. In addition, the display information indicating the position of the person can be other than the human detection icon. For example, it can be represented by an area as shown in Figure 9 (c).

[0170] Figure 9 (c) shows a modified example of the human detection map display area 1903 displayed by the display device D3. In Figure 9 In the human detection map display area 1903 shown in (c), instead of displaying the human detection icon, a circular area 1930 representing the periphery of the excavator 100 is displayed together with the excavator icon 1933. Moreover, the circular area 1930 is divided into specified areas, and each divided area is colored to indicate whether a person exists within it.

[0171] Moreover, in the human detection map display area 1903, when it is determined that the previously detected person has not been detected this time, the color of the area where the previously detected person existed is made different from the area where the person has not been detected.

[0172] For example, the area 1931 represents the area where the previously detected person existed but has not been detected this time.

[0173] By displaying the area 1931 by the display device D3, the operator can be made to recognize the existence of the person not detected.

[0174] In Figure 9 In the human detection map display area 1903 shown in (c) of, it is also possible to make the area where a person who was detected previously is not detected this time (for example, area 1931) have a different color from the area where a person is detected this time (for example, area 1932). Therefore, when the operator refers to the human detection map display area 1903, the current situation can be identified in detail through the color difference.

[0175] Figure 9 The human detection map display area 1903 shown in (c) of illustrates a method of displaying the human detection result for 360 degrees around with the excavator icon 1933 as a reference. However, this modified example is not limited to the method of displaying the human detection result for 360 degrees around with the excavator icon 1933 as a reference. For example, the human detection result can also be displayed for 180 degrees in the rear, or the human detection result can be displayed for 270 degrees in the rear and on the right side.

[0176] This embodiment is not limited to continuously displaying the above-mentioned human detection map display area when a person who was detected previously is not detected this time. For example, in addition to continuing to display the human detection map display area, the controller 30 can also display a pop-up screen showing a content urging attention. The pop-up screen is displayed, for example, as "There is a possibility of false negative. Please check the surroundings." The pop-up screen is displayed in a manner that does not overlap with the first image display area 422 and the second image display area 423. Thus, the operator can grasp the situation where a person is not detected on the basis of recognizing the surrounding situation.

[0177] (Second Embodiment)

[0178] In the above embodiment, an example of detecting a person from the captured image information captured by the imaging device S6 has been described. However, the above embodiment is not limited to the method of detecting a person from the captured image information captured by the imaging device S6. Therefore, in the second embodiment, in addition to the imaging device S6, the case where an object detection device S7 is provided is also described.

[0179] Figure 10 is a side view of the excavator 100A according to the second embodiment. Figure 11 is a top view of the excavator 100A according to the second embodiment. In addition, in this embodiment, the same reference numerals are assigned to the same structures as those in the first embodiment and the description thereof is omitted.

[0180] The object detection device S7 is an example of a space recognition device, and detects the presence or absence of an object existing in the space around the excavator 100A and the distance to the object, etc. The object detection device S7 outputs the result of measuring the space as measurement information to the controller 30A.

[0181] The object detection device S7 includes a rear object detection device S7B that detects the rear space of the excavator 100A, a left object detection device S7L that detects the left space of the excavator 100A, a right object detection device S7R that detects the right space of the excavator 100A, and a front object detection device S7F that detects the front space of the excavator 100A.

[0182] The object detection device S7 may use LIDAR to detect objects existing around the excavator 100A. LIDAR measures, for example, the distances between more than one million points within the monitoring range and the LIDAR. In addition, the present embodiment is not limited to the method using LIDAR, and any spatial recognition device capable of measuring the distance to an object may be used. For example, a stereo camera may be used, or a range image camera or a ranging device such as a millimeter-wave radar may be used. When a millimeter-wave radar or the like is used as the object detection device S7, the distance and direction of an object can be derived from the reflected signal by sending a plurality of signals (such as laser) from the object detection device S7 to the object and receiving the reflected signal.

[0183] The rear object detection device S7B is installed at the rear end of the upper surface of the upper slewing body 3. The left object detection device S7L is installed at the left end of the upper surface of the upper slewing body 3. The right object detection device S7R is installed at the right end of the upper surface of the upper slewing body 3. The front object detection device S7F is installed at the front end of the upper surface of the cab 10.

[0184] The front object detection device S7F, the rear object detection device S7B, the left object detection device S7L, and the right object detection device S7R are all installed on the upper slewing body 3 such that the optical axes face obliquely downward and a part of the upper slewing body 3 is included in the detection range. Therefore, the detection ranges of the front object detection device S7F, the rear object detection device S7B, the left object detection device S7L, and the right object detection device S7R each have a field of view angle of approximately 180 degrees when viewed from above. In Figure 11 the example, the detection range SF represents an example of the detection range of the front object detection device S7F, the detection range SB represents an example of the detection range of the rear object detection device S7B, the detection range SL represents an example of the detection range of the left object detection device S7L, and the detection range SR represents an example of the detection range of the right object detection device S7R.

[0185] As Figure 11 shown, the detection ranges SF, SB, SR, and SL are set to be narrower than the imaging ranges AF, AB, AR, and AL, but the detection ranges and imaging ranges vary depending on the embodiment.

[0186] The controller 30A according to this embodiment can perform the same processing as the controller 30 of the above embodiment. Compared with the controller 30, the only difference is that the detection result of the object detection device S7 is used to infer the position of a person in the actual space, such as the direction and distance of the person's existence with respect to the excavator 100A as a reference.

[0187] In the controller 30A according to this embodiment, based on the detection result of the object detection device S7, the position of a person in the actual space is inferred, such as the direction and distance of the person's existence with respect to the excavator 100A as a reference. A well-known method can be used to infer the direction and distance of the person's existence from the detection result. For example, a learned model can be used. For example, the controller 30A can receive the direction and distance of the person's existence by inputting the detection result of the object detection device S7 into the learned model.

[0188] The display device D3 according to this embodiment displays a human detection map display area based on the inference result of the direction and distance of the person obtained through the controller 30A on the display screen. The information displayed in other display areas is the same as that of the above embodiment, so the description is omitted.

[0189] That is, the display device D3 according to this embodiment is configured to display the captured image information captured by the imaging device S6 separately provided on the upper swing body 3 from the object detection device S7 together with the human detection map display area.

[0190] Moreover, the determination unit 305 according to this embodiment is not limited to making a determination based on the detection result of the object detection device S7, and can also make a determination by combining the detection result based on the object detection device S7 and the shooting result based on the imaging device S6. In this determination, the difference between the detection ranges SF, SB, SR, SL and the detection range in which a person can be detected from the captured image information of the imaging device S6 can be considered. For example, when it is determined that a person not detected by the object detection device S7 is detected by the imaging device S6, the display device D3 can display the position of the person in the human detection map display area.

[0191] (Third Embodiment)

[0192] In the above embodiment, the case where the operator performs construction in the excavator 100 on which the operator rides has been described. However, the above embodiment is not limited to the method of performing construction when the operator rides on the excavator 100. For example, when the excavator 100 works according to remote operation, the same display as the above embodiment can also be performed. Therefore, in the third embodiment, the case of performing remote operation of the excavator 100 is described.

[0193] Therefore, referring to Figure 12, an overview of the remote operation system SYS according to the third embodiment will be described. Figure 12 It is a schematic diagram showing an example of the remote operation system SYS according to the third embodiment.

[0194] As Figure 12 shown, the remote operation system SYS according to the third embodiment includes an excavator 100, a fixed-point camera 1201, and a remote operation room RC.

[0195] The excavator 100 and the remote operation room RC are connected via a communication line NW so as to be able to transmit and receive data.

[0196] The excavator 100 can perform wireless communication by using a communication device T1. Moreover, the excavator 100 can transmit and receive data with a device (for example, the remote operation room RC) connected to the communication line NW.

[0197] Moreover, the excavator 100 can send information related to the construction site to the remote operation room RC. Thus, the remote operation room RC can confirm the construction site based on the information from the excavator 100. In addition, in this embodiment, the device for measuring the construction site is not limited to the excavator 100, and other devices such as a drone flying over the construction site, a fixed-point camera, or a user-carried imaging device can also be used.

[0198] For example, the excavator 100 is provided with an imaging device S6. The excavator 100 sends imaging image information representing the shooting result of the imaging device S6 on the construction site to the remote operation room RC. Or, the fixed-point camera 1201 provided at the construction site sends imaging image information representing the shooting result of the construction site to the remote operation room RC. And, the device for monitoring the construction site is not limited to the fixed-point camera 1201, and it can also be a drone flying over the construction site or a user-carried imaging device. The drone or the imaging device can send imaging image information representing the shooting result of the construction site to the remote operation room RC.

[0199] The excavator 100 included in the remote operation system SYS can be one or multiple. Thus, the remote operation system SYS can provide information related to the construction site to the remote operation room RC through multiple excavators 100.

[0200] <Example structure of the remote operation room>

[0201] The remote operation room RC is equipped with a communication device T2, a remote controller R30, an operation device R26, an operation sensor R29, and a display device DR. And, an operation seat DS for the operator OP who remotely operates the excavator 100 is provided in the remote operation room RC.

[0202] The communication device T2 is configured to control communication with the communication device T1 installed in the excavator 100.

[0203] The remote controller (an example of a remote operation device) R30 is an arithmetic device that performs various operations. In the present embodiment, the remote controller R30 is constituted by a microcomputer including a CPU and a memory. Moreover, various functions of the remote controller R30 are realized by the CPU executing a program stored in the memory.

[0204] The display device DR displays a screen based on the information transmitted from the excavator 100, so that the operator OP located in the remote operation room RC can visually recognize the surroundings of the excavator 100. Although the operator is located in the remote operation room RC, the display device DR enables the operator to confirm the situation of the construction site including the surroundings of the excavator 100.

[0205] Moreover, similarly to the first embodiment, the display device DR displays a display screen including a human detection map display area together with the captured image information.

[0206] An operation sensor R29 for detecting the operation content of the operation device R26 is provided in the operation device R26. The operation sensor R29 is, for example, an inclination sensor that detects the inclination angle of the operation lever or an angle sensor that detects the swing angle of the operation lever around the swing axis. The operation sensor R29 may also be constituted by other sensors such as a pressure sensor, a current sensor, a voltage sensor, or a distance sensor. The operation sensor R29 outputs the detected information related to the operation content of the operation device R26 to the remote controller R30. The remote controller R30 generates an operation signal based on the received information and transmits the generated operation signal to the excavator 100. The operation sensor R29 may also be configured to generate an operation signal. In this case, the operation sensor R29 can output the operation signal to the communication device T2 without passing through the remote controller R30. Thus, the remote operation of the excavator 100 can be realized from the remote operation room RC.

[0207] Moreover, the communication device T1 of the excavator 100 receives the operation signal from the communication device T2 of the remote controller R30. The controller 30 of the excavator 100 performs various operations at the construction site based on the received operation signal.

[0208] In this embodiment, the remote controller R30 displays a display screen including a human detection map display area together with the captured image information in the display device DR. Moreover, when a person cannot be detected from the captured image information of the imaging device S6, the remote controller R30 performs the same control as in the above-described embodiment. Thus, this embodiment can achieve the same effects as the above-described embodiment. In addition, the display of the display screen including the human detection map display area together with the captured image information in this embodiment is not limited to the remote operation room RC. For example, it may also be displayed on a display device provided in a management center for managing a construction site or the like.

[0209] <Function>

[0210] In the above-described embodiment, the controllers 30, 30A, the display device D3, or the remote controller R30 display a display screen through the above-described control, whereby when there may be a person around the excavator, the operator can recognize the possibility of the presence of a person. Therefore, the operator can operate the excavator while considering the possibility of the presence of a person around, and thus the improvement of safety can be achieved.

[0211] As described above, embodiments of the excavator and the control system of the excavator according to the present invention have been described, but the present invention is not limited to the above-described embodiments and the like. Within the scope described in the technical solution, various changes, corrections, substitutions, additions, deletions, and combinations can be made. Of course, all of these are within the technical scope of the present invention.

Claims

1. An excavator comprising: Lower walking body; An upper rotating body, mounted on the lower walking body so as to be rotatable; A space identification device, installed on the upper rotating body; display device; and The control device is configured to display position information indicating the relationship between a person detected by the space recognition device and each position of the excavator on the display device, and when a person whose position is displayed in the position information is not detected by the space recognition device, the position of the person who has not been detected continues to be displayed in the position information if specified conditions are met.

2. The excavator according to claim 1, wherein: The predetermined condition is a condition based on a positional relationship between a detection range detectable by the space recognition device and a position of the undetectable person.

3. The excavator according to claim 2, wherein: The predetermined condition is that the last detected position of the person is not a position movable to the outside of the detection range or that the last detected position of the person is a position movable from the detection range to the shovel side.

4. The excavator according to any one of claims 1 to 3, wherein: The control device is configured to display on the display device an image captured by the camera device when the space recognition device is a camera device or an image captured by a camera device provided separately from the space recognition device on the upper rotating body together with the position information.

5. The excavator according to claim 4, wherein: The control device is configured to display the region where the person is detected on the image while associating the region with the position of the person indicated in the position information.

6. The excavator according to claim 5, wherein: When a person whose position is displayed in the position information is not detected by the space recognition device, the control device continues to display the position of the undetected person in the position information and suppresses the display of the area when the specified condition is satisfied.

7. The excavator according to claim 1, wherein: When a person whose position is indicated in the position information is not detected by the space recognition device, the control device changes the display method of the display information indicating the position of the person who could not be detected if the predetermined condition is satisfied.

8. A control system for an excavator, comprising: An excavator comprises a lower traveling body, an upper rotating body rotatably mounted on the lower traveling body, and a space recognition device installed on the upper rotating body; display device; and The control device is configured to display position information indicating the relationship between a person detected by the space recognition device and each position of the excavator on the display device, and when a person whose position is displayed in the position information is not detected by the space recognition device, the position of the person who was not detected continues to be displayed in the position information when specified conditions are met.

Images