Construction support system for excavators

Abstract

A construction assisting system (SYS) for assisting construction by a shovel (100) comprises a controller (50) that executes a simulation of the operation of the shovel (100) in a virtual environment set on the basis of a work environment of the shovel (100). The controller (50) may execute the simulation on the basis of an output from a space recognition device that recognizes the space around the shovel (100). The space recognition device may be mounted on the shovel (100), or may be installed outside the shovel (100). The construction assisting system (SYS) may also comprise a display device (D1) that displays the result of the simulation by the controller (50).

Description

[Technical Field]

[0001] The present disclosure relates to a construction support system for an excavator. [Background technology]

[0002] BACKGROUND ART There is known an operation support system that supports an operator of a construction machine by using operation data with high work quality from among past operation data of the work machine (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-156193 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned system only uses the operation data of the best workers in the past, and does not reflect the actual situation at the work site.

[0005] Therefore, it is desirable to provide a construction support system for an excavator that can reflect the actual situation at the work site when supporting construction work using an excavator. [Means for solving the problem]

[0006] A construction support system for a shovel according to an embodiment of the present invention is a system for supporting construction by a shovel, and includes a virtual environment set based on the work environment of the shovel. A virtual work site A computing device that executes a simulation of the operation of the shovel in The virtual work site is a three-dimensional virtual space in which the current topography of an actual work site is reproduced, and the computing device constructs the virtual work site based on the output of a space recognition device that recognizes the space around the shovel, and places the virtual shovel in the virtual work site based on the output of a positioning device that measures the position of the shovel so that the current topography of the virtual work site and the position and orientation of the virtual shovel coincide with the current topography of the actual work site and the position and orientation of the shovel, and the virtual shovel is configured to operate in response to an operation signal. . [Effects of the Invention]

[0007] The above-described construction support system for an excavator can reflect the actual conditions of a work site when supporting construction work by an excavator. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. [Figure 2] 2 is a diagram showing an example of the configuration of a drive system mounted on the excavator of FIG. 1. [Figure 3] FIG. 2 is a diagram showing an example of the configuration of an electrical system mounted on the excavator of FIG. 1. [Figure 4] FIG. 2 is a diagram showing the relationship between a coordinate system related to a remote control room and a coordinate system related to a shovel. [Figure 5] FIG. 1 is a schematic diagram illustrating a configuration example of a construction support system. [Figure 6] FIG. 1 is a block diagram illustrating a configuration example of a construction support system. [Figure 7] FIG. 1 is a side view of a shovel at a work site. [Figure 8] FIG. 10 is a functional block diagram showing another example of the configuration of the construction support system. DETAILED DESCRIPTION OF THE INVENTION

[0009] Non-limiting exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which:

[0010] Fig. 1 shows a shovel 100 as an excavator according to an embodiment of the present invention. An upper rotating body 3 is rotatably mounted on a lower traveling body 1 of the shovel 100 via a rotating mechanism 2. A boom 4 is attached to the upper rotating body 3. An arm 5 is attached to the tip of the boom 4, and a bucket 6 is attached to the tip of the arm 5 as an end attachment.

[0011] The boom 4, the arm 5, and the bucket 6 constitute an excavation attachment, which is an example of an attachment. The boom 4 is driven by a boom cylinder 7, the arm 5 is driven by an arm cylinder 8, and the bucket 6 is driven by a bucket cylinder 9.

[0012] A boom angle sensor S1 is attached to the boom 4, an arm angle sensor S2 is attached to the arm 5, and a bucket angle sensor S3 is attached to the bucket link. A swing angular velocity sensor S4 is attached to the upper swing body 3.

[0013] The boom angle sensor S1 is one of the attitude detection sensors and is configured to detect the rotation angle of the boom 4. In this embodiment, the boom angle sensor S1 is a stroke sensor that detects the stroke amount of the boom cylinder 7, and derives the rotation angle of the boom 4 around the boom foot pin that connects the upper rotating body 3 and the boom 4 based on the stroke amount of the boom cylinder 7.

[0014] The arm angle sensor S2 is one of the posture detection sensors and is configured to detect the rotation angle of the arm 5. In this embodiment, the arm angle sensor S2 is a stroke sensor that detects the stroke amount of the arm cylinder 8, and derives the rotation angle of the arm 5 around the connecting pin that connects the boom 4 and the arm 5 based on the stroke amount of the arm cylinder 8.

[0015] The bucket angle sensor S3 is one of the posture detection sensors and is configured to detect the rotation angle of the bucket 6. In this embodiment, the bucket angle sensor S3 is a stroke sensor that detects the stroke amount of the bucket cylinder 9, and derives the rotation angle of the bucket 6 around the connecting pin that connects the arm 5 and the bucket 6 based on the stroke amount of the bucket cylinder 9.

[0016] Each of the boom angle sensor S1, arm angle sensor S2, and bucket angle sensor S3 may be a rotary encoder, an acceleration sensor, a potentiometer (variable resistor), an inclination sensor, an inertial measurement unit, etc. The inertial measurement unit may be configured, for example, by combining an acceleration sensor and a gyro sensor.

[0017] The rotation angular velocity sensor S4 is configured to detect the rotation angular velocity of the upper rotating body 3. In this embodiment, the rotation angular velocity sensor S4 is a gyro sensor. The rotation angular velocity sensor S4 may be configured to calculate a rotation angle based on the rotation angular velocity. The rotation angular velocity sensor S4 may also be configured with other sensors such as a rotary encoder.

[0018] The upper rotating body 3 is equipped with a cabin 10 as a driver's room, an engine 11, a positioning device 18, a sound collection device A1, a spatial recognition device C1, a communication device T1, etc. A controller 30 is also installed inside the cabin 10. A driver's seat, operating devices, etc. are also installed inside the cabin 10. However, the excavator 100 may be an unmanned excavator in which the cabin 10 is omitted.

[0019] The engine 11 as a prime mover is the driving source of the excavator 100. In this embodiment, the engine 11 is a diesel engine. The output shaft of the engine 11 is connected to the input shafts of the main pump 14 and the pilot pump 15. The main pump 14 may be driven by an electric motor driven by electric power from a power storage device, instead of the engine 11.

[0020] The positioning device 18 is configured to measure the position of the excavator 100. In this embodiment, the positioning device 18 is a GNSS compass, and is configured to be able to measure the position and orientation of the upper rotating body 3.

[0021] The sound collector A1 is configured to collect sounds generated around the excavator 100. In this embodiment, the sound collector A1 is a microphone attached to the upper rotating body 3.

[0022] The spatial recognition device C1 is configured to recognize the space around the excavator 100. In this embodiment, the spatial recognition device C1 is an imaging device such as a monocular camera, a stereo camera, or an infrared camera. In the illustrated example, the spatial recognition device C1 is a monocular camera having an imaging element such as a CCD or CMOS. Specifically, the spatial recognition device C1 includes a rear camera C1B attached to the rear end of the upper surface of the upper rotating body 3, a front camera C1F attached to the front end of the upper surface of the cabin 10, a left camera C1L attached to the left end of the upper surface of the upper rotating body 3, and a right camera C1R attached to the right end of the upper surface of the upper rotating body 3. The spatial recognition device C1 may be a spherical camera installed at a predetermined position within the cabin 10. The predetermined position is, for example, a position corresponding to the eye position of an operator seated in a driver's seat installed within the cabin 10.

[0023] The spatial recognition device C1 may be a LIDAR, an ultrasonic sensor, a millimeter wave radar, a laser radar, an infrared sensor, or the like (hereinafter referred to as "LIDAR, etc."). The spatial recognition device C1 as a LIDAR, etc. may detect the distance from the spatial recognition device C1 to the object and the direction of the object as seen from the spatial recognition device C1 by emitting a number of signals (laser light, etc.) toward the object and receiving the reflected signals.

[0024] The spatial recognition device C1 may be configured to detect objects present around the shovel 100. The objects may be, for example, terrain shapes (slope, holes, etc.), dump trucks, electric wires, utility poles, people, animals, vehicles, construction machinery, buildings, walls, helmets, safety vests, work clothes, or predetermined marks on the helmets. In this case, the spatial recognition device C1 may be configured to identify at least one of the type, position, shape, etc. of the object. Furthermore, the spatial recognition device C1 may be configured to distinguish between people and non-human objects. If the spatial recognition device C1 determines that a person is present within a predetermined distance from the shovel 100 before the actuator operates, the controller 30 may disable the actuator or set it to a slow-speed state even if the operator operates the control lever. Specifically, if the controller 30 determines that a person is present within a predetermined distance from the shovel 100, it can disable the actuator by locking the gate lock valve. In the case of an electric control lever, the controller 30 can disable the actuator by disabling a signal to the operation control valve. The operation control valve is configured to output a pilot pressure corresponding to a control command from the controller 30 and apply the pilot pressure to the pilot port of the corresponding control valve in the control valve unit 17. Even when a different type of control lever is used, the same applies when an operation control valve is used. When it is desired to operate the actuator at a slow speed, the controller 30 can set the actuator to a slow-speed state by reducing the signal (e.g., the value of the current signal) to the operation control valve. In this way, when it is determined that a person is present within a predetermined distance from the shovel 100, the actuator is not driven even if the operating device is operated, or is driven at a slow speed with an output smaller than the output corresponding to the input to the operating device. Furthermore, when it is determined that a person is present within a predetermined distance from the shovel 100 while the operator is operating the operating lever, the controller 30 may stop or slow down the operation of the actuator regardless of the operator's operation. Specifically, when it is determined that a person is present within a predetermined distance from the shovel 100, the controller 30 stops the actuator by locking the gate lock valve. When an operation control valve is used, the controller 30 can disable the actuator or put it into a slow-speed state by disabling a signal to the operation control valve or by outputting a deceleration command to the operation control valve. Furthermore, when the object detected by the spatial recognition device C1 is a dump truck, stop control is not necessary. In this case, the actuator is controlled to avoid the detected dump truck.In this way, the actuator may be controlled based on knowledge of the type of object detected.

[0025] The communication device T1 is configured to control communication with devices external to the shovel 100. In this embodiment, the communication device T1 is configured to control wireless communication between the communication device T1 and devices external to the shovel 100 via a wireless communication network.

[0026] The controller 30 is a calculation device that executes various calculations. In this embodiment, the controller 30 is configured as a microcomputer including a CPU and a memory 30a. The various functions of the controller 30 are realized by the CPU executing programs stored in the memory 30a.

[0027] Fig. 2 is a block diagram showing an example of the configuration of the drive system of the excavator 100 of Fig. 1. In Fig. 2, mechanical power transmission lines are indicated by double lines, hydraulic oil lines by thick solid lines, pilot lines by dashed lines, and electrical control lines by dotted lines.

[0028] The drive system of the excavator 100 is made up of an engine 11, a regulator 13, a main pump 14, a pilot pump 15, a control valve unit 17, a controller 30, and a solenoid valve unit 45. The engine 11 is drive-controlled by an engine control unit 74.

[0029] The main pump 14 supplies hydraulic oil to a control valve unit 17 via a hydraulic oil line 16. In this embodiment, the main pump 14 is a swash plate type variable displacement hydraulic pump.

[0030] The regulator 13 is configured to control the discharge amount of the main pump 14. In this embodiment, the regulator 13 is configured to adjust the tilt angle of the swash plate of the main pump 14 in response to the discharge pressure of the main pump 14 or a control signal from the controller 30. The discharge amount (displacement volume) of the main pump 14 per rotation is controlled by the regulator 13.

[0031] The pilot pump 15 is configured to supply hydraulic oil to various hydraulic control devices via a pilot line 25. In this embodiment, the pilot pump 15 is a fixed displacement hydraulic pump. However, the pilot pump 15 may be omitted. In this case, the function of the pilot pump 15 may be realized by the main pump 14. That is, the main pump 14 may have a function of supplying hydraulic oil to the solenoid valve unit 45 and the like via a throttle or the like, in addition to the function of supplying hydraulic oil to the control valve unit 17.

[0032] The control valve unit 17 is configured to selectively supply hydraulic oil received from the main pump 14 to one or more hydraulic actuators. In this embodiment, the control valve unit 17 includes a plurality of control valves corresponding to the plurality of hydraulic actuators. The control valve unit 17 is configured to selectively supply hydraulic oil discharged from the main pump 14 to one or more hydraulic actuators. The hydraulic actuators include, for example, a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, a left-side traveling hydraulic motor 1L, a right-side traveling hydraulic motor 1R, and a swing hydraulic motor 2A.

[0033] The controller 30 is configured to control the solenoid valve unit 45 based on an operation signal received through the communication device T1. In this embodiment, the operation signal is transmitted from a remote control room. The operation signal may also be generated by an operation device provided in the cabin 10.

[0034] The solenoid valve unit 45 includes a plurality of solenoid valves arranged in the pilot lines 25 that connect the pilot pump 15 to the pilot ports of the control valves in the control valve unit 17 .

[0035] In this embodiment, the controller 30 can control the pilot pressure acting on the pilot port of each control valve by individually controlling the opening area of ​​each of the multiple solenoid valves. Therefore, the controller 30 can control the flow rate of hydraulic oil flowing into each hydraulic actuator and the flow rate of hydraulic oil flowing out of each hydraulic actuator, and therefore can control the movement of each hydraulic actuator.

[0036] In this way, the controller 30 can raise and lower the boom 4, open and close the arm 5, open and close the bucket 6, rotate the upper rotating body 3, and move the lower traveling body 1, etc., in response to operation signals from outside, such as a remote control room.

[0037] Fig. 3 is a diagram showing an example of the configuration of an electrical system mounted on the excavator of Fig. 1. As shown in Fig. 3, the engine 11 is connected to an engine control unit 74. Various data indicating the state of the engine 11 is transmitted from the engine control unit 74 to the controller 30. The controller 30 is configured to be able to store various data indicating the state of the engine 11 in a memory 30a.

[0038] The battery 70 is configured to supply power to various electrical loads mounted on the excavator 100. The alternator 11a (generator), starter 11b, controller 30, electrical components 72, etc. are configured to operate on the power stored in the battery 70. The starter 11b is configured to be driven by the power stored in the battery 70 and start the engine 11. The battery 70 is also configured to be charged with the power generated by the alternator 11a.

[0039] The water temperature sensor 11c transmits data related to the temperature of the engine coolant to the controller 30. The regulator 13 transmits data related to the swash plate tilt angle to the controller 30. The discharge pressure sensor 14b transmits data related to the discharge pressure of the main pump 14 to the controller 30. The positioning device 18 transmits data related to the position of the excavator 100 to the controller 30.

[0040] An oil temperature sensor 14c is provided in a pipe 14-1 between the main pump 14 and a hydraulic oil tank that stores hydraulic oil to be sucked by the main pump 14. The oil temperature sensor 14c transmits data relating to the temperature of the hydraulic oil flowing through the pipe 14-1 to the controller 30.

[0041] A urea water remaining amount sensor 21a provided in the urea water tank 21 transmits data relating to the remaining amount of urea water to the controller 30. A fuel remaining amount sensor 22a provided in the fuel tank 22 transmits data relating to the remaining amount of fuel to the controller 30.

[0042] The communication device T1 is configured to transmit and receive information to and from a communication device T2 installed in a remote control room RC via wireless communication. In this embodiment, the communication device T1 and the communication device T2 are configured to transmit and receive information via a fifth generation mobile communication line (5G line), an LTE line, a satellite line, or the like.

[0043] The remote control room RC is equipped with a remote controller 40, a sound output device A2, an indoor space recognition device C2, a display device D1, a communication device T2, etc. The remote control room RC is also equipped with a driver's seat DS where an operator OP who remotely operates the excavator 100 sits.

[0044] The remote controller 40 is a computing device that executes various calculations. In this embodiment, the remote controller 40 is configured with a microcomputer including a CPU and memory, similar to the controller 30. The various functions of the remote controller 40 are realized by the CPU executing programs stored in the memory.

[0045] The sound output device A2 is configured to output sound. In this embodiment, the sound output device A2 is a speaker and is configured to play back the sound collected by the sound collection device A1 attached to the shovel 100.

[0046] The interior space recognition device C2 is configured to be able to recognize the space inside the remote control room RC. In this embodiment, the interior space recognition device C2 is a camera installed inside the remote control room RC, and is configured to capture an image of the operator OP seated in the driver's seat DS.

[0047] The communication device T2 is configured to control wireless communication with the communication device T1 attached to the excavator 100.

[0048] In this embodiment, the driver's seat DS has the same structure as a driver's seat installed in the cabin of a normal excavator. Specifically, a left console box is disposed on the left side of the driver's seat DS, and a right console box is disposed on the right side of the driver's seat DS. A left operation lever is disposed on the front end of the top surface of the left console box, and a right operation lever is disposed on the front end of the top surface of the right console box. A travel lever and a travel pedal are also disposed in front of the driver's seat DS. Furthermore, a dial 75 is disposed in the center of the top surface of the right console box. The left operation lever, right operation lever, travel lever, travel pedal, and dial 75 each constitute an operation device 26.

[0049] The dial 75 is a dial for adjusting the rotation speed of the engine 11, and is configured to be able to switch the engine rotation speed in four stages, for example.

[0050] Specifically, the dial 75 is configured to be able to switch the engine speed among four modes: SP mode, H mode, A mode, and idling mode. The dial 75 transmits data relating to the setting of the engine speed to the controller 30.

[0051] The SP mode is a rotation speed mode selected when the operator OP wants to prioritize work volume, and uses the highest engine rotation speed. The H mode is a rotation speed mode selected when the operator OP wants to balance work volume and fuel efficiency, and uses the second highest engine rotation speed. The A mode is a rotation speed mode selected when the operator OP wants to operate the excavator at low noise while prioritizing fuel efficiency, and uses the third highest engine rotation speed. The idling mode is a rotation speed mode selected when the operator OP wants to idle the engine, and uses the lowest engine rotation speed. The engine 11 is then constantly controlled at the engine rotation speed of the rotation speed mode selected via the dial 75.

[0052] The operation device 26 is provided with an operation sensor 29 for detecting the operation content of the operation device 26. The operation sensor 29 is, for example, an inclination sensor that detects the inclination angle of the operation lever, or an angle sensor that detects the swing angle of the operation lever around the swing axis. The operation sensor 29 may be configured with other sensors such as a pressure sensor, a current sensor, a voltage sensor, or a distance sensor. The operation sensor 29 outputs information regarding the detected operation content of the operation device 26 to the remote controller 40. The remote controller 40 generates an operation signal based on the received information and transmits the generated operation signal to the shovel 100. The operation sensor 29 may be configured to generate an operation signal. In this case, the operation sensor 29 may output the operation signal to the communication device T2 without passing through the remote controller 40.

[0053] The display device D1 is configured to display information relating to the situation around the shovel 100. In this embodiment, the display device D1 is a multi-display made up of nine monitors arranged in three rows and three columns, and is configured to be able to display the state of the space in front of, to the left of, and to the right of the shovel 100. Each monitor is a liquid crystal monitor, an organic EL monitor, or the like. However, the display device D1 may be made up of one or more curved monitors, or may be made up of a projector.

[0054] The display device D1 may be a display device that can be worn by the operator OP. For example, the display device D1 may be a head-mounted display (VR goggles) configured to be able to transmit and receive information to and from the remote controller 40 via wireless communication. The head-mounted display may be connected to the remote controller 40 by wire. The head-mounted display may be a transparent head-mounted display or a non-transparent head-mounted display. The head-mounted display may be a monocular head-mounted display or a binocular head-mounted display.

[0055] The display device D1 is configured to display an image that enables the operator OP in the remote control room RC to visually recognize the surroundings of the shovel 100. In other words, the display device D1 displays an image so that the operator can confirm the situation around the shovel 100 as if he or she were inside the cabin 10 of the shovel 100, even though he or she is in the remote control room RC.

[0056] Next, the relationship between a first coordinate system having a reference point R1 in the remote control room RC as its origin and a second coordinate system having a reference point R2 in the excavator 100 as its origin will be described with reference to Fig. 4. In the following description, the first coordinate system will be referred to as the control room coordinate system, and the second coordinate system will be referred to as the excavator coordinate system. Fig. 4 is a diagram showing the relationship between the control room coordinate system and the excavator coordinate system.

[0057] The control room coordinate system is a three-dimensional UVW Cartesian coordinate system with its origin at reference point R1 in the remote control room RC, and has a U axis that extends parallel to the front-to-back direction of the driver's seat DS, a V axis that extends parallel to the left-to-right direction of the driver's seat DS, and a W axis that is perpendicular to the U axis and V axis.

[0058] The excavator coordinate system is a three-dimensional XYZ Cartesian coordinate system with a reference point R2 on the upper rotating body 3 as its origin, and has an X axis extending parallel to the front-to-rear direction of the upper rotating body 3, a Y axis extending parallel to the left-to-right direction of the upper rotating body 3, and a Z axis perpendicular to the X and Y axes. In the example of FIG. 4, the reference point R2 is a point on the rotation axis, the XY plane is a horizontal plane, and the Z axis is a vertical axis. That is, in the example of FIG. 4, the ground surface, which is a virtual plane on which the excavator 100 is located, is a horizontal plane. The X axis corresponds to the U axis of the control room coordinate system, the Y axis corresponds to the V axis of the control room coordinate system, and the Z axis corresponds to the W axis of the control room coordinate system.

[0059] In this embodiment, each three-dimensional coordinate in the control room coordinate system is previously associated with one of the three-dimensional coordinates in the shovel coordinate system. Therefore, once the three-dimensional coordinates of the operator viewpoint E1, which is the eye position of the operator OP in the remote control room RC, are determined, the three-dimensional coordinates of the virtual operator viewpoint E1', which is the eye position of the virtual operator in the shovel 100, are uniquely determined. Note that the eye position of the operator OP is, for example, the midpoint between the left eye position and the right eye position of the operator OP. However, the eye position of the operator OP may be a preset position. In other words, the operator viewpoint E1 and the virtual operator viewpoint E1' may be fixed points.

[0060] In the above-described embodiment, the display device D1 is installed in front, left front, and right front of the operator OP, but it may be installed in a rectangular or cylindrical shape so as to surround the operator OP. That is, the display device D1 may include a monitor installed behind the operator OP. Alternatively, the display device D1 may be installed in a hemispherical shape so as to surround the operator OP. That is, the display device D1 may include a monitor installed directly above the operator OP.

[0061] Next, a configuration example of the construction support system SYS of the excavator 100 will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a schematic diagram showing a configuration example of the construction support system SYS. Fig. 6 is a functional block diagram showing a configuration example of the construction support system SYS.

[0062] The construction support system SYS is mainly composed of a positioning device 18, a controller 30, a solenoid valve unit 45, a sound collection device A1, a spatial recognition device C1, and a communication device T1 mounted on the shovel 100, an operation sensor 29, a remote controller 40, a sound output device A2, an indoor spatial recognition device C2, a display device D1, and a communication device T2 installed in the remote control room RC, and a controller 50 and a communication device T3 as management devices installed in the information center 200.

[0063] In the example shown in Figure 5, the construction support system SYS is composed of a shovel 100a, a shovel 100b, a remote control room RCa for the shovel 100a, a remote control room RCb for the shovel 100b, a spatial recognition device C3 installed at the work site, and an information center 200.

[0064] The space recognition device C3 is configured to be able to recognize the space within the work site. In this embodiment, the space recognition device C3 is a camera installed at the work site and configured to capture images of the work site.

[0065] First, the functions of the controller 30 mounted on the shovel 100a will be described. As shown in Fig. 6, the controller 30 has, as functional blocks, an image generation unit 31, an shovel state identification unit 32, and an actuator driving unit 33. The same applies to the shovel 100b. Although the image generation unit 31, the shovel state identification unit 32, and the actuator driving unit 33 are depicted as being distinct for the sake of convenience of explanation, they do not need to be physically distinct and may be configured, in whole or in part, by common software or hardware components.

[0066] The image generation unit 31 is configured to generate a surrounding image including an image displayed on the display device D1. The surrounding image is an image used when displaying on the display device D1. Typically, the surrounding image is an image representing the surroundings of the excavator 100 that the operator would be able to see if he were inside the cabin 10. In this embodiment, the surrounding image is generated based on an image captured by an imaging device serving as the spatial recognition device C1. Specifically, the image generation unit 31 generates a first virtual viewpoint image as the surrounding image based on images captured by each of the rear camera C1B, the front camera C1F, the left camera C1L, and the right camera C1R. However, the image generation unit 31 may also generate the first virtual viewpoint image as the surrounding image based on images captured by at least one of the rear camera C1B, the front camera C1F, the left camera C1L, and the right camera C1R. The first virtual viewpoint, which is the virtual viewpoint of the first virtual viewpoint image, is a virtual operator viewpoint E1' (see FIG. 4) that corresponds to the position of the operator's eyes when the operator is seated in the driver's seat inside the cabin 10. However, the virtual operator viewpoint E1' may be located outside the cabin 10.

[0067] In this embodiment, the coordinates of the virtual operator viewpoint E1', which is the first virtual viewpoint, are derived based on the operator viewpoint E1 (see FIG. 4), which is the eye position of the operator OP when the operator OP is seated in the driver's seat DS of the remote control room RC. The coordinates of the operator viewpoint E1 are transmitted from the remote controller 40. The image generation unit 31 can derive the coordinates of the virtual operator viewpoint E1' by converting the coordinates of the operator viewpoint E1 in the control room coordinate system into coordinates in the excavator coordinate system. However, the coordinates of the operator viewpoint E1 may be preset fixed values.

[0068] Furthermore, in this embodiment, the first virtual viewpoint image corresponds to an image projected onto the inner peripheral surface of a virtual cylindrical virtual projection surface surrounding the first virtual viewpoint. The virtual projection surface may be the inner surface of a virtual sphere or hemisphere surrounding the first virtual viewpoint, or may be the inner surface of a virtual rectangular parallelepiped or cube surrounding the first virtual viewpoint. By viewing the first virtual viewpoint image generated in this manner, the operator OP can grasp the situation around the excavator 100 in three dimensions. That is, by viewing the first virtual viewpoint image, the operator OP can more accurately grasp, for example, the depth of the bed of a dump truck located in front of the excavator 100, the height of a mound on the ground, or the depth of a hole in the ground.

[0069] The image derived from the first virtual viewpoint image displayed on the display device D1 is a part of the first virtual viewpoint image generated by the image generation unit 31.

[0070] If the display device D1 is a head-mounted display, the area of ​​the image displayed on the display device D1 relative to the entire area of ​​the first virtual viewpoint image may be determined based on the line-of-sight direction of the operator OP seated in the driver's seat DS in the remote control room RC. In this case, information regarding the line-of-sight direction of the operator OP is transmitted from the remote controller 40. The image generation unit 31 generates the first virtual viewpoint image as a surrounding image based on the image output by the spatial recognition device C1 and the coordinates of the operator's viewpoint E1 transmitted from the remote controller 40. Then, the image generation unit 31 cuts out a part of the generated first virtual viewpoint image as a partial surrounding image based on the information regarding the line-of-sight direction of the operator OP transmitted from the remote controller 40, and transmits the cut-out partial surrounding image to the display device D1 in the remote control room RC.

[0071] The shovel state identifying unit 32 is configured to identify the state of the shovel 100. In this embodiment, the state of the shovel 100 includes the position and orientation of the shovel 100. The position of the shovel 100 is, for example, the latitude, longitude, and altitude of a reference point R2 on the shovel 100. The shovel state identifying unit 32 identifies the position and orientation of the shovel based on the output of the positioning device 18.

[0072] The actuator driving unit 33 is configured to drive the actuators mounted on the shovel 100. In this embodiment, the actuator driving unit 33 generates and outputs actuation signals for each of the plurality of solenoid valves included in the solenoid valve unit 45, based on an operation signal transmitted from the remote controller 40.

[0073] Each solenoid valve that receives the actuation signal increases or decreases the pilot pressure acting on the pilot port of the corresponding control valve in the control valve unit 17. As a result, the hydraulic actuator corresponding to each control valve operates at a speed that corresponds to the stroke amount of the control valve.

[0074] Next, the functions of the remote controller 40 installed in the remote control room RC will be described. The remote controller 40 has, as functional blocks, an operator state identification unit 41, an image synthesis unit 42, and an operation signal generation unit 43. For convenience of explanation, the operator state identification unit 41, the image synthesis unit 42, and the operation signal generation unit 43 are depicted as separate units, but they do not need to be physically separate units, and may be configured entirely or partially using common software or hardware components.

[0075] The operator state identification unit 41 is configured to identify the state of the operator OP in the remote control room RC. The state of the operator OP includes the eye position and line of sight of the operator OP. The operator state identification unit 41 identifies the eye position and line of sight of the operator OP based on the output of the indoor space recognition device C2. Specifically, the operator state identification unit 41 performs various image processing on the image captured by the imaging device serving as the indoor space recognition device C2, and identifies the coordinates of the eye position of the operator OP in the control room coordinate system as the coordinates of the operator viewpoint E1 (see FIG. 4). In addition, the operator state identification unit 41 performs various image processing on the image captured by the imaging device serving as the indoor space recognition device C2, and identifies the line of sight of the operator OP in the control room coordinate system.

[0076] The operator state identification unit 41 may derive the coordinates of the operator viewpoint E1 and the line of sight of the operator OP based on the output of a device other than the indoor space recognition device C2, such as a LIDAR or the like installed in the remote control room RC, or an inertial measurement unit attached to the head-mounted display as the display device D1. The inertial measurement unit may include a positioning device.

[0077] Then, the operator state identification unit 41 transmits information relating to the coordinates of the operator viewpoint E1 and the direction of the line of sight of the operator OP to the shovel 100 via the communication device T2.

[0078] The image synthesis unit 42 is configured to synthesize the partial surrounding image transmitted from the controller 30 with another image to generate a synthetic image.

[0079] The other image may be a design surface image, which is an image generated based on the design surface information DG. In this embodiment, the image composition unit 42 superimposes a graphic, such as computer graphics, representing the position of the design surface based on the design surface information DG pre-stored in a non-volatile storage device constituting the remote controller 40 as a design surface image on the partial surrounding image. The design surface is the ground when excavation work using the shovel 100 is completed. By looking at the design surface, the operator can understand the state of the surroundings of the shovel 100 when the excavation work is completed, even before the excavation work is completed. In this case, the image composition unit 42 determines the position on the partial surrounding image where the design surface image should be superimposed, based on the position and orientation of the shovel identified by the shovel state identification unit 32.

[0080] The operation signal generating unit 43 is configured to generate an operation signal. In this embodiment, the operation signal generating unit 43 is configured to generate an operation signal based on the output of the operation sensor 29.

[0081] Next, the functions of the controller 50 installed in the information center 200 will be described. The controller 50 is a calculation device that executes various calculations. In this embodiment, the controller 50 is configured as a microcomputer including a CPU and memory, similar to the controller 30 and the remote controller 40. The various functions of the controller 50 are realized by the CPU executing programs stored in the memory.

[0082] In this embodiment, the controller 50 has, as functional blocks, a determination unit 51, an operation prediction unit 52, an operation intervention unit 53, and an operation simulator 54. Although the determination unit 51, the operation prediction unit 52, the operation intervention unit 53, and the operation simulator 54 are depicted separately for the sake of convenience of explanation, they do not need to be physically separated, and may be configured entirely or partially using common software or hardware components.

[0083] The determination unit 51 is configured to determine whether there is any matter to be notified to the operator of the shovel 100 regarding the situation around the shovel 100. In this embodiment, the determination unit 51 is configured to determine whether there is any matter to be notified to the operator of the shovel 100 based on at least one of an image or a distance image (hereinafter referred to as "image, etc.") captured by a spatial recognition device C1 serving as an information acquisition device attached to the shovel 100, and the position, posture, and operation details of the shovel 100. The distance image is, for example, an image generated based on the output of a LIDAR or the like serving as the spatial recognition device C1. The determination unit 51 may be configured to determine at least one of the position, posture, and operation details of the shovel 100 based on the image, etc. captured by the spatial recognition device C1. The determination unit 51 may also be configured to determine whether there is any matter to be notified to the operator of the shovel 100 based on the image, etc. captured by the spatial recognition device C3 or construction terrain information (terrain data). Furthermore, the determination unit 51 may be configured to be able to determine at least one of the position, posture, and operation details of other construction machines based on images, etc. captured by the spatial recognition device C3. The determination unit 51 may determine whether or not there is anything that should be notified to the operator of the shovel 100 based on the situation around the shovel 100 and the position, posture, and operation details of the shovel 100 derived from images, etc. acquired by the spatial recognition device C1 and the spatial recognition device C3. Whether or not there is anything that should be notified may be determined by comparing with past cases and determining whether or not there is an identical or similar situation.

[0084] For example, when the determination unit 51 detects that a person is present outside the range covered by the image displayed on the display device D1, the determination unit 51 determines that there is something that the operator should be notified of. For example, when the determination unit 51 detects that a person is present to the left rear of the excavator 100, the determination unit 51 determines that there is something that the operator should be notified of. In this case, the determination unit 51 may detect the person based on the output of an imaging device, LIDAR, or the like serving as the spatial recognition device C1 attached to the upper rotating body 3. Alternatively, the determination unit 51 may detect the person based on the output of an imaging device, LIDAR, or the like serving as the spatial recognition device C3 installed at the work site. In this case, the spatial recognition device C3 may be, for example, a hemispherical camera attached to the tip of a pole installed at the work site. Note that the spatial recognition device C3 may be an imaging device, LIDAR, or the like attached to another work machine, or may be an imaging device, LIDAR, or the like attached to an aircraft such as a multicopter (drone) flying above the work site. The same applies when the determination unit 51 detects that a person is present inside the range covered by the image displayed on the display device D1.

[0085] Alternatively, the determination unit 51 may determine that there is an item that should be notified to the operator when it detects that an electric wire exists outside the range covered by the image displayed on the display device D1. For example, when it detects that there is an electric wire above the shovel 100, the determination unit 51 determines that there is an item that should be notified to the operator. In this case, the determination unit 51 may detect the electric wire based on the output of the spatial recognition device C1. Alternatively, the determination unit 51 may detect the electric wire based on an image captured by the spatial recognition device C3, etc. The same applies to the case where it detects that there is an electric wire inside the range covered by the image displayed on the display device D1.

[0086] Alternatively, when the determination unit 51 detects, based on the construction terrain information (terrain data), that there is a downslope ahead of the shovel 100, it determines that there is something that the operator should be notified of. For example, when the determination unit 51 detects that there is a downslope ahead of the shovel 100, it determines that there is something that the operator should be notified of. In this case, the determination unit 51 may detect the downslope based on the output of an object detection device. Alternatively, the determination unit 51 may detect the downslope based on an image captured by the spatial recognition device C3, or the like. Alternatively, the determination unit 51 may detect the downslope based on construction terrain information (terrain data) that is pre-stored in a non-volatile storage medium, or the like, attached to the controller 50.

[0087] When it is determined that there is an item that should be notified to the operator of the shovel 100, the determination unit 51 calls the operator's attention. In this embodiment, the determination unit 51 transmits information related to the item that should be notified to the remote controller 40. The image synthesis unit 42 of the remote controller 40 superimposes an image related to the information received from the determination unit 51 on the partial surrounding image.

[0088] The operation prediction unit 52 is configured to predict an operation signal after a predetermined time based on an operation signal received from the remote controller 40. This is to suppress a decrease in operation responsiveness due to communication delays, i.e., a delay until an operation by the operator OP in the remote control room RC is reflected in the movement of the excavator 100. The predetermined time is, for example, several milliseconds to several tens of milliseconds. For example, the operation prediction unit 52 predicts an operation signal after a predetermined time based on the transition of the operation signal (tilt angle of the operating lever) over a past predetermined time. For example, if the operation prediction unit 52 detects that the tilt angle of the operating lever has been on an increasing trend over a past predetermined time, it predicts that the tilt angle after a predetermined time will be larger than the current tilt angle.

[0089] Then, instead of transmitting the operation signal received from the remote controller 40 directly to the shovel 100, the operation prediction unit 52 transmits a predicted operation signal (hereinafter referred to as the "predicted operation signal") to the shovel 100.

[0090] With this configuration, the operation prediction unit 52 can transmit the operation signal generated in the remote control room RC to the excavator 100 substantially without delay.

[0091] The operation intervention unit 53 is configured to intervene in an operation by an operator OP in the remote control room RC. In this embodiment, the determination unit 51 is configured to determine whether or not to intervene in an operation by the operator OP based on an image captured by a spatial recognition device C1 attached to the shovel 100, etc.

[0092] For example, when the operation intervention unit 53 detects that there is a risk of contact between the shovel 100 and an object around the shovel 100, it determines that it should intervene in the operation by the operator OP. For example, when the operation intervention unit 53 detects that a person is present to the left of the shovel 100 and that a left rotation operation (an operation of pushing the left operating lever to the left) has begun, it determines that it should intervene in the operation by the operator OP. In this case, the operation intervention unit 53 invalidates the operation signal generated based on the left rotation operation, preventing the upper rotating body 3 from rotating left. Note that the operation intervention unit 53 may detect that there is a risk of contact between the shovel 100 and an object around the shovel 100 based on the output of an object detection device. Alternatively, the determination unit 51 may detect that there is a risk of contact between the shovel 100 and an object around the shovel 100 based on an image captured by the spatial recognition device C3, etc. In this manner, when it is determined that there is something that needs to be notified to the operator, the controller 30 may be configured to perform braking control, such as stopping or decelerating the shovel 100, based on the operation signal.

[0093] Thereafter, the operator can release the braking control, such as stopping or decelerating the shovel 100, by, for example, returning the control lever to neutral or pressing a release button, that is, by satisfying the release condition. Note that the release condition may include the shovel 100 being in a stopped state.

[0094] The operation simulator 54 is configured to be able to perform a simulation of the operation of the shovel 100. The operation of the shovel 100 constitutes various operations such as excavation work performed by the shovel 100. For example, excavation work is made up of a plurality of operations such as an arm closing operation, a bucket closing operation, and a boom raising operation. In the example shown in FIG. 6, the operation simulator 54 starts a simulation of the operation of the shovel 100 in response to a start command from the remote control room RC. Specifically, the operation simulator 54 constructs a virtual work site, which is a virtual model of the work site, based on environmental information. The environmental information is information about the work environment and includes, for example, information output by at least one of the spatial recognition device C1, the spatial recognition device C3, the positioning device 18, and the attitude detection sensor. The environmental information may also include design surface information DG.

[0095] The virtual work site is an example of a virtual environment, and is, for example, a three-dimensional virtual space (three-dimensional model) in which the current topography of the actual work site is reproduced. The operation simulator 54 transmits an image of the virtual work site to the display device D1 so that the operator OP in the remote control room RC can visually recognize the situation of the virtual work site. A virtual shovel is placed in the three-dimensional virtual space. The virtual shovel is placed in the virtual work site reproduced in the virtual space so that the current topography and the position and orientation of the virtual shovel in the virtual work site match the current topography and the position and orientation of the actual shovel 100 in the actual work site. The virtual shovel performs virtual operations in the virtual work site, just like the actual shovel 100. In other words, when the operator OP switches the connection destination of the operation device 26 from the actual shovel 100 to the virtual shovel, the operator OP can operate the virtual shovel via the operation device 26. For example, the operator OP can move the virtual arm of the virtual shovel by operating the left operation lever (arm operation lever) of the operation device 26. In this way, the operator OP can operate each actuator of the virtual shovel at the virtual work site. Furthermore, if there are buildings, electric wires, or other installed objects at the actual work site, the virtual installed objects are also reproduced at the virtual work site. Furthermore, if materials are scheduled to be delivered at a predetermined time at the actual work site, a situation in which the virtual materials are delivered at the scheduled time is also reproduced at the virtual work site. Furthermore, the ground characteristics (hardness, density, moisture content, etc.) of the ground and slope of the virtual work site are also reproduced to match the ground characteristics at the actual work site. Furthermore, if rain is predicted to fall at a predetermined time, a situation in which rain falls at the scheduled time is also virtually reproduced at the virtual work site. In this way, a three-dimensional virtual space is reproduced as a three-dimensional model. This allows the operator OP in the remote control room RC to perform virtual work at the virtual work site reproduced in the virtual space using a virtual shovel model placed in the three-dimensional virtual space. When performing virtual work, the image viewed by the operator OP in the remote control room RC corresponds to an image of the virtual work site obtained from a virtual space recognition device attached to a virtual shovel placed in a three-dimensional virtual space.The virtual space recognition device is disposed in, for example, a virtual cabin. The image of the virtual work site is typically a three-dimensional terrain image corresponding to the terrain of the actual work site and is constructed using computer graphics. However, the image of the virtual work site may also be generated, at least in part, using an image captured by an imaging device. The display device D1 that receives the image of the virtual work site can display the image of the virtual work site. The current terrain of the actual work site may be acquired through a spatial recognition device C1 (camera, LIDAR, etc.) provided on the excavator 100, a spatial recognition device (camera, LIDAR, etc.) provided on a multicopter, etc., or a spatial recognition device (camera, LIDAR, etc.) installed on a building, steel tower, etc. at the actual work site. In this way, the operation simulator 54 can reproduce the actual work site in a three-dimensional virtual space (three-dimensional model) based on the information acquired through the spatial recognition device. The virtual work site in the three-dimensional virtual space (three-dimensional model) may be updated according to the progress at the actual work site. For example, when a fallen tree occurs at an actual work site, the shovel 100 acquires information about the fallen tree at the actual work site (such as its location, size, or type of tree) using the spatial recognition device C1 and transmits it to the controller 50, which serves as a management device installed in the information center 200. The controller 50 reflects the latest information about the actual work site that it has received in the virtual work site. This allows the operator OP to operate the virtual shovel at the virtual work site and perform virtual work that takes the fallen tree into consideration.

[0096] The image related to the virtual work site typically includes an image of a virtual shovel. For example, at the time when a simulation of the operation of the shovel 100 is started, the virtual shovel corresponds to the shovel 100 at the actual work site. In other words, the position, attitude, etc. of the virtual shovel at the virtual work site at the time when the simulation is started correspond to the position, attitude, etc. of the shovel 100 at the actual work site. The position, attitude, etc. of the virtual shovel at the virtual work site at the time when the simulation is started are determined based on the output of at least one of the spatial recognition device C1 and the spatial recognition device C3, for example. However, the position, attitude, etc. of the virtual shovel at the virtual work site at the time when the simulation is started may also be determined or adjusted based on the output of at least one of the boom angle sensor S1, arm angle sensor S2, bucket angle sensor S3, swing angular velocity sensor S4, machine body inclination sensor, and positioning device 18, etc., mounted on the shovel 100.

[0097] Thereafter, the operation simulator 54 receives from the remote operation room RC an operation signal generated when the operator OP in the remote operation room RC operates the operating device 26, and operates the virtual shovel in the virtual work site in accordance with the received operation signal. The topography of the virtual work site changes in accordance with the movement of the virtual shovel. For example, the topography of the virtual work site changes in accordance with the virtual excavation work performed by the virtual shovel.

[0098] When the simulation is being performed, the operation signal is not transmitted to the shovel 100 at the actual work site. In other words, the operator OP in the remote control room RC cannot operate the shovel 100 at the actual work site when the simulation is being performed.

[0099] 6, a start command for starting a simulation is generated by the remote controller 40 when an operator OP in the remote control room RC operates a predetermined start button, and is transmitted from the remote controller 40 to the controller 50 in the information center 200. The start button is located, for example, on the top surface of the right console box.

[0100] Because the operator OP operates the excavator from a remote control room RC, it may be difficult for the operator OP to grasp the situation at the actual work site. Therefore, it is desirable for the operator OP to grasp the likelihood of an undesirable event occurring at the actual work site where the excavator 100 is located before starting work or during work before the undesirable event occurs. Therefore, the operator OP operates a predetermined start button to start a simulation before actually performing the planned work. An undesirable event is, for example, a cliff collapse when excavating a cliff. Here, a cliff refers to, for example, ground including a slope with an inclination angle equal to or greater than the angle of repose. By performing virtual excavation work using a virtual excavator at the virtual work site, the operator OP can confirm how cliffs will collapse depending on how they are excavated. That is, by virtually trying out multiple excavation work operations, the operator OP can confirm which parts of the cliff should be excavated, in what order, and to what extent to prevent cliff collapse. That is, the operator OP can derive a problem-free excavation procedure before actually performing the excavation work (without actually performing the excavation work).

[0101] Specifically, after virtually attempting one excavation task, the operator OP can operate a predetermined reset button to return the terrain of the virtual work site, which has changed as a result of the attempt, to its original terrain. The reset button is located, for example, on the top surface of the right console box. The original terrain is, for example, the terrain of the virtual work site at the time the start button is operated, and corresponds to the current terrain of the actual work site. However, the terrain of the virtual work site may be returned to the terrain at any time during the simulation. This configuration allows the operator OP to efficiently try out various excavation tasks.

[0102] Then, the operator OP ends the simulation after confirming that the excavation work is unlikely to cause cliff collapse. In the example shown in Fig. 6, the operator OP can end the simulation by operating a predetermined end button. The end button may be the same button as the start button used to start the simulation, or may be a button different from the start button.

[0103] When the simulation is completed, the image of the virtual work site displayed on the display device D1 is switched to an image of the actual work site based on an image captured by an imaging device or the like mounted on the shovel 100. Then, the operator OP can operate the operation device 26 to move the shovel 100 and perform actual excavation work.

[0104] In this state, the operator OP can perform the excavation work that is considered optimal as confirmed by the simulation, and excavate the cliff at the actual work site.

[0105] With this configuration, the controller 50 can, for example, prevent cliff collapses that may occur when excavating a cliff, thereby improving the safety of excavation work by the shovel 100.

[0106] Alternatively, the controller 50 may be configured to stop the movement of the shovel 100 and prompt the operator OP to run a simulation when it is determined that there is a high probability that an undesirable event will occur in response to the movement of the shovel 100. An undesirable event is, for example, a cliff collapse that may occur during cliff excavation.

[0107] For example, if the controller 50 determines, based on the output of the spatial recognition device C1 or the like, that the operator OP is about to excavate a cliff, it may be configured to stop the movement of the shovel 100 and prompt the operator OP to perform a simulation.

[0108] Next, with reference to Figure 7, the effect of the simulation executed by the operation simulator 54 in the controller 50 will be described. Figure 7 is a side view of the shovel 100. In the example shown in Figure 7, an operator OP of the shovel 100 is about to perform excavation work to excavate a cliff CL1 and expose a design surface TS. The operator OP remotely controls the shovel 100 using an operating device 26 installed in the remote control room RC. Before excavating the cliff CL1, the operator OP presses a start button installed in the remote control room RC to start the simulation.

[0109] When the start button is pressed, the remote controller 40 installed in the remote operation room RC generates a start command and transmits the start command to the controller 50 (motion simulator 54) in the information center 200.

[0110] Upon receiving the start command, the operation simulator 54 recognizes the topography of the actual work site around the shovel 100 based on the output of the LIDAR serving as the spatial recognition device C1 attached to the shovel 100, and generates a three-dimensional topographical image of the virtual work site corresponding to the topography of that work site.

[0111] The three-dimensional terrain image generated by the motion simulator 54 is transmitted to a display device D1 installed in the remote control room RC and displayed on the display device D1.

[0112] The operator OP can operate the virtual shovel by operating the operating device 26 while viewing the three-dimensional terrain image displayed on the display device D1.

[0113] In the example shown in Figure 7, the operator OP can simulate, for example, a first excavation operation in which the soil portion SP1 delimited by the dotted line is first excavated, and a second excavation operation in which the soil portion SP2 delimited by the dotted line is first excavated.

[0114] Then, when the operator OP tries out the first excavation work, he or she can confirm that if the soil portion SP1 is excavated first, the soil portion SP3 shown with diagonal lines will collapse. Furthermore, when the operator OP tries out the second excavation work, he or she can confirm that even if the soil portion SP2 is excavated first, the collapse of other soil portions will not occur. In this case, the operator OP may execute the second excavation work as the excavation work to be carried out at the actual work site based on the results of the simulation. Alternatively, the operator OP may execute the first excavation work as the excavation work to be carried out at the actual work site in order to proactively take advantage of the collapse of the soil portion SP3.

[0115] Next, another example of the configuration of the construction support system SYS of the excavator 100 will be described with reference to Fig. 8. Fig. 8 is a functional block diagram showing another example of the configuration of the construction support system SYS, and corresponds to Fig. 6.

[0116] The construction support system SYS shown in Fig. 8 differs from the construction support system SYS shown in Fig. 6 in that the controller 30 mounted on the excavator 100 has an abnormality detection unit 34, but in other respects it is the same as the construction support system SYS shown in Fig. 6. Therefore, in the following, a description of the common parts will be omitted and the different parts will be described in detail.

[0117] The abnormality detection unit 34 is configured to detect an abnormal event occurring in the vicinity of the shovel 100. In the example shown in Fig. 8, the abnormality detection unit 34 is configured to detect an abnormal event occurring in the vicinity of the shovel 100 in advance based on the output of the spatial recognition device C1. This is to prevent the occurrence of an abnormal event before it occurs. An abnormal event occurring in the vicinity of the shovel 100 is, for example, the shovel 100 falling off a cliff CL2 (see Fig. 7), or contact between the excavation attachment AT of the shovel 100 and an electric wire EW (see Fig. 7), etc.

[0118] The abnormality detection unit 34 recognizes the existence of cliff CL2 based on the output of the spatial recognition device C1, for example, and determines that there is a risk of the ground near cliff CL2 collapsing if the shovel 100 enters within a predetermined distance from the cliff CL2. The predetermined distance may be a value that is set in advance based on at least one of data on past cases, characteristics of the soil and sand that make up the ground, and data on the weight of the shovel 100, or may be a value that is determined dynamically.

[0119] When the shovel 100 approaches within a predetermined distance from the cliff CL2, the abnormality detection unit 34 may alert the operator OP of the shovel 100 by displaying information about the abnormality on the display device D1, or may slow down the movement of the shovel 100, or may stop the movement of the shovel 100.

[0120] Alternatively, the abnormality detection unit 34 recognizes the presence of the electric wire EW based on the output of the spatial recognition device C1, for example, and determines that there is a risk that the excavation attachment AT will come into contact with the electric wire EW when the upper end of the excavation attachment AT (arm 5) enters within a predetermined distance from the electric wire EW. The predetermined distance may be a value set in advance based on data on past cases, or may be a value that is dynamically determined.

[0121] When the upper end of the excavation attachment AT approaches within a predetermined distance from the electric wire EW, the abnormality detection unit 34 may alert the operator OP of the shovel 100, slow down the movement of the excavation attachment AT, or stop the movement of the excavation attachment AT.

[0122] The abnormality detection unit 34 may be configured to transmit information relating to the recognition results of objects present around the shovel 100 and information relating to the determination result as to whether or not an abnormal event is likely to occur to the information center 200. This is to enable the operation simulator 54 in the information center 200 to use the information.

[0123] In this case, the operation simulator 54 may be configured to call the operator OP's attention when the operator OP brings the virtual shovel close to the cliff CL2 during the simulation. Alternatively, the operation simulator 54 may be configured to call the operator OP's attention by displaying information about an abnormality on the display device D1 during the simulation when the operator OP raises the boom 4 and brings the upper end of the excavation attachment (arm 5) close to within a predetermined distance from the electric wire EW. With this configuration, the operation simulator 54 can make the operator OP aware of the presence of the cliff CL2 or the electric wire EW during the execution of the simulation, and can urge the operator OP to pay attention to the cliff CL2 or the electric wire EW.

[0124] In the above-described embodiment, the shovel 100 is operated by an operator OP in a remote control room RC, but it may also be operated by an operator in the cabin 10. In this case, the start button, reset button, and end button are installed in the cabin 10. Alternatively, the shovel 100 may be an autonomous shovel (unmanned shovel) that does not require operation by an operator. In this case, the start button, reset button, and end button may be omitted.

[0125] When the shovel 100 is an autonomous shovel, the shovel 100 is configured to perform work by using commands (operation commands) related to a series of operations that are set in advance. The operation commands are basically determined based on a work setup. The work setup means determining what operations and in what order the shovel 100 is to perform. For example, the work setup means determining which parts of the work site are to be excavated and in what order, taking into account various factors such as work efficiency and work safety. At work sites where general manned shovels are used, the work setup is typically determined based on the empirical judgment of a skilled operator. Even at work sites where autonomous shovels are used, the work setup is basically determined to be similar to the setup determined based on the empirical judgment of a skilled operator. Therefore, the work setup at work sites where autonomous shovels are used is determined based on information about the work site and various past data. At this time, machine learning techniques such as deep learning may be used.

[0126] Specifically, the motion command is determined by, for example, setting in advance a trajectory to be followed by a predetermined part, such as the tip of the bucket 6. The motion command is set by any method. For example, the motion command may be automatically generated based on data relating to past trajectories. Machine learning such as deep learning may be used to generate the motion command. In this case, "generation of motion command" is also referred to as "learning of motion command" or simply as "motion learning."

[0127] The operator OP operates the virtual shovel so that the virtual current terrain in the three-dimensional virtual space generated by the information output by the spatial recognition device C1 becomes a target surface in the virtual space (e.g., terrain based on the design surface information DG, etc.), and causes the controller 50 in the information center 200 to generate an operation command for the actual shovel 100. At this time, the controller 50 may set a reward (such as less fuel used or shorter work time) using reinforcement learning, an example of machine learning, and generate an operation command for the actuator. In this way, by using reinforcement learning, the controller 50 can generate an operation command with the highest reward when changing the virtual current terrain in the virtual space to a target surface in the virtual space (e.g., terrain based on the design surface information DG, etc.), thereby realizing a highly efficient work setup. As a result, the controller 50 can generate operation commands more efficiently than an experienced operator, thereby realizing a highly efficient work setup.

[0128] Specifically, the controller 50 in the information center 200 generates an operation command for the shovel 100 based on the design surface information DG, etc. Then, the controller 50 generates an operation signal so that the shovel 100 can automatically operate in accordance with the generated operation command. The generated operation signal is transmitted to the controller 30 mounted on the shovel 100.

[0129] If the shovel 100 is an autonomous shovel, the operation simulator 54 constituting the controller 50 can simulate the operations that constitute work by the shovel 100 before the actual work is performed.

[0130] For example, the operation simulator 54 virtually executes the operations that constitute the work performed by the shovel 100 based on information related to the operation command and information related to the current topography of the actual work site, thereby making it possible to virtually check changes in the terrain from the start to the completion of construction (work) by the shovel 100. Here, construction includes one or more operations (loading, compaction, excavation, lifting, etc.) that are performed in a predetermined sequence. Furthermore, the work includes one or more operations (excavation, swinging, soil removal, boom raising, etc.) that are performed in a predetermined order.

[0131] The operation simulator 54 can then recognize in advance any problems that may occur when the excavator 100 is operated in accordance with the operation command (any abnormal event requiring caution). In other words, the operation simulator 54 can determine the location, time, type, etc. at which any abnormal event requiring caution will occur. Furthermore, if there are multiple construction machines in the virtual work site, the operation simulator 54 can determine which construction machine will require caution. In this way, the operation simulator 54 can extract any abnormal event requiring caution.

[0132] The extracted cautionary notice may be displayed on the display device of the management device before the actual operation of the shovel 100 is performed. In this case, a three-dimensional virtual work site that is a reproduction of the actual work site is displayed on the display device. Then, for cautionary notices that occur in the three-dimensional virtual work site, the operation simulator 54 displays the location, time, type, etc. of the occurrence. The operation simulator 54 may also display the cause of the cautionary notice. Furthermore, the operation simulator 54 may reproduce the state of the virtual construction machine (virtual excavator) in the virtual work site before and after the occurrence of the cautionary notice. This allows the manager to confirm in advance how the cautionary notice will occur. Furthermore, when there are multiple construction machines in the virtual work site, the operation simulator 54 can determine which construction machine will require caution. For example, when virtual materials are delivered to the virtual work site at a scheduled time (e.g., 3:00 p.m.) based on the scheduled work content, if the virtual construction machine (virtual excavator) performs excavation work near where the virtual materials are unloaded (temporarily stored), "contact between the virtual excavator and the virtual materials" is extracted as a cautionary notice. In this case, the location, time, or type of a cautionary erroneous action occurring at the virtual work site is displayed on the display device, allowing the manager to recognize that a change to the temporary storage location of materials at the actual work site is necessary. The operation simulator 54 can also display an improvement plan for resolving the occurrence of a cautionary erroneous action. The display device may be a display unit of a mobile terminal. In this case, the operation simulator 54 performs a simulation, for example, of a case in which the loading and unloading location of the virtual materials is changed from the currently scheduled location to another location. The simulation of a case in which the loading and unloading location is changed may be performed for multiple changed locations. In this way, by performing simulations of multiple cases in which the scheduled work content is changed, the operation simulator 54 can extract a loading and unloading location that is more preferable than the currently scheduled original loading and unloading location of the virtual materials and display the preferable loading and unloading location as an improvement plan.In this way, by displaying the improvement plan on the display device, the manager can instruct the workers at the work site to change the loading and unloading location of materials at the actual work site. In this way, when an event occurring at the virtual work site requires attention, the operation simulator 54 changes the work content once or multiple times and re-runs the simulation to resolve the event requiring attention. Then, when the operation content or work arrangement, etc. for resolving the event requiring attention is derived, the operation simulator 54 continues the simulation of the subsequent construction work based on the derived work content or work arrangement, etc. In this way, even when an event requiring attention occurs, the operation simulator 54 can derive the work content for resolving the event requiring attention through simulation. Then, by displaying information on the operation content or work arrangement, etc. for resolving the event requiring attention obtained by the operation simulator 54 on the display device, the manager or worker can change the work content or work arrangement, etc. at the actual work site. The display device may be a display unit of a mobile device.

[0133] When correcting the operation command, the operation simulator 54 can find an optimal operation command by simulating the operation of the shovel 100 in accordance with the multiple new operation commands. Machine learning such as deep learning may be used to generate the multiple new operation commands, as in the case of the initial operation command.

[0134] Furthermore, the operation simulator 54 may be configured to simulate what kind of event will occur a predetermined time in the future when actual work is being performed by the shovel 100 as an autonomously operated shovel. With this configuration, the operation simulator 54 can perform virtual work (virtual operation) a predetermined time in advance of the actual work, and can recognize in advance the occurrence of an undesirable event. Then, when the operation simulator 54 recognizes in advance that an undesirable event will occur, it can correct the operation command to prevent the occurrence of such an undesirable event.

[0135] Furthermore, the operation simulator 54 may be configured to redo the simulation of the operation of the shovel 100 that constitutes the subsequent work when an event different from the expected event occurs. For example, the operation simulator 54 may be configured to redo the simulation of the operation of the shovel 100 that constitutes the subsequent work when it is determined that the soil is more clayey or sandy than expected based on the output of various sensors mounted on the shovel 100. This is because sandy ground is more likely to crumble than clayey ground, and affects the estimated results of topography that changes due to excavation work, etc. In other words, simulation results derived under the assumption that the soil is clayey will be inappropriate for soil that is actually sandier.

[0136] For example, the controller 50 can derive the amount of soil that collapses from the sides of an excavation hole, which is a hole formed by actual excavation, and accumulates at the bottom of the excavation hole, based on the output of the spatial recognition device C1. The controller 50 can then derive the characteristics of the ground on which the work is to be carried out, based on the amount of soil that accumulates at the bottom of the excavation hole. The characteristics of the ground include, for example, the degree of sandiness or the degree of clayiness. Typically, the controller 50 can determine that the greater the amount of soil that accumulates at the bottom of the excavation hole, the greater the degree of sandiness.

[0137] Furthermore, the controller 50 can compare the situation at the actual work site with the situation at the virtual work site assumed in the virtual space based on the output of the spatial recognition device C1, and evaluate the progress of the work. For example, if the actual work is delayed compared to the virtual work, the controller 50 re-executes a simulation based on the delayed situation at the actual work site using the operation simulator 54, and re-generates an operation command. Then, the controller 50 transmits the re-generated operation command to the shovel 100. The shovel 100 is controlled based on the re-generated operation command. Furthermore, the controller 50 may compare the situation at the actual work site with the situation at the virtual work site assumed in the virtual space based on the output of the spatial recognition device C1, and determine whether an event that was not anticipated (simulated) has occurred at the virtual work site.

[0138] With this configuration, the controller 50 can improve the accuracy of the simulation results relating to work performed by the shovel 100, and can improve the work efficiency of the shovel 100 as an autonomously operated shovel. In addition, the controller 50 can improve the safety of work performed by the shovel 100 as an autonomously operated shovel.

[0139] As described above, the system according to the embodiment of the present invention is a construction support system SYS for an shovel that supports construction work by the shovel 100, and includes the controller 50 as a computing device that executes a simulation of the operations of the shovel 100 in a virtual environment that is set based on the work environment of the shovel 100. The controller 50 sets a virtual work site, which is an example of a virtual environment, based on, for example, information about the work site where the shovel 100 is located, and executes a simulation of virtual operations that constitute virtual excavation work by the virtual shovel in the virtual work site. Specifically, the controller 50 derives how the virtual work site will change when certain virtual operations are performed.

[0140] With this configuration, the construction support system SYS can reflect the situation at the actual work site when supporting construction by the shovel 100 by performing a simulation of work by the shovel 100 based on information about the actual work site.

[0141] The controller 50 may execute a simulation based on the output of a spatial recognition device that recognizes the space around the shovel 100. The spatial recognition device may be mounted on the shovel 100 or may be installed outside the shovel 100. The controller 50 may execute a simulation based on the output of one spatial recognition device, or may execute a simulation based on the output of multiple spatial recognition devices. The spatial recognition device may be, for example, a spatial recognition device C1 mounted on the shovel 100. Alternatively, the spatial recognition device may be a spatial recognition device C3 installed outside the shovel 100. Specifically, the spatial recognition device C3 may be attached to a pole installed at the work site, or may be attached to another shovel other than the shovel 100, or may be attached to an aircraft flying above the work site.

[0142] The construction support system SYS may have a display device that displays the results of the simulation performed by the controller 50.

[0143] The controller 50 may be configured to be able to recognize, in advance, by simulation, events that may occur when the shovel 100 is actually operated. With this configuration, the construction support system SYS can recognize in advance undesirable events that may occur when the shovel 100 is actually operated, and can prevent such undesirable events from actually occurring.

[0144] The shovel 100 may be an autonomous shovel. In this case, the controller 50 may generate an operation signal based on information related to a preset operation command. The shovel 100 may then be configured to operate in accordance with the operation signal. Furthermore, the controller 50 may be configured to be able to change the operation command based on the results of a simulation.

[0145] With this configuration, the construction support system SYS can reflect the actual situation at the work site when supporting construction by the shovel 100, even if the shovel 100 is an automatically operated shovel.

[0146] The controller 50 may be configured to continuously derive, through simulation, a future state of the work site after a predetermined time has elapsed, which will be realized by the shovel 100 operating in accordance with the operation command.

[0147] With this configuration, the construction support system SYS can actually operate the shovel 100 and derive in real time the state of the work site at a predetermined time in the future through simulation. Therefore, the construction support system SYS can detect in advance that the state of the work site at a predetermined time in the future will become undesirable. In this case, the construction support system SYS can prevent such undesirable state from actually occurring by switching the currently used operation command to a different operation command.

[0148] The controller 50 may be configured to redo the simulation when the preconditions for the simulation change. A precondition for the simulation is, for example, that the sandiness of the soil to be worked on is within a predetermined range. The sandiness of the soil is derived, for example, based on the output of a spatial recognition device. This is because if the results of a simulation based on a changed precondition are used, the work site may end up in an undesirable state.

[0149] In another example, if the spatial recognition device detects that the position of a steel plate laid on the road surface at a construction site has shifted due to frequent passage of dump trucks, an earthquake, or the like, the controller 50 can alert the manager or worker that a caution is required and restart the simulation as the preconditions have changed.The controller 50 can then evaluate the impact of the change in preconditions on the construction work.If the position of the steel plate needs to be corrected immediately, the controller 50 may notify the manager or worker of this by displaying a message or the like.

[0150] In another example, when the controller 50 detects a change in the situation at the construction site using the spatial recognition device C1, it can alert the manager or worker if a caution is required and restart the simulation if the preconditions have changed. Changes in the situation at the construction site include, for example, changes in the topography of the construction site (such as the shape of the slope or the shape of the temporarily placed soil and sand) due to bad weather (such as rain), or the occurrence of fallen trees or falling rocks. The controller 50 can then evaluate the impact of the change in the preconditions on the construction work. If immediate improvement of the situation at the work site, such as repair of the topography, is required, the controller 50 may notify the manager or worker of this by displaying a message or the like. The operation simulator 54 may perform a simulation for repairing the topography of the work site.

[0151] In this way, the controller 50 receives the output of the spatial recognition device C1 before the start of construction or at predetermined intervals during construction. Based on the output of the spatial recognition device C1, the controller 50 compares the situation at the actual work site with the situation at the virtual work site assumed in the virtual space. If the controller 50 determines that the preconditions for the simulation have changed based on the comparison results, it determines whether the change in the preconditions requires attention. If it determines that attention is required, it notifies the manager or worker of this fact. The controller 50 then performs a simulation to resolve the situation requiring attention, and if it derives an improvement plan, such as the work content or work arrangement for resolving the situation requiring attention, it notifies the manager or worker of the derived improvement plan. If the manager or worker determines that the improvement plan derived by the operation simulator 54 is satisfactory, the manager or worker authorizes the transmission of an operation command from the controller 50 to the shovel 100. The operation command is then transmitted from the controller 50 to the shovel 100. The controller 50 controls the shovel 100 based on the received operation command.

[0152] With this configuration, if the construction support system SYS determines that the assumptions for the simulation have changed, it will redo the simulation, thereby more appropriately reflecting the actual work site conditions when supporting construction by the shovel 100.

[0153] The preferred embodiments of the present invention have been described above in detail. However, the present invention is not limited to the above-described embodiments. Various modifications or substitutions may be applied to the above-described embodiments without departing from the scope of the present invention. Furthermore, features described separately may be combined unless technical contradictions arise.

[0154] For example, in the above embodiment, the operation simulator 54 is realized as one function of the controller 50 installed in the information center 200, but it may also be realized as one function of the controller 30 or as one function of the remote controller 40. Alternatively, the operation simulator 54 may be realized as one function of a computing device separate from each of the controller 30, the remote controller 40, and the controller 50.

[0155] This application claims priority based on Japanese Patent Application No. 2020-092622, filed on May 27, 2020, the entire contents of which are incorporated herein by reference. [Explanation of symbols]

[0156] 1···Undercarriage 1L···Left-side travel hydraulic motor 1R···Right-side travel hydraulic motor 2···Slewing mechanism 2A···Slewing hydraulic motor 3···Upper rotating body 4···Boom 5···Arm 6···Bucket 7···Boom cylinder 8···Arm cylinder 9···Bucket cylinder 10···Cabin 11···Engine 11a···Alternator 11b···Starter 11c···Water temperature sensor 13···Regulator 14···Main pump 14b···Discharge pressure sensor 14c···Oil temperature sensor 15···Pilot pump 16···Hydraulic oil line 17···Control valve unit 18···Positioning device 21···Urea tank 21a···Urea remaining amount sensor 22···Fuel tank 22a...Fuel level sensor 25...Pilot line 26...Operation device 29...Operation sensor 30...Controller 30a...Memory 31...Image generation unit 32...Excavator state identification unit 33...Actuator drive unit 34...Abnormality detection unit 40...Remote controller 41...Operator state identification unit 42...Image synthesis unit 43...Operation signal generation unit 45...Solenoid valve unit 50...Controller 51...Judgment unit 52...Operation prediction unit 53...Operation intervention unit 54...Operation simulator 70...Battery 72...Electrical equipment 74...Engine control unit 75...Dial 100, 100a, 100b...Excavator 200...Information center A1...Sound collection device A2...Sound output device C1...Spatial recognition device C1B...Rear camera C1F...Front camera C1L...Left camera C1R...Right camera C2...Interior spatial recognition device C3...Spatial recognition device D1...Display device DG...Design surface information DS...Driver's seat E1...Operator's viewpoint E1'...Virtual operator's viewpoint OP...Operator RC, RCa, RCb...Remote control room S1...Boom angle sensor S2...Arm angle sensor S3...Bucket angle sensor S4...Slewing angular velocity sensor SYS...Construction support system T1, T2, T3...Communication device

Claims

1. A system for supporting construction using a shovel, a computing device that executes a simulation of the operation of the shovel in a virtual work site, which is a virtual environment set based on a work environment of the shovel, the virtual work site is a three-dimensional virtual space in which the current topography of an actual work site is reproduced; The computing device constructing the virtual work site based on the output of a space recognition device that recognizes the space around the shovel; based on an output of a positioning device that measures the position of the shovel, placing the virtual shovel at the virtual work site so that the current topography and the position and orientation of the virtual shovel at the virtual work site coincide with the current topography and the position and orientation of the shovel at the actual work site; The virtual shovel is configured to operate in response to an operation signal. Construction support system for excavators.

2. The spatial recognition device is mounted on the shovel or installed outside the shovel. The construction support system for a shovel according to claim 1.

3. a display device that displays the results of the simulation performed by the arithmetic unit; The construction support system for a shovel according to claim 1.

4. the computing device is configured to recognize, in advance, by the simulation, an event that will occur when the shovel is actually operated. The construction support system for a shovel according to claim 1.

5. the arithmetic device generates the operation signal based on information regarding an operation command which is a command related to a series of operations set in advance, and operates the virtual shovel in accordance with the operation signal in the simulation; The shovel is an automatically operated shovel and is configured to operate in response to the operation signal. The construction support system for a shovel according to claim 1.

6. the computing device changes the operation command based on the result of the simulation. The construction support system for a shovel according to claim 5.

7. the arithmetic device is configured to derive, by the simulation, a future state of a work site after a predetermined time has elapsed, which state will be realized by the shovel operating in accordance with the operation command. The construction support system for a shovel according to claim 5.

8. The computing device is configured to redo the simulation when a precondition for the simulation changes. The construction support system for a shovel according to claim 1.

9. The virtual shovel in the virtual environment is operated by an operation device. The construction support system for a shovel according to claim 1.

10. The operating device is selectively connected to the virtual shovel in the virtual environment or the shovel located at an actual work site. The construction support system for a shovel according to claim 9.

11. If the event requires attention, the computing device executes a simulation of work to resolve the event requiring attention. The construction support system for a shovel according to claim 4.

12. The computing device compares a situation at an actual work site with a situation at the virtual work site assumed in a virtual space using an output of the spatial recognition device. The construction support system for a shovel according to claim 2.

Images