Construction support system for a shovel

Abstract

A construction support system (SYS) that supports construction of an excavator (100) has a controller (50) that performs a simulation test of an action of the excavator (100) in a virtual environment set in accordance with a work environment of the excavator (100). The controller (50) can perform the simulation test in accordance with an output of a space recognition device that recognizes a surrounding space of the excavator (100). The space recognition device can be mounted on the excavator (100) or provided outside the excavator (100). Furthermore, the construction support system (SYS) can have a display device (D1) that displays a result of the simulation test of the controller (50).

Description

TECHNICAL FIELD

[0001] The present application relates to a construction support system for a shovel. BACKGROUND

[0002] An operation support system is known that supports an operator of a construction machine using operation data of high work quality in past operation data of the construction machine (see Patent Literature 1).

[0003] PRIOR ART DOCUMENTS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: Japanese Patent Application Publication No. 2016-156193 SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] However, the above system uses only operation data of a past best operator and does not reflect the situation of an actual work site.

[0008] Therefore, it is desirable to provide a construction support system for a shovel that can reflect the situation of an actual work site when supporting construction of the shovel.

[0009] MEANS FOR SOLVING THE PROBLEMS

[0010] The construction support system for a shovel according to an embodiment of the present application is a system that supports construction of a shovel and has a computing device that performs a simulation test of an action of the shovel in a hypothetical environment set in accordance with a work environment of the shovel.

[0011] EFFECTS OF THE INVENTION

[0012] The above construction support system for a shovel can reflect the situation of an actual work site when supporting construction of the shovel. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a side view of a shovel.

[0014] Figure 2 is a diagram showing a structure example of a drive system mounted on a shovel of Figure 1

[0015] Figure 3 is a diagram showing a structure example of an electric power system mounted on a shovel of Figure 1

[0016] Figure 4 is a diagram showing a relationship between a coordinate system related to a remote operation room and a coordinate system related to a shovel. ​​

[0017] Figure 5 is a schematic diagram showing a configuration example of a construction support system.

[0018] Figure 6 is a block diagram showing a configuration example of a construction support system.

[0019] Figure 7 is a side view of a shovel in a work site.

[0020] Figure 8 is a functional block diagram showing another configuration example of a construction support system. DETAILED DESCRIPTION

[0021] Next, a non-limiting example embodiment of the present application will be described with reference to the drawings.

[0022] Figure 1 A shovel 100 as an excavator to which an embodiment of the present application pertains is shown. On a lower traveling body 1 of the shovel 100, an upper swing body 3 is swingably mounted via a swing mechanism 2. A boom 4 is mounted on the upper swing body 3. A stick 5 is mounted on a front end of the boom 4, and a bucket 6 as a terminal attachment is mounted on a front end of the stick 5.

[0023] The boom 4, the stick 5, and the bucket 6 constitute an excavating attachment as an example of an attachment device. The boom 4 is driven by a boom cylinder 7, the stick 5 is driven by a stick cylinder 8, and the bucket 6 is driven by a bucket cylinder 9.

[0024] A boom angle sensor S1 is mounted on the boom 4, a stick angle sensor S2 is mounted on the stick 5, and a bucket angle sensor S3 is mounted on a bucket link. A swing angular velocity sensor S4 is mounted on the upper swing body 3.

[0025] The boom angle sensor S1 is one of posture detection sensors, and is configured to detect a rotation angle of the boom 4. In the present embodiment, the boom angle sensor S1 is a stroke sensor that detects a stroke amount of the boom cylinder 7, and derives a rotation angle of the boom 4 around a boom foot pin that links the upper swing body 3 and the boom 4, from the stroke amount of the boom cylinder 7.

[0026] The stick angle sensor S2 is one of posture detection sensors, and is configured to detect a rotation angle of the stick 5. In the present embodiment, the stick angle sensor S2 is a stroke sensor that detects a stroke amount of the stick cylinder 8, and derives a rotation angle of the stick 5 around a link pin that links the boom 4 and the stick 5, from the stroke amount of the stick cylinder 8.

[0027] The bucket angle sensor S3 is one of the posture detection sensors and is configured to detect a rotation angle of the bucket 6. In the present embodiment, the bucket angle sensor S3 is a stroke sensor that detects a stroke amount of the bucket cylinder 9, and derives a rotation angle of the bucket 6 around a joint pin that joins the boom 5 and the bucket 6, from the stroke amount of the bucket cylinder 9.

[0028] In addition, the boom angle sensor S1, the stick angle sensor S2, and the bucket angle sensor S3 can each be a rotary encoder, an acceleration sensor, a potentiometer (variable resistor), a tilt sensor, an inertial measurement device, or the like. The inertial measurement device can be composed of a combination of an acceleration sensor and a gyro sensor, for example.

[0029] The swing angle velocity sensor S4 is configured to detect a swing angle velocity of the upper swing body 3. In the present embodiment, the swing angle velocity sensor S4 is a gyro sensor. The swing angle velocity sensor S4 can also be configured to calculate a swing angle from the swing angle velocity. The swing angle velocity sensor S4 can also be composed of another sensor such as a rotary encoder.

[0030] A cab 10 as a driver's cabin, an engine 11, a positioning device 18, a sound collecting device Al, a space recognition device Cl, a communication device Tl, and the like are mounted on the upper swing body 3. Further, a controller 30 is mounted in the cab 10. Further, a driver's seat and an operation device, and the like are provided in the cab 10. However, the shovel 100 can also be an unmanned shovel that omits the cab 10.

[0031] The engine 11 as a motor is a driving source of the shovel 100. In the present embodiment, the engine 11 is a diesel engine. An output shaft of the engine 11 is connected to input shafts of the main pump 14 and the pilot pump 15, respectively. Instead of the engine 11, the main pump 14 can be driven by an electric motor that is driven by electric power from an electric storage device.

[0032] The positioning device 18 is configured to measure a position of the shovel 100. In the present embodiment, the positioning device 18 is a GNSS compass configured to be able to measure a position and an orientation of the upper swing body 3.

[0033] The sound collecting device Al is configured to collect sound generated around the shovel 100. In the present embodiment, the sound collecting device Al is a microphone mounted on the upper swing body 3.

[0034] The space recognition device C1 is configured to recognize a space around the shovel 100. In the present embodiment, the space recognition device C1 is an imaging device such as a monocular camera, a stereo camera, or an infrared camera. In the illustrated example, the space recognition device C1 is a monocular camera having an imaging element such as a CCD or a CMOS. Specifically, the space recognition device C1 includes a rear camera C1B installed at a rear end of an upper surface of the upper swing body 3, a front camera C1F installed at a front end of an upper surface of the cab 10, a left camera C1L installed at a left end of the upper surface of the upper swing body 3, and a right camera C1R installed at a right end of the upper surface of the upper swing body 3. The space recognition device C1 can be an omnidirectional camera provided at a predetermined position in the cab 10. The predetermined position is, for example, a position corresponding to an eye position of an operator seated on an operator's seat provided in the cab 10.

[0035] The space recognition device C1 can also be a LIDAR, an ultrasonic sensor, a millimeter wave radar, a laser radar, or an infrared sensor (hereinafter referred to as "LIDAR or the like"). The space recognition device C1 as the LIDAR or the like can transmit a large number of signals (laser light or the like) to an object, and detect a distance from the space recognition device C1 to the object and a direction of the object seen from the space recognition device C1 by receiving a reflected signal thereof.

[0036] The space recognition device C1 can be configured to detect an object present around the shovel 100. The object is, for example, a terrain shape (a slope or a pit or the like), a dump truck, a power line, a power pole, a person, an animal, a vehicle, a construction machine, a building, a wall, a helmet, a safety vest, a work clothes, or a predetermined mark on a helmet or the like. In this case, the space recognition device C1 can be configured to recognize at least one of a kind, a position, and a shape of the object or the like. Also, the space recognition device C1 can be configured to distinguish a person from an object other than a person.

[0037] Before the actuator is actuated, in a case where it is judged by the space recognition device C1 that a person is present within a prescribed distance range from the shovel 100, even if the operator operates the operating lever, the controller 30 can set the actuator to an inoperable state or a creep state. Specifically, the controller 30, in a case where it is judged that a person is present within a prescribed distance range from the shovel 100, can cause the actuator to be in an inoperable state by causing the gate lock valve to be in a locked state. In the case of an electric operating lever, the controller 30 can cause the actuator to be in an inoperable state by causing the signal to the control valve for operation to be invalid. The control valve for operation is configured to output a pilot pressure corresponding to the control command from the controller 30, and to cause the pilot pressure to act on the pilot port of the corresponding control valve within the control valve unit 17. The same applies even in the case of using other types of operating levers, when using the control valve for operation. In a case where it is desired to cause the actuator to be in a creep state, the controller 30 can cause the actuator to be in a creep state by reducing the signal (for example, the value of the electric current signal) to the control valve for operation. In this way, if it is judged that a person is present within a prescribed distance range from the shovel 100, even if the operating device is operated, the actuator is not driven, or is driven in a creep state by an output that is smaller than the output corresponding to the input to the operating device. Furthermore, when the operator operates the operating lever, in a case where it is judged that a person is present within a prescribed distance range from the shovel 100, the controller 30 can stop or decelerate the actuation of the actuator regardless of the contents of the operation by the operator. Specifically, in a case where it is judged that a person is present within a prescribed distance range from the shovel 100, the controller 30 can stop the actuator by causing the gate lock valve to be in a locked state. In the case of using the control valve for operation, the controller 30 can cause the actuator to be in an inoperable state or a creep state by causing the signal to the control valve for operation to be invalid, or by outputting a deceleration command to the control valve for operation. Also, in a case where the object detected by the space recognition device C1 is a dump truck, it is not necessary to perform the stop control. In this case, the actuator is controlled to avoid the detected dump truck. In this way, the actuator can be controlled in accordance with the recognition of the kind of detected object.

[0038] The communication device T1 is configured to control communication with a device located outside the shovel 100. In the present embodiment, the communication device T1 is configured to control wireless communication between the communication device T1 and a device located outside the shovel 100 via a wireless communication network.

[0039] The controller 30 is an arithmetic device that performs various kinds of arithmetic operations. In the present embodiment, the controller 30 is configured by a microcomputer including a CPU and a memory 30a. Then, various functions of the controller 30 are realized by the CPU executing a program stored in the memory 30a.

[0040] Figure 2 is a symbol indicating Figure 1a structure example of the drive system of the shovel 100. In Figure 2 In the figure, the mechanical power transmission line is indicated by a double line, the working oil pipe is indicated by a thick solid line, the pilot pipe is indicated by a broken line, and the electric control line is indicated by a dotted line.

[0041] The drive system of the shovel 100 is constituted by an engine 11, a regulator 13, a main pump 14, a pilot pump 15, a control valve unit 17, a controller 30, an electromagnetic valve unit 45, and the like. The engine 11 is driven and controlled by an engine control unit 74.

[0042] The main pump 14 supplies working oil to the control valve unit 17 via a working oil pipe 16. In the present embodiment, the main pump 14 is a swash plate type variable displacement hydraulic pump.

[0043] The regulator 13 is configured to control the discharge amount of the main pump 14. In the present embodiment, the regulator 13 is configured to adjust the swash plate deflection angle of the main pump 14 in accordance with the discharge pressure of the main pump 14 or a control signal from the controller 30, or the like. The main pump 14 is controlled by the regulator 13 to have a discharge amount (displacement) per rotation.

[0044] The pilot pump 15 is configured to supply working oil to various hydraulic control devices via a pilot pipe 25. In the present embodiment, the pilot pump 15 is a fixed displacement hydraulic pump. However, the pilot pump 15 can be omitted. In this case, the functions assumed by the pilot pump 15 can be implemented by the main pump 14. That is, the main pump 14 can have a function of supplying working oil to the electromagnetic valve unit 45 and the like via a throttle or the like in addition to a function of supplying working oil to the control valve unit 17.

[0045] The control valve unit 17 is configured to be able to selectively supply working oil received from the main pump 14 to one or a plurality of hydraulic actuators. In the present embodiment, the control valve unit 17 includes a plurality of control valves corresponding to a plurality of hydraulic actuators. Further, the control valve unit 17 is configured to be able to selectively supply working oil discharged from the main pump 14 to one or a plurality of hydraulic actuators. The hydraulic actuators include, for example, a boom cylinder 7, a stick cylinder 8, a bucket cylinder 9, a left traveling hydraulic motor 1L, a right traveling hydraulic motor 1R, and a swing hydraulic motor 2A.

[0046] The controller 30 is configured to control the electromagnetic valve unit 45 in accordance with an operation signal received through a communication device T1. In the present embodiment, the operation signal is transmitted from a remote operation room. The operation signal can also be generated by an operation device provided in the cab 10.

[0047] The electromagnetic valve unit 45 includes a plurality of electromagnetic valves arranged in each pilot pipe 25 connecting the pilot pump 15 and the pilot port of each control valve in the control valve unit 17.

[0048] In the present embodiment, the controller 30 is able to control the pilot pressure acting on the pilot port of each control valve by individually controlling the opening areas of the plurality of electromagnetic valves. Therefore, the controller 30 is able to control the flow rate of the working oil flowing into each hydraulic actuator and the flow rate of the working oil flowing out of each hydraulic actuator, and further, is able to control the operation of each hydraulic actuator.

[0049] Thus, the controller 30 is able to implement the raising and lowering of the boom 4, the opening and closing of the stick 5, the opening and closing of the bucket 6, the rotation of the upper swing body 3, and the travel of the lower traveling body 1, and the like, in accordance with an operation signal from the outside, such as the remote operation room.

[0050] Figure 3 is a diagram showing a configuration example of a power system mounted on a shovel shown in Figure 1 Figure 3 As shown in FIG. 1, the engine 11 is connected to an engine control unit 74. Various data indicating the state of the engine 11 are transmitted from the engine control unit 74 to the controller 30. The controller 30 is configured to be able to store the various data indicating the state of the engine 11 in the memory 30a.

[0051] The battery 70 is configured to supply electric power to various electric loads mounted on the shovel 100. The alternator 11a (generator), the starter 11b, the controller 30, and the electrical equipment 72, and the like are configured to operate by the electric power stored in the battery 70. The starter 11b is configured to be driven by the electric power stored in the battery 70 and to start the engine 11. Also, the battery 70 is configured to be charged by the electric power generated by the alternator 11a.

[0052] The water temperature sensor 11c transmits data related to the temperature of the engine cooling water to the controller 30. The governor 13 transmits data related to the swash plate deflection angle to the controller 30. The discharge pressure sensor 14b transmits data related to the discharge pressure of the main pump 14 to the controller 30. The positioning device 18 transmits data related to the position of the shovel 100 to the controller 30.

[0053] An oil temperature sensor 14c is provided in a pipe 14-1 between a working oil tank storing the working oil sucked by the main pump 14 and the main pump 14. The oil temperature sensor 14c transmits data related to the temperature of the working oil flowing through the pipe 14-1 to the controller 30.

[0054] A urea water remaining amount sensor 21a provided in the urea water tank 21 transmits data related to the remaining amount of urea water to the controller 30. A fuel remaining amount sensor 22a provided in the fuel tank 22 transmits data related to the remaining amount of fuel to the controller 30.

[0055] ​The communication device T1 is configured to transmit and receive information with the communication device T2 provided in the remote operation room RC via wireless communication. In the present embodiment, the communication device T1 and the communication device T2 are configured to transmit and receive information via a 5th generation mobile communication line (5G line), an LTE line, a satellite line, or the like.

[0056] In the remote operation room RC, a remote controller 40, a sound output device A2, an indoor space recognition device C2, a display device D1, a communication device T2, and the like are provided. Also, in the remote operation room RC, a driver seat DS on which an operator OP who remotely operates the shovel 100 sits is provided.

[0057] The remote controller 40 is an arithmetic device that performs various kinds of arithmetic operations. In the present embodiment, like the controller 30, the remote controller 40 is configured by a microcomputer including a CPU and a memory. Then, various functions of the remote controller 40 are realized by the CPU executing a program stored in the memory.

[0058] The sound output device A2 is configured to output sound. In the present embodiment, the sound output device A2 is a speaker configured to play sound collected by the sound collecting device A1 mounted on the shovel 100.

[0059] The indoor space recognition device C2 is configured to be able to recognize a space in the remote operation room RC. In the present embodiment, the indoor space recognition device C2 is a camera provided inside the remote operation room RC and is configured to take an image of the operator OP who sits on the driver seat DS.

[0060] The communication device T2 is configured to control wireless communication with the communication device T1 mounted on the shovel 100.

[0061] In the present embodiment, the driver seat DS has the same structure as a driver seat provided in a driver's cabin of a general shovel. Specifically, a left console box is arranged on the left side of the driver seat DS, and a right console box is arranged on the right side of the driver seat DS. Also, a left lever is arranged on the front end of the upper surface of the left console box, and a right lever is arranged on the front end of the upper surface of the right console box. Also, a travel lever and a travel pedal are arranged in front of the driver seat DS. Further, a control panel 75 is arranged on the central portion of the upper surface of the right console box. The left lever, the right lever, the travel lever, the travel pedal, and the control panel 75 each constitute an operation device 26.

[0062] The control panel 75 is a control panel for adjusting the number of revolutions of the engine 11 and is configured to be able to switch the engine revolutions in four stages, for example.

[0063] Specifically, the control dial 75 is configured to switch the engine speed in four stages of an SP mode, an H mode, an A mode, and an idle mode. The control dial 75 transmits data related to the setting of the engine speed to the controller 30.

[0064] The SP mode is a speed mode selected when the operator OP desires to prioritize the work amount, and the highest engine speed is used. The H mode is a speed mode selected when the operator OP desires to balance the work amount and the fuel consumption rate, and the second highest engine speed is used. The A mode is a speed mode selected when the operator OP desires to prioritize the fuel consumption rate while operating the shovel at low noise, and the third highest engine speed is used. The idle mode is a speed mode selected when the operator OP desires to set the engine to an idle state, and the lowest engine speed is used. Then, the engine 11 is controlled to a constant speed at the engine speed of the speed mode selected via the control dial 75.

[0065] An operation sensor 29 for detecting the operation content of the operation device 26 is provided in the operation device 26. The operation sensor 29 is, for example, a tilt sensor that detects the tilt angle of the operation lever, or an angle sensor that detects the swing angle of the operation lever around the swing axis, or the like. The operation sensor 29 can be constituted by a pressure sensor, a current sensor, a voltage sensor, or another sensor such as a distance sensor. The operation sensor 29 outputs information related to the detected operation content of the operation device 26 to the remote controller 40. The remote controller 40 generates an operation signal based on the received information, and transmits the generated operation signal to the shovel 100. The operation sensor 29 can be configured to generate the operation signal. In this case, the operation sensor 29 can output the operation signal to the communication device T2 without passing through the remote controller 40.

[0066] The display device D1 is configured to display information related to the situation around the shovel 100. In the present embodiment, the display device D1 is a multi-monitor constituted by nine monitors of three levels vertically and three columns horizontally, and is configured to be able to display the states of the spaces in front of, to the left of, and to the right of the shovel 100. Each monitor is a liquid crystal monitor or an organic EL monitor, or the like. However, the display device D1 can be constituted by one or a plurality of curved monitors, or can be constituted by a projector.

[0067] The display device D1 can be a display device that the operator OP can wear. For example, the display device D1 is a head-mounted display (VR goggles), which can be configured to be able to transmit and receive information between the remote controller 40 through wireless communication. The head-mounted display can be connected to the remote controller 40 by wire. The head-mounted display can be a see-through type head-mounted display, or a non-see-through type head-mounted display. The head-mounted display can be a monocular head-mounted display, or a binocular head-mounted display.

[0068] The display device D1 is configured to display an image that enables the operator OP in the remote operation room RC to visually recognize the surroundings of the shovel 100. That is, the display device D1 displays an image so that the operator OP in the remote operation room RC can confirm the situation around the shovel 100 as if the operator were in the cab 10 of the shovel 100.

[0069] Next, the relationship between the first coordinate system with the reference point R1 in the remote operation room RC as the origin and the second coordinate system with the reference point R2 in the shovel 100 as the origin will be described with reference to Figure 4 Figure 4 is a diagram showing the relationship between the operation room coordinate system and the shovel coordinate system.

[0070] The operation room coordinate system is a three-dimensional UVW orthogonal coordinate system with the reference point R1 in the remote operation room RC as the origin, and has a U-axis extending in parallel with the front-rear direction of the driver seat DS, a V-axis extending in parallel with the left-right direction of the driver seat DS, and a W-axis orthogonal to the U-axis and the V-axis.

[0071] The shovel coordinate system is a three-dimensional XYZ orthogonal coordinate system with the reference point R2 on the upper swing body 3 as the origin, and has an X-axis extending in parallel with the front-rear direction of the upper swing body 3, a Y-axis extending in parallel with the left-right direction of the upper swing body 3, and a Z-axis orthogonal to the X-axis and the Y-axis. In Figure 4 In the example of Figure 4 In the example of

[0072] In the present embodiment, each three-dimensional coordinate in the operation room coordinate system is associated in advance with one of the three-dimensional coordinates in the shovel coordinate system. Therefore, if the three-dimensional coordinate of the eye position of the operator OP in the remote operation room RC, i.e., the operator eye point E1, is determined, the three-dimensional coordinate of the eye position of the hypothetical operator in the shovel 100, i.e., the hypothetical operator eye point E1', is uniquely determined. In addition, the eye position of the operator OP is, for example, a point midway between the left eye position and the right eye position of the operator OP. However, the eye position of the operator OP can be a position set in advance. That is, the operator eye point E1 and the hypothetical operator eye point E1' can be fixed points.

[0073] ​In the above-described embodiments, the display device D1 is provided in front of the operator OP, in front left of the operator OP, and in front right of the operator OP, but can be provided in a square tubular shape or a circular tubular shape so as to surround the operator OP. That is, the display device D1 can include a monitor provided behind the operator OP. Alternatively, the display device D1 can be provided in a hemispherical shape so as to surround the operator OP. That is, the display device D1 can include a monitor provided directly above the operator OP.

[0074] Next, the structure of the construction support system SYS of the shovel 100 will be described with reference to Figure 5 and Figure 6 . Figure 5 is a schematic diagram showing a structure example of the construction support system SYS. Figure 6 is a functional block diagram showing a structure example of the construction support system SYS.

[0075] The construction support system SYS is mainly composed of the positioning device 18 mounted on the shovel 100, the controller 30, the solenoid valve unit 45, the sound collecting device Al, the space recognition device Cl, and the communication device Tl, the operation sensor 29, the remote controller 40, the sound output device A2, the indoor space recognition device C2, the display device Dl, and the communication device T2 provided in the remote operation room RC, and the controller 50 as a management device and the communication device T3 provided in the information center 200.

[0076] In Figure 5 the illustrated example, the construction support system SYS is composed of the shovel 100a, the shovel 100b, the remote operation room RCA related to the shovel 100a, the remote operation room RCb related to the shovel 100b, the space recognition device C3 provided at the work site, and the information center 200.

[0077] The space recognition device C3 is configured to be able to recognize a space within the work site. In the present embodiment, the space recognition device C3 is a camera provided at the work site and is configured to capture a state of the work site.

[0078] First, the functions possessed by the controller 30 mounted on the shovel 100a will be described. As shown in Figure 6 , the controller 30 has an image generation section 31, a shovel state determination section 32, and an actuator driving section 33 as functional blocks. The same applies to the shovel 100b. The image generation section 31, the shovel state determination section 32, and the actuator driving section 33 are distinguished for the sake of explanation, but need not be physically distinguished and can be constituted as a whole or a part thereof by a general-purpose software component or a hardware component.

[0079] The image generation section 31 is configured to generate a surrounding image including an image displayed in the display device Dl. The surrounding image is an image used when displayed in the display device Dl. Typically, the surrounding image is an image indicating a state of the surroundings of the shovel 100 that an operator can see when assumed to be present in the cab 10. In the present embodiment, the surrounding image is generated based on an image captured by the camera serving as the space recognition device Cl. Specifically, the image generation section 31 generates a first virtual viewpoint image as the surrounding image based on images captured by the rear camera ClB, the front camera ClF, the left camera ClL, and the right camera ClR. However, the image generation section 31 can generate the first virtual viewpoint image as the surrounding image based on an image captured by at least one of the rear camera ClB, the front camera ClF, the left camera ClL, and the right camera ClR. The first virtual viewpoint as the first virtual viewpoint image is a virtual operator viewpoint El' corresponding to an eye position of an operator when assumed to be seated on a driver's seat in the cab 10 (refer to FIG. 1). However, the virtual operator viewpoint El' can be outside the cab 10. Figure 4

[0080] In the present embodiment, the coordinates of the virtual operator viewpoint El' as the first virtual viewpoint are derived based on an eye position of the operator OP when assumed to be seated on the driver's seat DS in the remote control room RC, i.e., an operator viewpoint El (refer to FIG. 1). In addition, the coordinates of the operator viewpoint El are transmitted from the remote controller 40. The image generation section 31 can derive the coordinates of the virtual operator viewpoint El' by converting the coordinates of the operator viewpoint El in the room coordinate system into coordinates in the shovel coordinate system. However, the coordinates of the operator viewpoint El can be a fixed value set in advance. Figure 4

[0081] Also, in the present embodiment, the first virtual viewpoint image corresponds to an image projected onto an inner peripheral surface of a virtual cylindrical virtual projection surface surrounding the first virtual viewpoint. The virtual projection surface can be an inner surface of a virtual sphere or a virtual hemisphere surrounding the first virtual viewpoint, or an inner surface of a virtual cuboid or a virtual cube surrounding the first virtual viewpoint. By observing the first virtual viewpoint image thus generated, the operator OP can stereoscopically grasp the state of the surroundings of the shovel 100. That is, the operator OP can more accurately grasp, for example, a depth of a dump truck bed located in front of the shovel 100, a height of a fill on the ground, or a depth of a pit on the ground, by observing the first virtual viewpoint image.

[0082] The image derived from the first virtual viewpoint image displayed in the display device Dl is a portion of the first virtual viewpoint image generated by the image generation section 31.

[0083] ​​In addition, in the case where the display device D1 is a head-mounted display, the region of the image displayed in the display device D1 in the entire region of the first virtual viewpoint image can be determined in accordance with the line-of-sight direction of the operator OP seated on the driver seat DS of the remote control room RC. In this case, information about the line-of-sight direction of the operator OP is transmitted from the remote controller 40. The image generation section 31 generates the first virtual viewpoint image as the surrounding image in accordance with the image output from the space recognition device C1 and the coordinates of the operator viewpoint E1 transmitted from the remote controller 40. Then, the image generation section 31 cuts out a part of the generated first virtual viewpoint image as the partial surrounding image in accordance with the information about the line-of-sight direction of the operator OP transmitted from the remote controller 40, and transmits the cut-out partial surrounding image to the display device D1 located in the remote control room RC.

[0084] The shovel state determination section 32 is configured to determine the state of the shovel 100. In the present embodiment, the state of the shovel 100 includes the position and the orientation of the shovel 100. The position of the shovel 100 is, for example, the latitude, the longitude, and the altitude of the reference point R2 in the shovel 100. The shovel state determination section 32 determines the position and the orientation of the shovel in accordance with the output of the positioning device 18.

[0085] The actuator driving section 33 is configured to drive the actuators mounted on the shovel 100. In the present embodiment, the actuator driving section 33 generates and outputs an operation signal to each of the plurality of electromagnetic valves included in the electromagnetic valve unit 45 in accordance with the operation signal transmitted from the remote controller 40.

[0086] Each of the electromagnetic valves receiving the operation signal increases or decreases the pilot pressure acting on the pilot port of the corresponding control valve in the control valve unit 17. As a result, the hydraulic actuator corresponding to each control valve operates at a speed corresponding to the stroke amount of the control valve.

[0087] Next, the functions of the remote controller 40 provided in the remote control room RC will be described. The remote controller 40 has an operator state determination section 41, an image synthesis section 42, and an operation signal generation section 43 as functional blocks. The operator state determination section 41, the image synthesis section 42, and the operation signal generation section 43 are distinguished for the sake of explanation, but do not need to be physically distinguished, and can be constituted as a whole or a part thereof by a general-purpose software component or a hardware component.

[0088] The operator state determination section 41 is configured to determine the state of the operator OP in the remote control room RC. The state of the operator OP includes the eye position and the line-of-sight orientation of the operator OP. The operator state determination section 41 determines the eye position and the line-of-sight orientation of the operator OP from the output of the indoor space recognition device C2. Specifically, the operator state determination section 41 performs various image processing on the image captured by the camera as the indoor space recognition device C2, and determines the coordinates of the eye position of the operator OP in the operator room coordinate system as the coordinates of the operator eye point El (see FIG. 2). Also, the operator state determination section 41 performs various image processing on the image captured by the camera as the indoor space recognition device C2, and determines the line-of-sight orientation of the operator OP in the operator room coordinate system. Figure 4

[0089] The operator state determination section 41 can also derive the coordinates of the operator eye point El and the line-of-sight orientation of the operator OP from the output of another device than the indoor space recognition device C2, such as a LIDAR provided in the remote control room RC or an inertial measurement device mounted in a head-mounted display as the display device Dl. Note that the inertial measurement device can include a positioning device.

[0090] Then, the operator state determination section 41 transmits information about the coordinates of the operator eye point El and the line-of-sight orientation of the operator OP to the shovel 100 via the communication device T2.

[0091] The image synthesis section 42 is configured to synthesize the partial surrounding image transmitted from the controller 30 and another image to generate a synthesized image.

[0092] The other image can be an image generated based on the design surface information DG, i.e., a design surface image. In the present embodiment, the image synthesis section 42 superimposes a figure such as a computer graphic representing the design surface position as the design surface image on the partial surrounding image based on the design surface information DG stored in advance in a nonvolatile storage device constituting the remote controller 40. The design surface is the ground surface at the time when the excavation work using the shovel 100 is completed. The operator can grasp the state of the surroundings of the shovel 100 at the time when the excavation work is completed by observing the design surface even before the completion of the excavation work. In this case, the image synthesis section 42 determines the position in the partial surrounding image where the design surface image should be superimposed and displayed based on the position and the orientation of the shovel determined by the shovel state determination section 32.

[0093] The operation signal generation section 43 is configured to generate an operation signal. In the present embodiment, the operation signal generation section 43 is configured to generate an operation signal based on the output of the operation sensor 29.

[0094] ​Next, the functions that the controller 50 provided in the information center 200 has will be described. The controller 50 is an arithmetic device that performs various kinds of arithmetic operations. In the present embodiment, like the controller 30 and the remote controller 40, the controller 50 is configured by a microcomputer including a CPU and a memory. Then, various functions of the controller 50 are realized by the CPU executing a program stored in the memory.

[0095] In the present embodiment, the controller 50 has a determination section 51, an operation prediction section 52, an operation intervention section 53, and a motion simulator 54 as functional blocks. The determination section 51, the operation prediction section 52, the operation intervention section 53, and the motion simulator 54 are distinguished for the sake of explanation, but need not be physically distinguished, and can be constituted in whole or in part by a general-purpose software component or a hardware component.

[0096] The determination section 51 is configured to determine whether there is a matter that should be notified to the operator of the shovel 100, with respect to the surrounding situation of the shovel 100. In the present embodiment, the determination section 51 is configured to determine whether there is a matter that should be notified to the operator of the shovel 100, on the basis of at least one of an image or a distance image (hereinafter referred to as "image or the like") captured by the space recognition device CI as an information acquisition device installed on the shovel 100, the position, the posture, and the motion content of the shovel 100. The distance image is, for example, an image generated on the basis of the output of a LIDAR or the like as the space recognition device CI. The determination section 51 can also be configured to be able to determine at least one of the position, the posture, and the motion content of the shovel 100, on the basis of the image or the like captured by the space recognition device CI. Also, the determination section 51 can be configured to determine whether there is a matter that should be notified to the operator of the shovel 100, on the basis of the image or the like captured by the space recognition device C3 or the construction site topography information (topography data). Further, the determination section 51 can be configured to be able to determine at least one of the position, the posture, and the motion content of the other construction machine, on the basis of the image or the like captured by the space recognition device C3. The determination section 51 can also be configured to determine whether there is a matter that should be notified to the operator of the shovel 100, on the basis of the surrounding situation of the shovel 100 derived from the image or the like acquired by the space recognition device CI and the space recognition device C3, and the position, the posture, and the motion content of the shovel 100. Whether there is a matter that should be notified can be determined by collating with past cases, on the basis of the presence or absence of the same or similar situation.

[0097] For example, the determination section 51 determines that there is a matter that should be notified to the operator in a case where it is detected that a person exists outside the range covered by the image displayed on the display device Dl. For example, the determination section 51 determines that there is a matter that should be notified to the operator in a case where it is detected that a person exists to the rear left of the shovel 100. In this case, the determination section 51 can detect the person on the basis of the output of the camera or the LIDAR or the like installed on the upper swing body 3 as the space recognition device C l. Alternatively, the determination section 51 can detect the person on the basis of the output of the camera or the LIDAR or the like provided at the work site as the space recognition device C3. In this case, the space recognition device C3 can be, for example, a hemispherical camera installed at the front end of a pole provided at the work site. Further, the space recognition device C3 can be a camera or a LIDAR or the like installed on another construction machine, or a camera or a LIDAR or the like installed on a flying object such as a multicopter (drone) that flies in the sky above the work site. The same applies to a case where it is detected that a person exists inside the range covered by the image displayed on the display device Dl.

[0098] Alternatively, the determination section 51 can determine that there is a matter that should be notified to the operator in a case where it is detected that a wire exists outside the range covered by the image displayed on the display device Dl. For example, the determination section 51 determines that there is a matter that should be notified to the operator in a case where it is detected that a wire exists above the shovel 100. In this case, the determination section 51 can detect the wire on the basis of the output of the space recognition device C l. Alternatively, the determination section 51 can detect the wire on the basis of the image or the like captured by the space recognition device C3. The same applies to a case where it is detected that a wire exists inside the range covered by the image displayed on the display device Dl.

[0099] Alternatively, the determination section 51 can determine that there is a matter that should be notified to the operator in a case where it is detected that a downhill exists in front of the shovel 100 on the basis of the construction terrain information (terrain data). For example, the determination section 51 determines that there is a matter that should be notified to the operator in a case where it is detected that a downhill exists in front of the shovel 100. In this case, the determination section 51 can detect the downhill on the basis of the output of the object detection device. Alternatively, the determination section 51 can detect the downhill on the basis of the image or the like captured by the space recognition device C3. Alternatively, the determination section 51 can detect the downhill on the basis of the construction terrain information (terrain data) that is stored in advance in a nonvolatile storage medium or the like attached to the controller 50.

[0100] In a case where it is determined that there is a matter that should be notified to the operator of the shovel 100, the determination section 51 calls the operator's attention. In the present embodiment, the determination section 51 transmits information about the matter that should be notified to the remote controller 40. The image synthesizing section 42 of the remote controller 40 causes an image about the information received from the determination section 51 to be superimposed and displayed on the partial surrounding image.

[0101] The operation prediction section 52 is configured to predict an operation signal after a prescribed time, based on an operation signal received from the remote controller 40. This is to suppress a decrease in operation responsiveness caused by communication delay, that is, a delay until an operation by the operator OP in the remote operation room RC is reflected in movement of the shovel 100. The prescribed time is, for example, several milliseconds to several tens of milliseconds. For example, the operation prediction section 52 predicts an operation signal after a prescribed time, based on a change in an operation signal (a tilt angle of an operation lever) in the past prescribed time. For example, the operation prediction section 52 predicts that a tilt angle after a prescribed time is larger than a current tilt angle, in a case where it is detected that there is a tendency for the tilt angle of the operation lever to increase in the past prescribed time.

[0102] Then, the operation prediction section 52 transmits the predicted operation signal (hereinafter, referred to as a "predicted operation signal") to the shovel 100, instead of directly transmitting an operation signal received from the remote controller 40 to the shovel 100.

[0103] According to this structure, the operation prediction section 52 can substantially transmit an operation signal generated in the remote operation room RC to the shovel 100 without delay.

[0104] The operation intervention section 53 is configured to intervene in an operation by the operator OP in the remote operation room RC. In the present embodiment, the determination section 51 is configured to determine whether or not to intervene in an operation by the operator OP, based on an image and the like captured by the space recognition device CI installed on the shovel 100.

[0105] For example, the operation intervention section 53 determines that the operation of the operator OP should be intervened in a case where the shovel 100 is detected to possibly contact with an object located around the shovel 100. For example, the operation intervention section 53 determines that the operation of the operator OP should be intervened in a case where a person is detected to exist on the left side of the shovel 100, and a left turning operation (an operation of tilting the left operation lever to the left side) is detected to be started. In this case, the operation intervention section 53 invalidates the operation signal generated in accordance with the left turning operation, so as to cause the upper swing body 3 not to turn left. In addition, the operation intervention section 53 can detect that the shovel 100 possibly contacts with an object located around the shovel 100, in accordance with the output of the object detection device. Alternatively, the determination section 51 can also detect that the shovel 100 possibly contacts with an object located around the shovel 100, in accordance with the image or the like captured by the space recognition device C3. In a case where it is thus determined that there is a matter that should be notified to the operator, the controller 30 can be configured to perform braking control such as stop or deceleration of the shovel 100 in accordance with the operation signal.

[0106] Then, the operator can cancel the braking control such as stop or deceleration of the shovel 100, for example, by performing an operation of returning the operation lever to the neutral position temporarily or pressing a cancel button or the like, that is, by satisfying a cancel condition. In addition, the cancel condition can include that the shovel 100 is in a stopped state.

[0107] The action simulator 54 is configured to be able to perform an action simulation experiment (simulation experiment) of the shovel 100. The action of the shovel 100 constitutes various works such as excavation work of the shovel 100. For example, the excavation work is constituted by a bucket lever closing action, a bucket closing action, and a boom lifting action, and the like. In the example shown in FIG. 1, the action simulator 54 is configured to be able to perform the simulation experiment of the action of the shovel 100 in accordance with the operation signal from the remote operation room RC. Figure 6 In the example shown, the action simulator 54 starts the simulation experiment of the action of the shovel 100 in accordance with a start instruction from the remote operation room RC. Specifically, the action simulator 54 constructs a virtual model of a work site, that is, a virtual work site, in accordance with environment information. The environment information is information related to a work environment, and for example, includes information output from at least one of the space recognition device CI, the space recognition device C3, the positioning device 18, and the posture detection sensor. Also, the environment information can include the design surface information DG.

[0108] The virtual work site is an example of a virtual environment, for example, a three-dimensional virtual space (three-dimensional model) that reproduces the current terrain of the actual work site. Then, the motion simulator 54 transmits images relating to the virtual work site to the display device Dl so that the operator OP in the remote control room RC can visually recognize the situation of the virtual work site. The virtual shovel is arranged in the three-dimensional virtual space. In order to make the current terrain in the virtual work site and the position and orientation of the virtual shovel consistent with the current terrain in the actual work site and the position and orientation of the actual shovel 100, the virtual shovel is arranged in the virtual work site reproduced in the virtual space. As with the actual shovel 100, the virtual shovel performs virtual motions in the virtual work site. That is, if the operator OP switches the connection destination of the operating device 26 from the actual shovel 100 to the virtual shovel, the operator OP can operate the virtual shovel via the operating device 26. For example, if the operator OP operates the left operating lever (arm lever) of the operating device 26, the virtual arm lever of the virtual shovel can be moved. In this way, the operator OP can cause each actuator of the virtual shovel to perform a motion in the virtual work site. Also, in the case where there are settings such as buildings or electric wires in the actual work site, virtual settings are also reproduced in the virtual work site. Also, in the case where materials are scheduled to be carried in at a predetermined time in the actual work site, the situation where virtual materials are carried in at a predetermined time is also reproduced in the virtual work site. Also, in the case where it is predicted that it will rain at a predetermined time, the situation where it rains at a predetermined time is also virtually reproduced in the virtual work site. In this way, the three-dimensional virtual space is reproduced as a three-dimensional model. Thus, the operator OP of the remote control room RC can perform virtual work in the virtual work site reproduced in the virtual space using the model of the virtual shovel arranged in the three-dimensional virtual space. When the virtual work is performed, the image visually recognized by the operator OP of the remote control room RC corresponds to an image of the virtual work site obtained from a virtual space recognition device mounted on the virtual shovel arranged in the three-dimensional virtual space. The virtual space recognition device is arranged, for example, in the virtual cab. The image relating to the virtual work site is typically a stereoscopic terrain image corresponding to the terrain of the actual work site, which is composed of computer graphics. However, at least a part of the image relating to the virtual work site can be generated using an image captured by a camera. The display device Dl that receives the image relating to the virtual work site can display an image of the virtual work site.The current topography of the actual work site can be acquired by a space recognition device (a camera or a LIDAR, etc.) provided in the shovel 100, a space recognition device (a camera or a LIDAR, etc.) provided in a multicopter, etc., or a space recognition device (a camera or a LIDAR, etc.) provided in a building or a tower, etc. in the actual work site. In this way, the action simulator 54 can reproduce the actual work site in the three-dimensional imaginary space (three-dimensional model) based on the information acquired by the space recognition device. The imaginary work site in the three-dimensional imaginary space (three-dimensional model) can be updated according to the progress status in the actual work site. For example, in the case where a tree has fallen in the actual work site, in the shovel 100, the space recognition device C1 acquires information (position, size, or kind of the tree, etc.) of the tree falling in the actual work site and transmits the information to the controller 50 provided in the information center 200 as a management device. The controller 50 reflects the latest information of the actual work site received to the imaginary work site. Thus, the operator OP can operate the imaginary shovel in the imaginary work site and perform the imaginary work taking into account the tree falling.

[0109] The image related to the imaginary work site typically includes an image of the imaginary shovel. The imaginary shovel corresponds to the shovel 100 in the actual work site, for example, at the time when the simulation test of the action of the shovel 100 is started. That is, the position and posture of the imaginary shovel in the imaginary work site at the time when the simulation test is started correspond to the position and posture of the shovel 100 in the actual work site. The position and posture of the imaginary shovel in the imaginary work site at the time when the simulation test is started are determined, for example, based on the output of at least one of the space recognition device C1 and the space recognition device C3. However, the position and posture of the imaginary shovel in the imaginary work site at the time when the simulation test is started can be determined or adjusted based on the output of at least one of the boom angle sensor S1, the stick angle sensor S2, the bucket angle sensor S3, the swing angular velocity sensor S4, the body inclination sensor, and the positioning device 18, etc. mounted on the shovel 100.

[0110] Then, the action simulator 54 receives an operation signal generated when the operator OP in the remote operation room RC operates the operation device 26 from the remote operation room RC and causes the imaginary shovel in the imaginary work site to act based on the received operation signal. The topography of the imaginary work site changes according to the movement of the imaginary shovel. For example, the topography of the imaginary work site changes according to the imaginary excavation work of the imaginary shovel.

[0111] When the simulation test is performed, the operation signal is not transmitted to the shovel 100 in the actual work site. That is, the operator OP in the remote operation room RC cannot operate the shovel 100 in the actual work site when the simulation test is performed.

[0112] In Figure 6 In the example shown, when the operator OP in the remote operation room RC operates a prescribed start button, a start instruction for starting the simulation test is generated by the remote controller 40, and is sent from the remote controller 40 to the controller 50 of the information center 200. The start button is provided, for example, on the upper surface of the right console box.

[0113] Since the operator OP operates from the remote operation room RC, it is sometimes difficult to grasp the situation of the actual work site. Therefore, it is desirable that the operator OP grasp the probability that an undesirable event occurs in the actual work site where the shovel 100 is located before the work starts or during the work. Therefore, the operator OP starts the simulation test by operating a prescribed start button before actually performing the predetermined work. The undesirable event is, for example, a cliff collapse when excavating a cliff, and the like. The cliff here refers to, for example, a ground surface including a sloped surface having an inclination angle of a rest angle or more. The operator OP can confirm how the cliff collapses when excavating the cliff by performing a hypothetical excavation work of a hypothetical shovel in a hypothetical work site. That is, the operator OP can confirm in what order and to what extent excavating which part of the cliff can suppress the cliff collapse by hypothetically trying a plurality of excavation works. That is, the operator OP can derive a method of performing the work without problems before performing the actual excavation work (without performing the actual excavation work).

[0114] Specifically, the operator OP can restore the topography of the hypothetical work site changed by trying one excavation work to the original topography by operating a prescribed reset button after hypothetically trying one excavation work. The reset button is provided, for example, on the upper surface of the right console box. In addition, the original topography is, for example, the topography of the hypothetical work site at the time when the start button is operated, and corresponds to the current topography in the actual work site. However, the topography of the hypothetical work site can be restored to the topography at any time in the simulation test. According to this structure, the operator OP can efficiently try various excavation works.

[0115] Then, the operator OP ends the simulation test after confirming the excavation work that does not easily cause the cliff collapse. In Figure 6 In the example shown, the operator OP can end the simulation test by operating a prescribed end button. The end button can be the same button as the start button used at the start of the simulation test, or can be a different button from the start button.

[0116] If the simulation test ends, the image of the virtual work site displayed on the display device Dl is switched to an image of the actual work site based on an image captured by the camera or the like mounted on the shovel 100. Then, the operator OP can move the shovel 100 by operating the operation device 26, and thus can perform the actual excavation work.

[0117] In this state, the operator OP can perform the excavation work considered to be the best by the simulation test to excavate the cliff in the actual work site.

[0118] According to this configuration, the controller 50 can suppress, for example, a cliff collapse that can occur when excavating a cliff, and can improve the safety of the excavation work of the shovel 100.

[0119] Alternatively, the controller 50 can be configured to urge the operator OP to perform the simulation test on the basis of stopping the movement of the shovel 100 in a case where it is determined that the probability of occurrence of an undesirable event according to the movement of the shovel 100 is high. The undesirable event is, for example, a cliff collapse that can occur when excavating a cliff.

[0120] For example, the controller 50 can be configured to urge the operator OP to perform the simulation test on the basis of stopping the movement of the shovel 100 in a case where it is determined that the operator OP intends to excavate a cliff on the basis of the output of the space recognition device Cl or the like.

[0121] Next, the effect of the simulation test performed by the action simulator 54 in the controller 50 will be described with reference to Figure 7 Figure 7 is a side view of the shovel 100. In Figure 7 In the example shown, the operator OP of the shovel 100 intends to perform the excavation work of excavating the cliff CL1 to expose the design surface TS. In addition, the operator OP remotely operates the shovel 100 using the operation device 26 provided in the remote operation room RC. The operator OP presses the start button provided in the remote operation room RC to start the simulation test before excavating the cliff CL1.

[0122] If the start button is pressed, the remote controller 40 provided in the remote operation room RC generates a start instruction, and transmits the start instruction to the controller 50 (the action simulator 54) of the information center 200.

[0123] The action simulator 54 that receives the start instruction recognizes the topography of the actual work site around the shovel 100 on the basis of the output of the LIDAR installed on the shovel 100 as the space recognition device Cl, and generates a three-dimensional topographic image of the virtual work site corresponding to the topography of the work site.

[0124] ​The stereoscopic terrain image generated by the action simulator 54 is sent to a display device Dl provided in the remote operation room RC, and displayed on the display device Dl.

[0125] The operator OP can operate the operation device 26 while observing the stereoscopic terrain image displayed on the display device Dl, to cause the virtual shovel to perform an action.

[0126] In Figure 7 In the illustrated example, the operator OP can attempt, by simulation test, for example, a first digging work of first digging the sand portion SP1 divided by the dotted line, and a second digging work of first digging the sand portion SP2 divided by the single-dot chain line.

[0127] Then, the operator OP can confirm, when the first digging work is attempted, that the collapse of the sand portion SP3 indicated by the slant line occurs when the sand portion SP1 is first dug. Also, the operator OP can confirm, when the second digging work is attempted, that the collapse of other sand portions does not occur even if the sand portion SP2 is first dug. In this case, the operator OP can perform the second digging work as the digging work implemented in the actual work site, based on the simulation test result. Or, the operator OP can perform the first digging work as the digging work implemented in the actual work site, in order to actively utilize the collapse of the sand portion SP3.

[0128] Next, reference will be made to Figure 8 to describe another configuration example of the construction support system SYS of the shovel 100. Figure 8 is a functional block diagram showing another configuration example of the construction support system SYS, and corresponds to Figure 6 .

[0129] Figure 8 The controller 30 of the construction support system SYS shown in the drawing is mounted on the shovel 100, and has an abnormality detection section 34, which is different from the construction support system SYS shown in Figure 6 in other respects from the construction support system SYS shown in Figure 6 . Therefore, the following description will omit the description of common parts, and will describe the different parts in detail.

[0130] The abnormality detection section 34 is configured to detect an abnormal event occurring in the vicinity of the shovel 100. In Figure 8 the illustrated example, the abnormality detection section 34 is configured to detect an abnormal event occurring in the vicinity of the shovel 100 in advance from the output of the space recognition device Cl. This is to prevent the occurrence of the abnormal event in advance. The abnormal event occurring in the vicinity of the shovel 100 is, for example, the fall of the shovel 100 from the cliff CL2 (refer to Figure 7 . ) or the contact of the digging attachment AT of the shovel 100 with the electric wire EW (refer to Figure 7. ) contact, etc.

[0131] For example, the anomaly detection unit 34 identifies the presence of cliff CL2 based on the output of the space recognition device C1. If the shovel 100 intrudes within a predetermined distance from the cliff CL2, the unit determines that the ground near the cliff CL2 is at risk of collapse. The predetermined distance may be a preset value based on at least one of data related to past events, the properties of the ground, and data related to the weight of the shovel 100, or may be a dynamically determined value.

[0132] When the excavator 100 approaches a specified distance range from the cliff CL2, the abnormality detection unit 34 may draw the attention of the operator OP of the excavator 100 by displaying information related to the abnormality on the display device D1, or may slow down or stop the movement of the excavator 100.

[0133] Alternatively, the abnormality detection unit 34 detects the presence of the electric wire EW based on the output of the space recognition device C1, for example, and determines that there is a possibility that the excavating attachment AT is in contact with the electric wire EW if the upper end of the excavating attachment AT (arm 5) intrudes within a predetermined distance from the electric wire EW. The predetermined distance may be a preset value based on data related to past events, or may be a dynamically determined value.

[0134] When the upper end of the excavating attachment AT approaches a predetermined distance from the electric wire EW, the abnormality detecting unit 34 may call the attention of the operator OP of the shovel 100 , slow down the movement of the excavating attachment AT, or stop the movement of the excavating attachment AT.

[0135] Furthermore, the abnormality detection unit 34 may be configured to transmit information related to the recognition result of objects around the shovel 100 and the determination result of whether an abnormal event is likely to occur to the information center 200. This is to enable the motion simulator 54 in the information center 200 to use this information.

[0136] In this case, the motion simulator 54 is configured to call the operator OP's attention when the operator OP moves the virtual excavator close to the cliff CL2 during the simulation test. Alternatively, the motion simulator 54 may be configured to call the operator OP's attention by displaying information regarding an abnormality on the display device D1 when the operator OP raises the boom 4 so that the upper end of the excavating attachment (arm 5) approaches the electric wire EW within a predetermined distance during the simulation test. With this configuration, the motion simulator 54 can alert the operator OP to the presence of the cliff CL2 or the electric wire EW during these simulation tests and urge the operator OP to pay attention to the cliff CL2 or the electric wire EW.

[0137] In the above-described embodiment, the shovel 100 is operated by the operator OP in the remote operation room RC, but can be operated by the operator in the cab 10. In this case, the start button, the reset button, and the end button are provided in the cab 10. Alternatively, the shovel 100 can be an automatic operation shovel (unmanned shovel) that does not need to be operated by an operator. In this case, the start button, the reset button, and the end button can be omitted.

[0138] In a case where the shovel 100 is an automatic operation shovel, the shovel 100 is configured to perform work by using an instruction (action instruction) related to a series of actions that are set in advance. The action instruction is basically determined in accordance with a work preparation procedure. The work preparation procedure refers to determination of what movement the shovel 100 performs in what order. For example, the work preparation procedure refers to determination of what part of a work site is excavated in what order in consideration of various factors such as work efficiency and work safety. In a work site where a general manned shovel is used, typically, the work preparation procedure is determined in accordance with an experienced judgment of a skilled operator. In a work site where an automatic operation shovel is used, basically, the work preparation procedure is also determined to be the same as the preparation procedure determined in accordance with the experienced judgment of the skilled operator. Therefore, the work preparation procedure in a work site where an automatic operation shovel is used is determined in accordance with information related to the work site and various data in the past. At this time, a technique related to machine learning such as deep learning can be used.

[0139] Specifically, the action instruction is determined, for example, by setting an orbit followed by a prescribed portion such as a tip of the shovel 6 in advance. Then, the action instruction is set by an arbitrary method. For example, the action instruction can be automatically generated in accordance with data related to an orbit in the past. In the generation of the action instruction, machine learning such as deep learning can be used. In this case, the "generation of the action instruction" is also referred to as "learning of the action instruction", or is also simply referred to as "action learning".

[0140] The operator OP causes the imaginary excavator to act in such a manner that the imaginary current ground surface within the three-dimensional imaginary space generated from the information output from the space recognition device C1 becomes a target surface (for example, a ground surface based on the design surface information DG or the like) within the imaginary space, and causes the controller 50 in the information center 200 to generate an operation command for the actual excavator 100. At this time, the controller 50 can also set a reward (use less fuel or short operation time, or the like) by reinforcement learning as an example of machine learning, and generate an operation command for the actuator. In this way, the controller 50 can generate an operation command with the highest reward when the imaginary current ground surface within the imaginary space is set to the target surface (for example, a ground surface based on the design surface information DG or the like) within the imaginary space by using reinforcement learning, and can realize an efficient work preparation process. Thus, the controller 50 can generate an operation command more efficient than a skilled operator, and can realize an efficient work preparation process.

[0141] Specifically, the controller 50 in the information center 200 generates an operation command for the excavator 100 based on the design surface information DG or the like. Then, the controller 50 generates an operation signal so that the excavator 100 can automatically act in accordance with the generated operation command. The generated operation signal is transmitted to the controller 30 mounted on the excavator 100.

[0142] Then, in the case where the excavator 100 is an automatic excavator, the action simulator 54 constituting the controller 50 performs a simulation test of the action constituting the work of the excavator 100 before the actual work is performed.

[0143] For example, the action simulator 54 virtually performs the action constituting the work of the excavator 100 based on information related to the operation command and information related to the current ground surface of the actual work site, whereby it is possible to virtually confirm the ground surface change from the start to the completion of the construction (work) of the excavator 100. Here, the construction includes one or more works (loading work, compaction work, excavation work, or lifting work, or the like) performed in a prescribed preparation process. Also, the work includes one or more actions (excavation action, turning action, dumping action, or boom lifting action, or the like) performed in a prescribed order.

[0144] Then, the action simulator 54 can identify in advance a problem (a matter to be noted as an abnormal event) that occurs when the excavator 100 is caused to act in accordance with the operation command. That is, the action simulator 54 can judge the place, timing, or type, or the like, at which the matter to be noted occurs. Also, in the case where a plurality of construction machines exist at the virtual work site, the action simulator 54 can judge which construction machine the matter to be noted occurs to. In this way, the action simulator 54 can perform extraction of the matter to be noted.

[0145] The extracted matters to be noted can be displayed on the display device of the management device before actual operation of the shovel 100 is performed. In this case, a three-dimensional virtual work site in which actual work site is reproduced is displayed on the display device. Then, the action simulator 54 displays the place, time, or kind, or the like, at which the matter to be noted occurs in the three-dimensional virtual work site. The action simulator 54 can display the cause of the matter to be noted that is to occur. Further, the action simulator 54 can reproduce the state of the virtual construction machine (virtual shovel) in the virtual work site before and after the occurrence of the matter to be noted. Thereby, the manager can confirm in advance how the matter to be noted occurs. Also, in a case where a plurality of construction machines exist in the virtual work site, the action simulator 54 can determine to which construction machine the matter to be noted occurs. For example, in the virtual work site, when the virtual material is carried in at a predetermined time (for example, 3 p.m.) according to the predetermined work content, in a case where the virtual construction machine (virtual shovel) is performing a digging work in the vicinity of the unloading (temporary placement) of the virtual material, "contact of the virtual shovel with the virtual material" is extracted as the matter to be noted. In this case, the place, time, or kind, or the like, at which the matter to be noted occurs in the virtual work site is displayed on the display device, whereby the manager can recognize that the temporary placement place of the material in the actual work site needs to be changed. Also, the action simulator 54 can display an improvement plan for eliminating the occurrence of the matter to be noted. The display device can be a display portion of a mobile terminal. In this case, the action simulator 54 performs a simulation test on, for example, a case where the unloading place of the virtual material is changed from the currently predetermined place to another place. The simulation test on the case where the unloading place is changed can be performed on a plurality of changed places. In this way, by performing simulation tests on a plurality of cases where the predetermined work content is changed, the action simulator 54 extracts a more preferable unloading place than the currently predetermined initial unloading place of the virtual material, and can display the preferable unloading place as the improvement plan. In this way, by displaying the improvement plan on the display device, the manager can instruct the worker of the work site to change the unloading place of the material in the actual work site. In this way, in a case where an event that occurs in the virtual work site is the matter to be noted, the action simulator 54 re-performs a simulation test by changing the work content once or a plurality of times in order to eliminate the occurred matter to be noted. Then, in a case where the work content or work preparation procedure, or the like, for eliminating the occurred matter to be noted is derived, a simulation test related to the subsequent construction is continuously performed according to the derived work content or work preparation procedure, or the like. In this way, the action simulator 54 can also derive the work content for eliminating the matter to be noted by the simulation test in a case where the matter to be noted occurs. Then, the information on the work content or work preparation procedure, or the like, for eliminating the matter to be noted, which is derived by the action simulator 54, is displayed on the display device, whereby the manager or the worker can change the work content or work preparation procedure, or the like, in the actual work site.Further, the display device can be a display portion of a mobile terminal.

[0146] When the operation command is revised, the operation simulator 54 is able to find the optimal operation command by performing a simulation test of the operation of the shovel 100 in accordance with a plurality of new operation commands. In the generation of the plurality of new operation commands, as in the case of the initial operation command, machine learning such as deep learning can be utilized.

[0147] Further, the operation simulator 54 can be configured to, when actually operating the shovel 100 as an automatic operation shovel, perform a simulation test of what kind of event will occur at a future time after a prescribed time. According to this configuration, the operation simulator 54 is able to perform a hypothetical operation (hypothetical operation) a prescribed time in advance of the actual operation, and is able to recognize the occurrence of an unexpected event in advance. Then, the operation simulator 54 is able to prevent the occurrence of such an unexpected event in advance by revising the operation command in the case where an unexpected event is recognized in advance.

[0148] Further, the operation simulator 54 can be configured to, in the case where an event different from the assumed event occurs, perform a simulation test of the operation of the shovel 100 that constitutes the subsequent operation again. For example, the operation simulator 54 can be configured to, in the case where it is determined from the output of various sensors mounted on the shovel 100 that the sand is clayey or sandy compared to the assumed sand, perform a simulation test of the operation of the shovel 100 that constitutes the subsequent operation again. This is because a sandy ground collapses more easily than a clayey ground, and affects the estimation result of the terrain changed by the excavation operation or the like. That is, this is because the simulation test result derived on the premise of clay is not actually suitable for a more sandy ground.

[0149] For example, the controller 50 is able to derive the amount of sand collapsed from the side surface of a pit, that is, an excavation pit, formed by actual excavation and accumulated on the bottom of the excavation pit from the output of the space recognition device C1. Then, the controller 50 is able to derive the characteristics of the ground of the work object from the amount of sand accumulated on the bottom of the excavation pit. The characteristics of the ground are, for example, the degree of sandiness or the degree of clayeyness, or the like. Typically, the controller 50 is able to determine that the more the amount of sand accumulated on the bottom of the excavation pit, the greater the degree of sandiness.

[0150] Further, the controller 50 compares the state of the work site in the actual work site with the state of the hypothetical work site assumed in the hypothetical space based on the output of the space recognition device C1, whereby it is possible to evaluate the progress of the work. For example, in the case where the actual work is delayed from the hypothetical work, the controller 50 re-executes the simulation test based on the state of the delayed actual work site by the action simulator 54 and re-generates the action command. Then, the controller 50 transmits the re-generated action command to the shovel 100. The shovel 100 is controlled based on the re-generated action command. Further, the controller 50 compares the state of the work site in the actual work site with the state of the hypothetical work site assumed in the hypothetical space based on the output of the space recognition device C1, whereby it is possible to determine whether an event not assumed (simulation test) occurs in the hypothetical work site.

[0151] According to this structure, the controller 50 can improve the accuracy of the simulation test result related to the work of the shovel 100 and can improve the work efficiency of the shovel 100 as an automatically operated shovel. Further, the controller 50 can improve the work safety of the shovel 100 as an automatically operated shovel.

[0152] As described above, the system according to the embodiment of the present application is a shovel construction support system SYS that supports the construction of a shovel 100, which has a controller 50 as a computation device that executes a simulation test of the action of a shovel 100 in a hypothetical environment set according to the work environment of the shovel 100. The controller 50 sets a hypothetical work site as an example of a hypothetical environment, for example, based on information related to the work site where the shovel 100 is located, and executes a simulation test of a hypothetical action constituting a hypothetical excavation work of a hypothetical shovel in the hypothetical work site. Specifically, the controller 50 derives how the hypothetical work site changes when a certain hypothetical action is performed.

[0153] According to this structure, the construction support system SYS executes a simulation test of the work of the shovel 100 based on information related to the actual work site, whereby it is possible to reflect the state of the actual work site when supporting the construction of the shovel 100.

[0154] The controller 50 can perform the simulation test in accordance with an output of a space recognition device that recognizes a space around the shovel 100. The space recognition device can be mounted on the shovel 100 or can be provided outside the shovel 100. Also, the controller 50 can perform the simulation test in accordance with an output of one space recognition device or can perform the simulation test in accordance with outputs of a plurality of space recognition devices. The space recognition device can be, for example, the space recognition device Cl mounted on the shovel 100. Alternatively, the space recognition device can be the space recognition device C3 provided outside the shovel 100. Specifically, the space recognition device C3 can be installed on a pole provided at a work site, can be installed on another shovel other than the shovel 100, or can be installed on a flying object that flies above the work site.

[0155] The construction support system SYS can have a display device that displays a result of the simulation test by the controller 50.

[0156] The controller 50 can be configured to be able to recognize, by the simulation test, an event that is likely to occur when the shovel 100 is actually caused to act. According to this configuration, the construction support system SYS recognizes, in advance, an undesirable event that is likely to occur when the shovel 100 is actually caused to act, and thus can prevent the actual occurrence of the undesirable event in advance.

[0157] The shovel 100 can be an automatic shovel. In this case, the controller 50 can generate an operation signal in accordance with information about a pre-set action instruction. Then, the shovel 100 can be configured to act in accordance with the operation signal. Also, the controller 50 can be configured to be able to change the action instruction in accordance with a result of the simulation test.

[0158] According to this configuration, even if the shovel 100 is an automatic shovel, the construction support system SYS can reflect a situation of an actual work site when supporting the construction of the shovel 100.

[0159] The controller 50 can be configured to be able to derive, by the simulation test, a future state of the work site after a prescribed time, which is achieved by the shovel 100 acting in accordance with the action instruction.

[0160] According to this configuration, the construction support system SYS can cause the shovel 100 to actually act and can derive, in real time, a state of the work site at a future time after the prescribed time by the simulation test. Thus, the construction support system SYS can recognize, in advance, a case where the state of the work site at the future time after the prescribed time becomes an undesirable state. In this case, the construction support system SYS can prevent the actual occurrence of the undesirable state in advance by switching the currently used action instruction to another action instruction.

[0161] The controller 50 can be configured to redo the simulation test when a precondition of the simulation test changes. The precondition of the simulation test is, for example, that the degree of sandiness of the sand of the work object is a value within a prescribed range. The degree of sandiness of the sand is derived, for example, from the output of the spatial recognition device. This is because, if the result of the simulation test performed based on the precondition is used even though the precondition has changed, the state of the work site can become an undesirable state.

[0162] Also, as another example, for example, the controller 50 can also be able to consider that an item requiring attention has occurred and call the attention of the manager or the worker, and redo the simulation test considering that the precondition has changed, in a case where the spatial recognition device detects that the position of the iron plate laid on the road surface of the work site has shifted due to frequent passage of the dump truck or an earthquake or the like. Then, the controller 50 can evaluate the influence of the change in the precondition on the construction work. In a case where the position of the iron plate needs to be corrected immediately, the controller 50 can notify the manager or the worker of this by display or the like.

[0163] Also, as another example, the controller 50 can consider that an item requiring attention has occurred and call the attention of the manager or the worker, and redo the simulation test considering that the precondition has changed, when the spatial recognition device C1 detects a change in the work site condition. The change in the work site condition includes, for example, a change in the topography of the work site (the shape of the slope or the shape of the sand temporarily placed, etc.) due to bad weather (rain, etc.), or the occurrence of a fallen tree or a rockslide, etc. Then, the controller 50 can evaluate the influence of the change in the precondition on the construction work. In a case where the topography of the work site needs to be improved immediately or the like, the controller 50 can notify the manager or the worker of this by display or the like. The action simulator 54 can perform a simulation test for repairing the topography of the work site.

[0164] Thus, the controller 50 receives the output of the space recognition device C1 at a predetermined time interval before or during the construction. Then, the controller 50 compares the condition of the work site in the actual work site with the condition of the assumed work site assumed in the virtual space based on the output of the space recognition device C1. Then, the controller 50 determines whether the change in the premise of the simulation test is a matter for attention if the comparison result is determined to be a change in the premise of the simulation test. If the determination is that it is a matter for attention, the controller 50 notifies the manager or the worker of the situation. Then, the controller 50 performs a simulation test for eliminating the matter for attention and derives a work content or a work preparation procedure or the like for eliminating the occurred matter for attention as an improvement scheme. The manager or the worker allows the transmission of the operation command from the controller 50 to the shovel 100 if the improvement scheme derived by the action simulator 54 is determined to be no problem. Then, the operation command is transmitted from the controller 50 to the shovel 100. The controller 30 controls the shovel 100 based on the received operation command.

[0165] According to this configuration, the construction support system SYS re-performs the simulation test if the premise of the simulation test is determined to be changed, and thus it is possible to more appropriately reflect the condition of the actual work site when supporting the construction of the shovel 100.

[0166] The preferred embodiments of the present application have been described in detail above. However, the present application is not limited to the above-described embodiments. The above-described embodiments can be applied to various modifications or substitutions and the like without departing from the scope of the present application. Also, each feature described separately can be combined as long as there is no technical contradiction.

[0167] For example, in the above-described embodiments, the action simulator 54 is implemented as a function of the controller 50 provided in the information center 200, but can be implemented as a function of the controller 30, or as a function of the remote controller 40. Alternatively, the action simulator 54 can be implemented as a function of an arithmetic device that is different from the controller 30, the remote controller 40, and the controller 50.

[0168] This application claims priority based on Japanese Patent Application 2020-092622 filed on May 27, 2020, the entire contents of which are incorporated herein by reference.

[0169] Explanation of Symbols

[0170] 1 - lower traveling body, 1L - left traveling hydraulic motor, 1R - right traveling hydraulic motor, 2 - swing mechanism, 2A - swing hydraulic motor, 3 - upper swing body, 4 - boom, 5 - arm, 6 - bucket, 7 - boom cylinder, 8 - arm cylinder, 9 - bucket cylinder, 10 - cab, 11 - engine, 11a - alternator, 11b - starter device, 11c - water temperature sensor, 13 - regulator, 14 - main pump, 14b - discharge pressure sensor, 14c - oil temperature sensor, 15 - pilot pump, 16 - hydraulic line, 17 - control valve unit, 18 - positioning device, 21 - urea water tank, 21a - urea water remaining amount sensor, 22 - fuel tank, 22a - fuel remaining amount sensor, 25 - pilot line, 26 - operation device, 29 - operation sensor, 30 - controller, 30a - memory, 31 - image generation section, 32 - shovel state determination section, 33 - actuator drive section, 34 - abnormality detection section, 40 - remote controller, 41 - operator state determination section, 42 - image synthesis section, 43 - operation signal generation section, 45 - solenoid valve unit, 50 - controller, 51 - determination section, 52 - operation prediction section, 53 - operation intervention section, 54 - action simulator, 70 - battery, 72 - electrical installation, 74 - engine control unit, 75 - control panel, 100, 100a, 100b - shovel, 200 - information center, A1 - sound collecting device, A2 - sound output device, C1 - space recognition device, C1B - rear camera, C1F - front camera, C1L - left camera, C1R - right camera, C2 - indoor space recognition device, C3 - space recognition device, D1 - display device, DG - design surface information, DS - driver seat, E1 - operator viewpoint, E1' - imaginary operator viewpoint, OP - operator, RC, RCa, RCb - remote operation room, S1 - boom angle sensor, S2 - arm angle sensor, S3 - bucket angle sensor, S4 - swing angular velocity sensor, SYS - construction support system, T1, T2, T3 - communication device.

Claims

1. A construction support system for a shovel, which is a system for supporting construction of a shovel, comprising: an arithmetic device that performs a simulation test of an action of the shovel in a virtual environment set in accordance with an operating environment of the shovel, and is configured to recognize in advance, through the simulation test, an event that occurs when the shovel actually performs an action, the arithmetic device correcting an action command in a case where it is recognized based on a result of the simulation test that an unexpected event will occur.

2. The construction support system for a shovel according to claim 1, wherein: the simulation test of a virtual shovel disposed in a virtual environment set in accordance with an operating environment of the shovel is performed while the shovel in an actual operating site cannot be operated.

3. The construction support system for a shovel according to claim 2, wherein: the simulation test is performed in accordance with a new action command when the action command is corrected.

4. The construction support system for a shovel according to claim 2, wherein: the virtual environment changed by the simulation test of the virtual shovel can be restored to an original virtual environment by operating a prescribed reset button.

5. The construction support system for a shovel according to claim 1, wherein: the arithmetic device performs the simulation test in accordance with an output of a space recognition device that recognizes a space around the shovel, the space recognition device is mounted on the shovel or is disposed outside the shovel.

6. The construction support system for a shovel according to claim 1, comprising: a display device that displays a result of the simulation test of the arithmetic device.

7. The construction support system for a shovel according to claim 1, wherein: the arithmetic device generates an operation signal for controlling a hydraulic actuator of the shovel in accordance with information on an action command that is a command on a series of actions set in advance, the shovel is an automatic operation shovel configured to perform an action in accordance with the operation signal, the arithmetic device changes the action command in a case where it is recognized based on a result of the simulation test that an unexpected event will occur.

8. The construction support system for a shovel according to claim 7, wherein: the arithmetic device is configured to derive, through the simulation test, a future state of an operating site after a prescribed time is elapsed by the shovel performing an action in accordance with the action command.

9. The construction support system for a shovel according to claim 1, wherein: the arithmetic device is configured to perform the simulation test again when a premise of the simulation test is changed.

10. The construction support system for a shovel according to claim 1, wherein: the shovel in the virtual environment is operated by an operation device.

11. The construction support system for a shovel according to claim 10, wherein: the operation device is selectively connected to the shovel in the virtual environment or to a shovel disposed in an actual operating site.

12. The construction support system for a shovel according to claim 1, wherein: the arithmetic device performs a simulation test of an operating content for eliminating the matter to be noted in a case where the event is the matter to be noted. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ 13. The construction support system for a shovel according to claim 5, wherein The operation device compares the condition of the work site in the actual work site with the condition of the hypothetical work site assumed in the hypothetical space using the output of the space recognition device.

Images