Construction support system

Abstract

The system displays height information indicating the ground level while also making the work area easier to see. [Solution] The system includes an imaging device that captures images of the surroundings of a work machine, a display device that displays the images captured by the imaging device, and a control device that displays the images captured by the imaging device on the display device, and superimposes height information indicating the current ground height onto the images when displaying the images captured by the imaging device on the display device, wherein the control device displays or hides the height information according to the difference in height between the current ground and a set reference surface.

Description

Technical Field

[0001] The present disclosure relates to a construction support system.

Background Art

[0002] Patent Document 1 discloses that when remotely operating a work machine, a mesh-like diagram (grid) according to distance information is superimposed and displayed on the ground.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when a grid is superimposed and displayed on the ground or an object placed on the ground, there is a problem that the ground or the object itself becomes difficult to see, making the work difficult.

[0005] Therefore, it is preferable to provide a system that can make the work area easy to see while displaying height information indicating the height of the ground or an object.

Means for Solving the Problems

[0006] The construction support system according to the present disclosure includes an imaging device that captures an image around a work machine, a display device that displays the image captured by the imaging device, and a control device that causes the image captured by the imaging device to be displayed on the display device and superimposes height information indicating the height of a work target on the image when causing the image captured by the imaging device to be displayed on the display device. The control device displays or hides the height information according to the height difference between the work object and the set reference surface. [Effects of the Invention]

[0007] According to this disclosure, it is possible to make the work area easy to see while displaying height information indicating the height of the ground or objects. [Brief explanation of the drawing]

[0008] [Figure 1] This is a side view of the shovel. [Figure 2] Figure 1 is a block diagram showing an example of the drive system configuration of an excavator. [Figure 3] This is a block diagram showing the controller configuration. [Figure 4] This diagram shows one form of implementation of a construction support system. [Figure 5] This is a flowchart explaining the image display process in the SYS construction support system. [Figure 6] This figure shows an example of how images are displayed. [Figure 7] This figure shows other examples of how images are displayed. [Figure 8] This figure shows other examples of how images are displayed. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. Furthermore, the embodiments described below are illustrative and not limiting to the invention, and not all features or combinations thereof described in the embodiments are necessarily essential to the invention. In addition, identical or corresponding components in each drawing are denoted by the same or corresponding reference numerals, and their descriptions may be omitted.

[0010] <Working machinery> First, we will describe the work machinery used in the construction support system of this disclosure.

[0011] Figure 1 shows an excavator 100 as a work machine according to an embodiment of the present invention. An upper revolving structure 3 is mounted on a lower traveling body 1 of the excavator 100 so as to be rotatable via a slewing mechanism 2. A boom 4 is attached to the upper revolving structure 3. An arm 5 is attached to the tip of the boom 4, and a bucket 6 as an end attachment is attached to the tip of the arm 5.

[0012] The boom 4, the arm 5, and the bucket 6 constitute an excavation attachment which is an example of an attachment. The boom 4 is driven by a boom cylinder 7, the arm 5 is driven by an arm cylinder 8, and the bucket 6 is driven by a bucket cylinder 9.

[0013] A boom angle sensor S1 is attached to the boom 4, an arm angle sensor S2 is attached to the arm 5, and a bucket angle sensor S3 is attached to a bucket link. A slewing angular velocity sensor S4 is attached to the upper revolving structure 3.

[0014] The boom angle sensor S1 is one of the attitude detection sensors, is attached to the boom 4, and detects a pitching angle of the boom 4 with respect to the upper revolving structure 3, for example, in a side view, an angle formed by a straight line connecting the fulcrums at both ends of the boom 4 with respect to the revolving plane of the upper revolving structure 3. In the present embodiment, the boom angle sensor S1 is an IMU (Inertial Measurement Unit), but other sensors such as a stroke sensor may be used.

[0015] The arm angle sensor S2 is one of the attitude detection sensors, is attached to the arm 5, and detects a rotation angle of the arm 5 with respect to the boom 4, for example, in a side view, an angle formed by a straight line connecting the fulcrums at both ends of the arm 5 with respect to a straight line connecting the fulcrums at both ends of the boom 4. In the present embodiment, the arm angle sensor S2 is an IMU (Inertial Measurement Unit), but other sensors such as a stroke sensor may be used.

[0016] The bucket angle sensor S3 is one of the attitude detection sensors. It is attached to the bucket 6 and detects the rotation angle of the bucket 6 with respect to the arm 5, for example, in a side view, the angle formed by the straight line connecting the fulcrums at both ends of the arm 5 and the straight line connecting the fulcrum and the tip (cutting edge) of the bucket 6. In this embodiment, the bucket angle sensor S3 is an IMU (Inertial Measurement Unit), but other sensors such as a stroke sensor may also be used.

[0017] Note that each of the boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3 may be a rotary encoder, an acceleration sensor, a potentiometer (variable resistor), an inclination sensor, or an inertial measurement device, etc. The inertial measurement device may be composed of, for example, a combination of an acceleration sensor and a gyro sensor.

[0018] The swing angular velocity sensor S4 is configured to detect the swing angular velocity of the upper swing body 3. In this embodiment, the swing angular velocity sensor S4 is a gyro sensor. The swing angular velocity sensor S4 may be configured to calculate the swing angle based on the swing angular velocity. The swing angular velocity sensor S4 may be composed of other sensors such as a rotary encoder.

[0019] The upper swing body 3 is equipped with a cabin 10 as a driver's cab, an engine 11, a positioning device 18, a sound collection device A1, an imaging device C1, a space recognition device 70, a communication device T1, etc. Also, a controller 30 is installed in the cabin 10. Further, a driver's seat and an operating device, etc. are installed in the cabin 10. However, the excavator 100 may be a driverless excavator with the cabin 10 omitted.

[0020] The lower traveling body 1 is not limited to using a crawler as shown in the figure, and may be a wheeled excavator type with tires.

[0021] Engine 11 is the power source for the shovel 100. In this embodiment, engine 11 is a diesel engine. The output shaft of engine 11 is connected to the input shafts of the main pump 14 (see Figure 2) and the pilot pump 15 (see Figure 2), respectively.

[0022] The positioning device 18 is configured to measure the position of the shovel 100. In this embodiment, the positioning device 18 is a GNSS (Global Navigation Satellite System) compass and is configured to measure the position and orientation of the upper rotating body 3. The positioning device 18 transmits data regarding the position and orientation of the shovel 100 to the controller 30.

[0023] The sound collection device A1 is configured to collect sounds generated around the shovel 100. In this embodiment, the sound collection device A1 is a microphone attached to the upper rotating body 3.

[0024] The imaging device C1 is configured to image the area around the shovel 100. In this embodiment, the imaging device C1 includes a rear camera C1B mounted on the rear end of the upper surface of the upper rotating body 3, a front camera C1F mounted on the front end of the upper surface of the cabin 10, a left camera C1L mounted on the left end of the upper surface of the upper rotating body 3, and a right camera C1R mounted on the right end of the upper surface of the upper rotating body 3. The imaging device C1 may also be a 360-degree camera installed at a predetermined position inside the cabin 10. The predetermined position is, for example, a position corresponding to the eye position of an operator seated in the driver's seat installed inside the cabin 10. The imaging device C1 is a monocular camera and is equipped with a function to adjust the focus and depth of field in the optical system.

[0025] The spatial recognition device 70 is configured to recognize objects in the three-dimensional space surrounding the shovel 100 and to measure (calculate) the positional relationship, such as the distance, from the spatial recognition device 70 or the shovel 100 to the recognized objects. The spatial recognition device 70 may include, for example, an ultrasonic sensor, millimeter-wave radar, a monocular camera, a stereo camera, LIDAR (Light Detecting and Ranging), a distance image sensor, an infrared sensor, etc. The spatial recognition device 70 acquires, for example, ground surface irregularity information within the imaging range of the front camera C1F.

[0026] The communication device T1 is configured to control communication with equipment located outside the shovel 100. In this embodiment, the communication device T1 is configured to control wireless communication between the communication device T1 and equipment located outside the shovel 100 via a wireless communication network.

[0027] Controller 30 is an example of a control device in the present invention and is an arithmetic unit that performs various calculations. In this embodiment, controller 30 is composed of a microcomputer including a CPU and memory. The various functions of controller 30 are realized by the CPU executing a program stored in memory.

[0028] Figure 2 is a block diagram showing an example of the drive system configuration of the excavator 100 shown in Figure 1. In Figure 2, mechanical power transmission lines are shown as double lines, hydraulic fluid lines as thick solid lines, pilot lines as dashed lines, and electrical control lines as dotted lines.

[0029] The drive system of the Shovel 100 consists of an engine 11, a regulator 13, a main pump 14, a pilot pump 15, a control valve unit 17, a controller 30, and a solenoid valve unit 19, etc. The engine 11 is driven and controlled by an engine control unit 74.

[0030] The main pump 14 supplies hydraulic fluid to the control valve unit 17 via the hydraulic fluid line 16. In this embodiment, the main pump 14 is a swashplate type variable displacement hydraulic pump.

[0031] The regulator 13 is configured to control the discharge rate of the main pump 14. In this embodiment, the regulator 13 is configured to adjust the swash plate tilt angle of the main pump 14 according to the discharge pressure of the main pump 14 or a control signal from the controller 30, etc. The discharge rate (displaced volume) per revolution of the main pump 14 is controlled by the regulator 13.

[0032] The pilot pump 15 is configured to supply hydraulic fluid to various hydraulic control devices via the pilot line 25. In this embodiment, the pilot pump 15 is a fixed-displacement hydraulic pump. However, the pilot pump 15 may be omitted. In this case, the function that the pilot pump 15 performed may be realized by the main pump 14. That is, the main pump 14 may have a function to supply hydraulic fluid to the solenoid valve unit 19, etc., via a throttle, in addition to its function of supplying hydraulic fluid to the control valve unit 17.

[0033] The control valve unit 17 is configured to selectively supply hydraulic fluid received from the main pump 14 to one or more hydraulic actuators. In this embodiment, the control valve unit 17 includes a plurality of control valves corresponding to a plurality of hydraulic actuators. The control valve unit 17 is configured to selectively supply hydraulic fluid discharged from the main pump 14 to one or more hydraulic actuators. The hydraulic actuators include, for example, a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, a left-side travel hydraulic motor 1L, a right-side travel hydraulic motor 1R, and a slewing hydraulic motor 2A.

[0034] The controller 30 is configured to control the solenoid valve unit 19 based on operation signals received via the communication device T1. In this embodiment, the operation signals are transmitted from the remote control room. The operation signals may also be generated by an operating device located inside the cabin 10.

[0035] The solenoid valve unit 19 includes a plurality of solenoid valves arranged in each pilot line 25 that connects the pilot pump 15 to the pilot port of each control valve in the control valve unit 17.

[0036] In this embodiment, the controller 30 can control the pilot pressure acting on the pilot port of each control valve by individually controlling the opening area of ​​each of the multiple solenoid valves. Therefore, the controller 30 can control the flow rate of hydraulic fluid flowing into each hydraulic actuator and the flow rate of hydraulic fluid flowing out of each hydraulic actuator, and consequently, the movement of each hydraulic actuator can be controlled.

[0037] In this way, the controller 30 can perform actions such as raising and lowering the boom 4, opening and closing the arm 5, opening and closing the bucket 6, rotating the upper slewing body 3, and driving the lower traveling body 1 in response to operation signals from an external location such as a remote control room.

[0038] Furthermore, the excavator 100 may be configured such that some or all of its driven elements, such as the lower traveling body 1, upper slewing body 3, boom 4, arm 5, and bucket 6, are electrically driven by electric actuators. In other words, the excavator 100 may be a hybrid excavator or an electric excavator.

[0039] <Configuration of Controller 30> Figure 3 is a block diagram showing the configuration of the controller 30.

[0040] As shown in Figure 3, the controller 30 includes an image acquisition unit 31, a surface irregularity information acquisition unit 32, a height information generation unit 33, a reference surface information acquisition unit 34, a height display information generation unit 35, and a display control unit 36.

[0041] The image acquisition unit 31 acquires the image captured by the front camera C1F of the imaging device C1. As a result, the image acquisition unit 31 acquires an image that includes the ground in front of the shovel 100.

[0042] The unevenness information acquisition unit 32 acquires the unevenness information of the ground acquired by the spatial recognition device 70. As a result, the unevenness information acquisition unit 32 acquires the unevenness information of the ground in front of the shovel 100.

[0043] The height information generation unit 33 generates grid information based on the unevenness information of the ground acquired by the unevenness information acquisition unit 32. The grid information is an example of height information in the present invention. The grid information is grid-like information displayed on the display device D1, and is represented by a grid whose shape changes according to the slope of the ground. For example, the front-to-back direction of the shovel 100 (the X-axis direction shown in Figure 1) may be represented by vertical lines extending in the vertical direction, and the left-to-right direction of the shovel 100 (the direction the paper faces in Figure 1) may be represented by horizontal lines extending in the horizontal direction, and these may intersect to form a grid. In areas where the ground height does not change, the vertical and horizontal lines are straight, but when the ground height changes, the vertical and horizontal lines curve or bend according to the change in height. Furthermore, the spacing between the vertical lines may be made wider as the object is displayed closer to the viewer to give a sense of perspective. Therefore, the unevenness of the ground can be recognized by the shape of the grid.

[0044] The reference surface information acquisition unit 34 acquires information indicating, for example, the height and inclination of the design target surface. The position of the shovel 100 is measured by the positioning device 18. Therefore, if the construction information from the shovel 100 is set in the controller 30, the reference surface information acquisition unit 34 can acquire information indicating the height and inclination of the design target surface.

[0045] The height display information generation unit 35 generates grid display information in which the portion of the grid information generated by the height information generation unit 33 that corresponds to the height of the reference plane acquired by the reference plane information acquisition unit 34 is hidden.

[0046] The display control unit 36 ​​overlays the grid display information generated by the height display information generation unit 35 onto the image acquired by the image acquisition unit 31, and displays it superimposed on the display device D1 of the remote control room RC (see Figure 4), which will be described later.

[0047] <Configuration of the construction support system> Next, a construction support system using the shovel 100 configured in this embodiment will be described.

[0048] Figure 4 shows one form of implementation of the construction support system of this disclosure.

[0049] As shown in Figure 4, the construction support system SYS in this embodiment includes an excavator 100 and a remote control room RC for remotely operating the excavator 100.

[0050] In the construction support system SYS configured in this way, the communication device T1 of the shovel 100 is configured to send and receive information with the communication device T2 installed in the remote control room RC using wireless communication. In this embodiment, the communication devices T1 and T2 are configured to send and receive information via a fifth-generation mobile communication line (5G line), LTE line, or satellite line, etc.

[0051] The remote control room RC is equipped with a remote controller 40, a sound output device A2, an indoor imaging device C2, a display device D1, and a communication device T2, among other things. The remote control room RC also has a driver's seat DS where the operator OP sits to remotely control the shovel 100.

[0052] The remote controller 40 is an arithmetic unit that performs various calculations. In this embodiment, the remote controller 40, like the controller 30, is composed of a microcomputer including a CPU and memory. The various functions of the remote controller 40 are realized by the CPU executing a program stored in memory.

[0053] The sound output device A2 is configured to output sound. In this embodiment, the sound output device A2 is a speaker and is configured to reproduce the sound collected by the sound collection device A1 attached to the shovel 100.

[0054] The indoor imaging device C2 is configured to image the inside of the remote control room RC. In this embodiment, the indoor imaging device C2 is a camera installed inside the remote control room RC and is configured to image the operator OP seated in the driver's seat DS.

[0055] The communication device T2 is configured to control wireless communication with the communication device T1, which is attached to the shovel 100.

[0056] In this embodiment, the driver's seat DS has a structure similar to that of a driver's seat installed in the cabin of a normal excavator. Specifically, a left console box is located to the left of the driver's seat DS, and a right console box is located to the right of the driver's seat DS. A left operating lever is located at the front of the upper surface of the left console box, and a right operating lever is located at the front of the upper surface of the right console box. In addition, a drive lever and a drive pedal are located in front of the driver's seat DS. Furthermore, an engine speed adjustment dial 75 is located in the center of the upper surface of the right console box. The left operating lever, right operating lever, drive lever, drive pedal and engine speed adjustment dial 75 each constitute an operating device 26.

[0057] The engine speed adjustment dial 75 is a dial for adjusting the rotational speed of the engine 11, and is configured to allow switching between, for example, four engine speed settings.

[0058] Specifically, the engine speed adjustment dial 75 is configured to allow switching between four engine speed settings: SP mode, H mode, A mode, and idling mode. The engine speed adjustment dial 75 transmits data related to the engine speed setting to the controller 30.

[0059] SP mode is the rotation speed mode selected when the operator (OP) wants to prioritize work volume, and it uses the highest engine speed. H mode is the rotation speed mode selected when the operator (OP) wants to balance work volume and fuel efficiency, and it uses the second highest engine speed. A mode is the rotation speed mode selected when the operator (OP) wants to operate the shovel with low noise while prioritizing fuel efficiency, and it uses the third highest engine speed. Idling mode is the rotation speed mode selected when the operator (OP) wants the engine to idle, and it uses the lowest engine speed. The engine 11 is then controlled to a constant rotation speed at the engine speed of the rotation speed mode selected via the engine speed adjustment dial 75.

[0060] The operating device 26 is equipped with an operation sensor 29 for detecting the operation of the operating device 26. The operation sensor 29 is, for example, a tilt sensor that detects the tilt angle of the operating lever, or an angle sensor that detects the oscillation angle of the operating lever around its pivot axis. The operation sensor 29 may also consist of other sensors such as a pressure sensor, a current sensor, a voltage sensor, or a distance sensor. The operation sensor 29 outputs information regarding the detected operation of the operating device 26 to the remote controller 40. The remote controller 40 generates an operation signal based on the received information and transmits the generated operation signal to the shovel 100. The operation sensor 29 may be configured to generate an operation signal. In this case, the operation sensor 29 may output the operation signal to the communication device T2 without going through the remote controller 40.

[0061] Upon receiving an operation signal, the controller 30 will perform actions such as raising and lowering the boom 4, opening and closing the arm 5, opening and closing the bucket 6, rotating the upper slewing body 3, and moving the lower traveling body 1, in accordance with the received operation signal.

[0062] The display device D1 is configured to display information about the surrounding conditions of the shovel 100. In this embodiment, the display device D1 is a multi-display consisting of nine monitors arranged in three vertical rows and three horizontal columns, and is configured to display the conditions of the space in front of, to the left of, and to the right of the shovel 100. Each monitor is a liquid crystal monitor or an organic EL monitor, etc. However, the display device D1 may consist of one or more curved monitors, or it may consist of a projector.

[0063] The display device D1 is configured to display an image that allows the operator OP in the remote control room RC to visually inspect the area around the shovel 100. In other words, the display device D1 displays an image that allows the operator to check the situation around the shovel 100 as if they were inside the cabin 10 of the shovel 100, even though they are in the remote control room RC.

[0064] Furthermore, an input device 27 may be provided in the remote control room RC. The input device 27 allows the operator OP, seated in the driver's seat DS, to input information, and the input information may be transmitted to the controller 30 of the shovel 100 via the communication device T2. The operator OP can also set a reference plane via the input device 27 to hide a portion of the grid information generated by the height information generation unit 33. For example, by displaying the design target plane on the display device D1 and adjusting its height up or down or changing its inclination through input to the input device 27, a reference plane different from the design target plane can be set. As a result, the reference plane information acquisition unit 34 acquires information indicating the height and inclination of the reference plane set via the input device 27. The height display information generation unit 35 then generates grid display information in which the portion of the grid information generated by the height information generation unit 33 corresponding to the height of the reference plane acquired by the reference plane information acquisition unit 34 is hidden.

[0065] The display device D1 may be a display device that can be worn by the operator OP. For example, the display device D1 may be a head-mounted display (HMD) and may be configured to send and receive information with the remote controller 40 via wireless communication. The head-mounted display may be wired to the remote controller 40. The head-mounted display may be a transparent head-mounted display or an opaque head-mounted display. The head-mounted display may be a monocular head-mounted display or a binocular head-mounted display. In this way, if the display device D1 is a display device that can be worn by the operator OP, the display device D1 may be equipped with an eye-tracking device 28.

[0066] Furthermore, in this embodiment, for the sake of clarity, a configuration having one shovel 100 and one remote control room RC was used as an example. However, multiple shovels 100 and remote control rooms RC may be provided. In that case, each of the multiple shovels 100 may be associated with a remote control room RC, or multiple shovels 100 may be operated from a single remote control room RC.

[0067] <Image display processing> The image display process in the SYS construction support system, configured as described above, is explained below.

[0068] Figure 5 is a flowchart illustrating the image display process in the SYS construction support system.

[0069] In the case of the shovel 100, the area around the shovel 100 is imaged by the imaging device C1. At that time, the front camera C1F of the imaging device C1 is mounted on the front end of the upper surface of the cabin 10, so that an image of the area in front of the shovel 100 is captured along with the ground in front of the shovel 100.

[0070] The image captured by the front camera C1F is sent to the controller 30 and acquired by the image acquisition unit 31 (step ST11). As a result, the image acquisition unit 31 acquires an image including the ground in front of the shovel 100 that was captured by the front camera C1F.

[0071] Furthermore, the shovel 100 can acquire ground surface topography information using the spatial recognition device 70. The spatial recognition device 70 can acquire ground surface topography information by, for example, irradiating the ground in front of the shovel 100 with laser light and receiving the reflected light. Specifically, when LIDAR is used as the spatial recognition device 70, laser light is irradiated within the imaging range of the front camera C1F, and ground surface topography information within the imaging range of the front camera C1F is acquired using point cloud data of the distance determined from the reflected light. The ground surface topography information acquired by the spatial recognition device 70 is sent to the controller 30 and acquired by the surface topography information acquisition unit 32 (step ST12). As a result, the surface topography information acquisition unit 32 acquires ground surface topography information contained in the image of the ground in front of the shovel 100, that is, the image captured by the front camera C1F and acquired by the image acquisition unit 31. Here, when a LIDAR is used as the spatial recognition device 70, the laser light emitted from the LIDAR may be reflected off the attachments of the boom 4, arm 5, and bucket 6 when acquiring ground surface irregularities information with the LIDAR. In that case, it becomes impossible to accurately acquire ground surface irregularities information using the reflected light of the laser light emitted from the LIDAR. Therefore, the surface irregularities information acquisition unit 32 may acquire information regarding the posture of the boom 4 detected by the boom angle sensor S1, the posture of the arm 5 detected by the arm angle sensor S2, and the posture of the bucket 6 detected by the bucket angle sensor S3. In this case, the surface irregularities information acquisition unit 32 can recognize the postures of the boom 4, arm 5, and bucket 6. As a result, the surface irregularities information acquisition unit 32 can exclude the reflected light reflected off the attachments of the boom 4, arm 5, and bucket 6 from the reflected light of the laser light emitted from the LIDAR, and thus be able to accurately acquire ground surface irregularities information.

[0072] When the unevenness information of the ground acquired by the spatial recognition device 70 is acquired by the unevenness information acquisition unit 32, the height information generation unit 33 generates grid information based on the unevenness information of the ground acquired by the unevenness information acquisition unit 32 (step ST13). The grid information is a grid-like information consisting of vertical lines and horizontal lines, and the shape of the grid changes according to the slope of the ground. Specifically, in flat areas of the ground, the horizontal lines become straight lines parallel to each other, and the vertical lines become straight lines with wider spacing the closer they are displayed to create a sense of perspective, as described above. In areas where the ground has a slope relative to the flat area, if the slope is constant, at least one of the horizontal and vertical lines becomes a straight line with a constant angle relative to the flat area. By the change in the angle of at least one of the horizontal and vertical lines, it is possible to recognize that there is a slope relative to the flat area of ​​the ground, and the degree of the slope can be recognized by the magnitude of the angle. Furthermore, if the slope changes, the horizontal or vertical lines become curves according to the change in the slope. And by the shape of these curves, it is possible to recognize how the slope is changing.

[0073] Thus, height information may also be represented as grid information, where the shape of the grid changes according to the slope of the ground. Since grid information is commonly used to display unevenness information, it makes it easier to understand the unevenness of the ground.

[0074] In this embodiment, a grid-like information consisting of vertical and horizontal lines is used as height information. However, the height information may also be represented by concentric circles, or by a sector with lines extending radially from the pivot point of the shovel 100. Furthermore, color may be used to indicate height. Thus, by using a graphic or color to indicate height, it becomes easier to grasp the unevenness of the ground compared to when the height information is represented by text.

[0075] Furthermore, the reference surface information acquisition unit 34 acquires information indicating, for example, the height and inclination of the design target surface as reference surface information (step ST14). As described above, since the position of the shovel 100 is measured by the positioning device 18, the reference surface information acquisition unit 34 can acquire information indicating the height and inclination of the design target surface using construction information in which the height of the design target surface has been set.

[0076] When grid information is generated in the height information generation unit 33 and reference surface information indicating the height of the reference surface is acquired in the reference surface information acquisition unit 34, the height display information generation unit 35 processes the grid information generated in the height information generation unit 33 and generates grid display information in which the portion of the grid information generated in the height information generation unit 33 corresponding to the height of the reference surface acquired in the reference surface information acquisition unit 34 is hidden (step ST15). For example, grid display information is generated in which the grid information of the height information generation unit 33 that is higher than the reference surface acquired in the reference surface information acquisition unit 34 is hidden. In this case, the grid information generated in the height information generation unit 33 is based on the unevenness information of the ground acquired in the unevenness information acquisition unit 32, and the unevenness information of the ground is acquired using point cloud data of the distance between the LIDAR and the ground acquired by the LIDAR when a LIDAR is used as the spatial recognition device 70. Therefore, by using the installation position (height) of the LIDAR and the point cloud data, the height of each region of the grid information generated in the height information generation unit 33 can be determined.

[0077] Subsequently, in the display control unit 36, the grid display information generated by the height display information generation unit 35 is superimposed on the image acquired by the image acquisition unit 31 (step ST16). Then, the image information, which is the image acquired by the image acquisition unit 31 with the grid display information generated by the height display information generation unit 35 superimposed on it, is transmitted to the remote control room RC via the communication device T1.

[0078] Image information transmitted via communication device T1 is received via communication device T2 in the remote control room RC and displayed on the display device D1 of the remote control room RC under the control of the display control unit 36 ​​(step ST17). As a result, on the display device D1 of the remote control room RC, grid display information generated by the height display information generation unit 35 is superimposed on the image acquired by the image acquisition unit 31.

[0079] <Image display format> The following describes the display format of images produced by the display processing described above.

[0080] <Display format 1> Figure 6 shows an example of how images are displayed.

[0081] As shown in Figure 6, in this example, the display device D1 of the remote control room RC displays an image including a flat area 201, a convex area 202, and a concave area 203. This image was captured by the front camera C1F and acquired by the image acquisition unit 31. In other words, it is an image including the ground in front of the shovel 100.

[0082] The flat portion 201 is a plane with the same height as the design target surface in the construction information. The convex portion 202 is a region that protrudes from the flat portion 201 and is therefore higher in height than the flat portion 201. The concave portion 203 is a region that is recessed from the flat portion 201 and is therefore lower in height than the flat portion 201.

[0083] In this way, grid display information generated by the height display information generation unit 35 is superimposed on the image which includes the flat portion 201, the convex portion 202, and the concave portion 203.

[0084] Here, the grid display information generated by the height display information generation unit 35 does not have to be superimposed on all areas of the image captured by the front camera C1F and acquired by the image acquisition unit 31. As shown in Figure 6, the grid display information generated by the height display information generation unit 35 may be superimposed only on the set grid display area 300 of the image which includes the flat area 201, the convex area 202, and the concave area 203. The grid display area 300 may be set, for example, by specifying it via the input device 27. In this case, the grid display area 300 may be specified with the grid display information generated by the height display information generation unit 35 superimposed on the image acquired by the image acquisition unit 31, or the grid display area 300 may be set in advance. If the grid display area 300 is set in advance, for example, the range for acquiring the unevenness information of the ground acquired by the spatial recognition device 70 may be set in the unevenness information acquisition unit 32.

[0085] The grid display information generated by the height display information generation unit 35 is superimposed as grid lines 301 on the flat area 201, but not on the convex area 202.

[0086] Thus, in this example, grid lines 301, which represent grid information, are displayed in areas where the current ground level is the same as the reference plane, and grid lines, which represent grid information, are hidden in areas where the current ground level is higher than the reference plane. As a result, grid information is not displayed in the area of ​​the ground in front of the shovel 100, as captured by the front camera C1F, where excavation by the bucket 6 is required, making that area easier to see.

[0087] Furthermore, the grid display information generated by the height display information generation unit 35 is superimposed on the recess 203, but it is superimposed as grid lines 302 with a different display format than grid lines 301. For example, grid lines 301 may be displayed as solid lines while grid lines 302 are displayed as dashed lines. Also, the display colors of grid lines 301 and grid lines 302 may be different. As a result, in the recess 203, along with the shape of the grid grid formed by grid lines 302, it is possible to recognize, for example, that the recess is lower than the flat area 201 due to excessive excavation and needs to be filled with soil, based on the difference in display format.

[0088] Thus, in the area displaying grid information, which represents height, the display format of the grid lines may be varied depending on the current ground height. This makes the height easier to understand even in the area displaying grid information.

[0089] As described above, this embodiment includes an imaging device C1 that captures images of the area around the shovel 100, a display device D1 that displays the images captured by the imaging device C1, and a controller 30 that displays the images captured by the imaging device C1 on the display device D1 and superimposes height information indicating the height of the work object onto the image when the images captured by the imaging device C1 are displayed on the display device C1. The controller 30 displays or hides the height information according to the height difference between the work object and a set reference surface. This makes it possible to make the work area easy to see while displaying height information indicating the height of the ground. At that time, the display of height information indicating the height of the ground provides a sense of depth and three-dimensionality, maintaining a state that makes it easy to work. In this case, the height of the work object may be the current ground height as described above.

[0090] Furthermore, as described above, the controller 30 may display or hide height information for a set area of ​​the image captured by the imaging device C1. This makes it possible to display or hide height information for the area of ​​the image captured by the imaging device C1 that is necessary for the work of the shovel 100.

[0091] <Display format 2> Figure 7 shows other examples of image display formats.

[0092] In this example, the format of the grid display information in the grid display area 300 is different from the example shown in Figure 6.

[0093] As shown in Figure 7, in this example, the grid display information generated by the height display information generation unit 35 is superimposed as grid lines 301 on the upper surface of the protrusion 202, except for the areas of protrusion 204a and 204b that further protrude from the upper surface of the protrusion 202, but is not superimposed on the protrusion 204a and 204b.

[0094] Furthermore, although the grid display information generated by the height display information generation unit 35 is superimposed on the sides of the flat portion 201 and the convex portion 202, it is superimposed as grid line 302, which has a different display format than grid line 301. Also, although the grid display information generated by the height display information generation unit 35 is superimposed on the concave portion 203, it is superimposed as grid line 303, which has a different display format than grid lines 301 and 302. For example, grid line 301 may be displayed as a solid line, grid line 302 as a dashed line, and grid line 303 as a dotted line. In addition, the display colors of grid lines 301 to 303 may be different from each other.

[0095] In this way, the height of the reference plane used as a basis for displaying or hiding grid lines that represent height information can be changed from the design target plane. For example, as described above, the design target plane can be displayed on the display device D1, and its height can be raised or lowered, or its inclination changed, by inputting to the input device 27, thereby setting a reference plane different from the design target plane. In this case, the reference plane information acquisition unit 34 acquires information indicating the height and inclination of the set reference plane via the input device 27. The height display information generation unit 35 then generates grid display information in which the portion of the grid information generated by the height information generation unit 33 corresponding to the height of the reference plane acquired by the reference plane information acquisition unit 34 is hidden. In this way, the operator OP can set a reference plane via the input device 27 to hide a portion of the grid information generated by the height information generation unit 33.

[0096] Alternatively, the bottom surface of the bucket 6 or a plane parallel to the reference surface and passing through the tip of the bucket 6 may be set as the reference surface. In that case, for example, first, the bucket 6 is moved to the area to be used as the reference surface, the bottom surface of the bucket 6 or the tip of the bucket 6 is brought into contact with the area to be used as the reference surface, and in that state, the intention to set the reference surface is input, for example, via the input device 27. Here, as described above, the position of the shovel 100 is measured by the positioning device 18. In addition, the posture of the boom 4 is detected by the boom angle sensor S1, the posture of the arm 5 is detected by the arm angle sensor S2, and the posture of the bucket 6 is detected by the bucket angle sensor S3. Therefore, the reference surface information acquisition unit 34 can acquire information indicating the height and inclination of the reference surface set with the bottom surface of the bucket 6 or the tip of the bucket 6 in contact with the ground. Alternatively, the surface on which the shovel 100 is currently located may be set as the reference surface.

[0097] In this way, because the height or inclination of the reference plane can be changed, the operator OP can arbitrarily set the height at which the grid lines are hidden in the grid display area 300 where grid information is displayed, according to the task.

[0098] <Display format 3> Figure 8 shows another example of how images are displayed.

[0099] In this example as well, the format of the grid display information in the grid display area 300 is different from the example shown in Figure 6.

[0100] As shown in Figure 8, in this example, the grid display information generated by the height display information generation unit 35 is superimposed as grid lines 301 on the convex portion 202, but not on the flat portion 201 which is the same height as the reference surface. Furthermore, the grid display information generated by the height display information generation unit 35 is superimposed on the concave portion 203, but it is superimposed as grid lines 302, which have a different display format than grid lines 301. For example, grid lines 301 may be displayed as solid lines while grid lines 302 are displayed as dashed lines. Also, the display colors of grid lines 301 and grid lines 302 may be different.

[0101] In this example, the grid lines are hidden only in the flat area 201 of the grid display area 300 that is at the same height as the target design surface. In other words, in areas where the current ground height is the same as the reference surface, the grid lines that represent height information are hidden. This configuration allows for clear distinction and recognition of the irregularities in the area where work is required from the target design surface.

[0102] Furthermore, the display format of the grid lines may be varied depending on the height difference from the reference plane. For example, grid lines in areas where the absolute height difference from the reference plane is less than 100 mm may be displayed in blue, and grid lines in areas where the absolute height difference from the reference plane is 100 mm or more may be displayed in red. Alternatively, grid lines in areas where the absolute height difference from the reference plane is less than 100 mm may be displayed as solid lines, and grid lines in areas where the absolute height difference from the reference plane is 100 mm or more may be displayed as dashed lines.

[0103] Furthermore, in the embodiments described above, ground surface topography information is acquired using a spatial recognition device 70 such as LIDAR. However, the acquisition of ground surface topography information is not limited to using a spatial recognition device 70. For example, ground surface topography information may be acquired by the trajectory of the bucket 6 during excavation. In that case, the ground surface topography information acquired by the trajectory of the bucket 6 can be shared among multiple shovels 100 or used as work history in subsequent excavation work.

[0104] Furthermore, in the embodiment described above, height information is displayed or hidden depending on the difference between the current ground level and the height of the reference surface. However, height information may also be displayed or hidden depending on the difference between the height of an object placed on the reference surface and the height of the reference surface. Examples of objects placed on the reference surface include wood, recyclable metal (metal scrap), cones, signs, etc. Objects placed on the reference surface also include those that are partially embedded in the reference surface, such as fences.

[0105] Furthermore, in the embodiments described above, an example was shown in which the controller 30 of the shovel 100 is an example of the control device of the present invention. However, a remote controller 40 may also function as an example of the control device of the present invention, or the functions of the controller 30 shown in Figure 3 may be realized on the cloud. [Explanation of symbols]

[0106] 1. Lower running body 1L hydraulic motor for left-side travel 1R Right-side hydraulic motor 2. Swivel mechanism 2A Swivel Hydraulic Motor 3. Upper rotating body 4 Boom 5 Arms 6 buckets 7 Boom Cylinder 8 Arm Cylinder 9 Bucket Cylinder 10 cabins 11 Engine 13 Regulator 14 Main pump 15 Pilot pump 16. Hydraulic fluid line 17 Control Valve Unit 18 Positioning device 19 Solenoid valve unit 25 Pilot Line 26 Operating device 27 Input devices 29 Operation Sensors 30 controllers 31 Image acquisition unit 32 Concave / convex information acquisition section 33 Height Information Generation Unit 34 Reference surface information acquisition unit 35 Height display information generation unit 36 Display Control Unit 40 Remote Controllers 70 Spatial recognition device 74 Engine control unit 75 Engine speed adjustment dial 100 Shovel A1 Sound collection device A2 Sound output device C1 Imaging device C1B Rear Camera C1F Front Camera C1L Left Camera C1R Right Camera C2 Indoor Imaging Device D1 display device DS driver's seat OP Operator RC Remote Control Room S1 Boom Angle Sensor S2 Arm Angle Sensor S3 Bucket Angle Sensor S4 Swivel Angular Velocity Sensor SYS Construction Support System T1, T2 communication devices 201 Flat area 202, 204a, 204b protrusions 203 Recess 300 grid display area 301,302 grid lines

Claims

1. An imaging device that captures images of the surroundings of a work machine, A display device that displays an image captured by the aforementioned imaging device, The system includes a control device that displays an image captured by the imaging device on the display device, and, when displaying an image captured by the imaging device on the display device, superimposes height information indicating the height of the work object onto the image. The control device is a construction support system that displays or hides the height information according to the height difference between the work object and a set reference surface.

2. The construction support system according to claim 1, wherein the height of the work target is the current ground level.

3. The construction support system according to claim 1, wherein the height of the work object is the height of the object placed on the reference surface.

4. The construction support system according to claim 1, wherein the height information is information indicating height by a shape or color.

5. The construction support system according to claim 4, wherein the height information is grid information represented by a grid whose grid shape changes according to the slope of the ground.

6. The construction support system according to claim 1, wherein the control device hides the height information in areas where the height of the work object is higher than the reference surface.

7. The construction support system according to claim 1, wherein the control device hides the height information in areas where the height of the work object is the same as the reference surface.

8. The construction support system according to claim 1, wherein the control device, in the area for displaying the height information, displays the height information in a different manner depending on the height of the work object.

9. The construction support system according to claim 1, wherein the height or inclination of the reference surface is adjustable.

10. The construction support system according to claim 1, wherein the control device displays or hides the height information for a set area of ​​the image captured by the imaging device.

Images