Display control system, display control method, and remote operation system

Abstract

To provide a display control system, a display control method, and a remote operation system, which can supplement information in a height direction.SOLUTION: A display control system of a display device comprises: an acquisition part that acquires measurement information measured with a work machine; and a display part that displays on the display device a composite image including a side-face image of the work machine and a reference image showing a reference in a height direction. The display part draws the side-face image at a position corresponding to an actual height with the reference image as the reference, using a predetermined height in the periphery of the work machine which is different in height from a ground plane of the work machine, based on the measurement information.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present disclosure relates to a display control system, a display control method, and a remote operation system.

Background Art

[0002] Patent Document 1 describes a display control system that controls a display device of a remote operation cab for remotely operating a work vehicle (hereinafter referred to as a work machine). In the display control system described in Patent Document 1, a supplementary image, which is an image for supplementing the lack of the feeling obtained when riding on the work machine, is displayed on the display device. Here, the supplementary image is, for example, a posture image including an image depicting a side view of the work machine representing the current posture of the work machine.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When a work machine performs a loading operation on a transport vehicle using a remote operation system, it may be difficult for an operator to recognize the height of the transport vehicle on the camera image. Also, in a manned work machine, the same problem exists for an operator who is not used to the operation. In contrast, the device described in Patent Document 1 has a problem that information in the height direction may be insufficient in some cases.

[0005] An object of the present disclosure is to provide a display control system, a display control method, and a remote operation system that can supplement information in the height direction in order to solve the above problems.

Means for Solving the Problems

[0006] The display control system of the present disclosure is a display control system for a display device, and includes an acquisition unit that acquires measurement information measured by a work machine, and a display unit that displays on the display device a composite image including a side image of the work machine and a reference image representing a reference in the height direction. The display unit uses a predetermined height around the work machine at a height different from the ground contact surface of the work machine as a reference for the reference image, and based on the measurement information, draws the side image at a position corresponding to the actual height with reference to the reference image.

[0007] The display control method of the present disclosure is a display control method for a display device, and includes a step of acquiring measurement information measured by a work machine, a step of displaying on the display device a composite image including a side image of the work machine and a reference image representing a reference in the height direction, and a step of using a predetermined height around the work machine at a height different from the ground contact surface of the work machine as a reference for the reference image, and based on the measurement information, drawing the side image at a position corresponding to the actual height with reference to the reference image.

[0008] The remote operation system of the present disclosure includes an acquisition unit that acquires measurement information measured by a work machine provided with an imaging device, and a display unit that superimposes on an imaging image captured by the imaging device a composite image including a side image of the work machine and a reference image representing a reference in the height direction and displays the composite image on a display device provided at a remote location of the work machine. The display unit uses a predetermined height around the work machine at a height different from the ground contact surface of the work machine as a reference for the reference image, and based on the measurement information, draws the side image at a position corresponding to the actual height with reference to the reference image, and includes a display control system, the display device, and an operation device of the work machine provided at the remote location.

Advantages of the Invention

[0009] According to the display control system, display control method, and remote operation system of the present disclosure, information in the height direction can be supplemented by the composite image.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In each figure, the same or corresponding configurations are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

[0012] (Remote operation system) FIG. 1 is a schematic diagram showing a configuration example of a remote operation system according to an embodiment of the present disclosure. FIG. 4 is a schematic block diagram showing a configuration example of the remote operation system 1 according to an embodiment of the present disclosure. The remote operation system 1 is a remote operation system for remotely operating a work machine 100 including a work implement, and includes a work machine 100 that operates by remote operation, a transport vehicle 200 such as a dump truck that is a loading target for loading excavated materials excavated by a bucket 133 as a work implement, and a remote operation cab 500 for performing remote operation. The work machine 100 operates at a work site (for example, a mine, a quarry, etc.). The remote operation cab 500 is provided at a location away from the work machine 100 (a remote location of the work machine 100; for example, in a city, within the work site, etc.). The work machine 100 and the remote operation cab 500 are connected via communication means such as a mobile communication network and the Internet. The remote operation system 1 is a system for operating the work machine 100 using the remote operation cab 500. The transport vehicle 200 and the remote operation cab 500 are connected via communication means such as a mobile communication network and the Internet, or further connected via an unmanned dump truck operation control device or the like described later.

[0013] The work machine 100 operates according to an operation signal received from the remote operation cab 500. That is, no operator boards the work machine 100. The remote operation cab 500 receives the operation of the work machine 100 by the operation of the operator and transmits an operation signal to the work machine 100. The operation signal will be described later.

[0014] (Remote operation cab) The remote operation cab 500 includes a driver's seat 510, a first display device 520, a second display device 530, an operation device 540, a control device 550, and a communication device 560. The first display device 520 is disposed in front of the driver's seat 510. The first display device 520 is positioned in front of the operator's eyes when the operator is sitting on the driver's seat 510. As shown in FIG. 1, the first display device 520 is composed of an arranged central display 521, a left display 522, a right display 523, an upper display 524, and a lower display 525. The left display 522 is provided on the left side of the central display 521. The right display 523 is provided on the right side of the central display 521. The upper display 524 is provided on the upper side of the central display 521. The lower display 525 is provided on the lower side of the central display 521. The first display device 520 divides and displays a part or all of the captured image captured by the work machine 100 on the five displays.

[0015] In addition, in other embodiments, the number of displays constituting the first display device 520 is not limited to this. For example, the first display device 520 may be composed of one display. Further, the first display device 520 may project the captured image onto a curved surface or a spherical surface by a projector or the like. The first display device 520 is also a configuration example of the "display device" according to the present disclosure.

[0016] The second display device 530 is disposed diagonally in front of the driver's seat 510. Vehicle body information (such as remaining fuel amount and engine water temperature) transmitted from the work machine 100, notifications of abnormalities of the work machine 100, etc. are displayed on the second display device 530. In addition, in other embodiments, the position of the second display device 530 does not have to be diagonally in front of the driver's seat 510 as long as it is a position visible to the operator. The remote operation cab 500 according to other embodiments may not include the second display device 530. Preferably, the vehicle body information, notifications of abnormalities, etc. displayed on the second display device 530 may be displayed on the first display device 520.

[0017] The operating device 540 is arranged near the driver's seat 510. The operating device 540 is located within the operable range of the operator when the operator is sitting on the driver's seat 510. The operating device 540 includes, for example, an electric lever and an electric pedal. When the operator operates the electric lever and the electric pedal, the operating device 540 outputs a work implement operation signal, which is an operation signal for the boom 131, the arm 132, and the bucket 133, a slewing operation signal for the slewing body 120, and a traveling operation signal for the traveling body 110. In this embodiment, the work implement operation signal, the slewing operation signal, and the traveling operation signal are also collectively referred to as operation signals.

[0018] The control device 550 causes the first display device 520 to display the captured image and the vehicle body information received from the work machine 100. That is, the control device 550 is an example of a display control system. The control device 550 also transmits the operation signal input to the operating device 540 to the work machine 100. The display control system may be the remote control cab 500 including the control device 550, or may further include the work machine control device 126 of the work machine 100 and the front camera 122. The display control system includes a positioning device 123 that measures the height of the work machine 100 as measurement information. The acquisition unit 552 acquires the measured measurement information. The display unit 553 generates a composite image P2 including a side image P21 of the work machine 100 and a reference image P25 representing a reference in the height direction. The side image P21 is generated based on the reference image P25 having a predetermined height different from that of the work machine 100 and the acquired measurement information. The display device displays the composite image including the side image P21 and the reference image P25.

[0019] The communication device 560 transmits and receives a video signal representing the captured image, an operation signal, a signal representing vehicle body information, etc. between the communication device 143 of the work machine 100.

[0020] (Work machine) FIG. 2 is an external view of a work machine 100 according to an embodiment of the present disclosure. The work machine 100 according to the embodiment of the present disclosure is a backhoe loader, which is a type of hydraulic excavator. Note that the work machine 100 according to other embodiments may be other hydraulic excavators such as a face shovel or a rope shovel other than the backhoe loader, or work machines such as a wheel loader or a bulldozer other than the hydraulic excavator. The work machine 100 includes a working device 130 driven by hydraulic pressure, a revolving body 120 that supports the working device 130, and a traveling body 110 that supports the revolving body 120. The revolving body 120 revolves about a revolving axis CR as a rotation center axis.

[0021] The working device 130 includes a boom 131, an arm 132, and a bucket 133 as a working tool. The working device 130 is driven by the telescoping of a boom cylinder 134, an arm cylinder 135, and a bucket cylinder 136. A boom angle sensor 137, an arm angle sensor 138, and a bucket angle sensor 139 are respectively attached to the boom 131, the arm 132, and the bucket 133.

[0022] The base end portion of the boom 131 is attached to the revolving body 120 via a pin. The arm 132 connects the boom 131 and the bucket 133. The base end portion of the arm 132 is attached to the tip end portion of the boom 131 via a pin. The bucket 133 includes a cutting edge 133T for excavating earth and sand and a container for accommodating the excavated earth and sand. The base end portion of the bucket 133 is attached to the tip end portion of the arm 132 via a pin.

[0023] The boom cylinder 134 is a hydraulic cylinder for driving the boom 131. The base end portion of the boom cylinder 134 is attached to the slewing body 120. The tip end portion of the boom cylinder 134 is attached to the boom 131. The arm cylinder 135 is a hydraulic cylinder for driving the arm 132. The base end portion of the arm cylinder 135 is attached to the boom 131. The tip end portion of the arm cylinder 135 is attached to the arm 132. The bucket cylinder 136 is a hydraulic cylinder for driving the bucket 133. The base end portion of the bucket cylinder 136 is attached to the arm 132. The tip end portion of the bucket cylinder 136 is attached to the bucket 133.

[0024] The boom angle sensor 137 is attached to the boom 131, for example, and detects the tilt angle of the boom 131. The arm angle sensor 138 is attached to the arm 132, for example, and detects the tilt angle of the arm 132. The bucket angle sensor 139 is attached to the bucket 133, for example, and detects the tilt angle of the bucket 133. The boom angle sensor 137, the arm angle sensor 138, and the bucket angle sensor 139 according to the embodiment of the present disclosure detect the tilt angle with respect to the ground plane. Note that the angle sensors according to other embodiments are not limited to this, and may detect the tilt angle with respect to other reference planes. For example, in other embodiments, the angle sensor may detect the relative rotation angle by a potentiometer provided at the base end portions of the boom 131, the arm 132, and the bucket 133, or may measure the cylinder lengths of the boom cylinder 134, the arm cylinder 135, and the bucket cylinder 136, and detect the tilt angle by converting the cylinder lengths into angles.

[0025] The revolving body 120 is provided with a driver's cab 121. The driver's cab 121 is provided on the left side of the working machine 130. The driver's cab 121 is provided with a front camera 122. The front camera 122 is installed at the front and upper part inside the driver's cab 121. The front camera 122 images the front of the driver's cab 121 through the front glass of the front part of the driver's cab 121. Here, "front" refers to the direction in which the working machine 130 is mounted on the revolving body 120, and "rear" refers to the opposite direction of "front". "Side" refers to the direction (left and right direction) intersecting the front-rear direction. Examples of the front camera 122 include imaging devices using, for example, a CCD (Charge Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor. Note that the working machine 100 according to another embodiment may not be provided with the driver's cab 121. Even in this case, the front camera 122 is installed at a position corresponding to the driver's cab 121 so as to image the front. Also, in another embodiment, the front camera 122 may be constituted by two or more cameras. Note that the front camera 122 is an example of the imaging device provided in the working machine 100.

[0026] FIG. 3 is an example of an imaging image captured by an imaging device which is the front camera 122 of the working machine 100 according to an embodiment of the present disclosure. The front camera 122 images a range in which a work target in front of the working machine 130 and the driver's cab 121 is reflected. That is, in the imaging image P1 captured by the front camera 122, as shown in FIG. 3, a work target in front of the working machine 130 and the driver's cab 121 is reflected. Also, since the driver's cab 121 is provided on the left side of the working machine 130, a part of the boom 131 is reflected in the right side portion of the imaging image P1.

[0027] As shown in FIG. 2 or FIG. 4, the working machine 100 includes a working machine control device 126, a positioning device 123, a front camera 122, an inclination measuring device 124, a hydraulic device 125, a working machine posture acquisition device 142, and a communication device 143. Note that the working machine posture acquisition device 142 includes a boom angle sensor 137, an arm angle sensor 138, and a bucket angle sensor 139.

[0028] The positioning device 123 acquires the position of the slewing body 120 and the azimuth in which the slewing body 120 faces. The positioning device 123 includes two receivers that receive positioning signals from artificial satellites constituting GNSS (Global Navigation Satellite System). The two receivers are installed at different positions of the slewing body 120 respectively. The positioning device 123 detects information indicating latitude, longitude, and altitude, and the position of the representative point (the origin of the excavator coordinate system) of the slewing body 120 in the field coordinate system based on the positioning signals received by the receivers. The positioning device 123 calculates the azimuth in which the slewing body 120 faces as the relationship between the installation positions of the two receivers with respect to the installation position of one of the receivers using the respective positioning signals received by the two receivers. In other embodiments, the positioning device 123 may detect the azimuth in which the slewing body 120 faces based on measurement values of a rotary encoder or an IMU (Inertial Measurement Unit).

[0029] The inclination measuring device 124 measures the acceleration and angular velocity of the slewing body 120, and detects the posture of the slewing body 120 (for example, inclination angles such as roll angle, pitch angle, yaw angle, etc.) based on the measurement results. The inclination measuring device 124 is installed, for example, on the lower surface of the slewing body 120. The inclination measuring device 124 can use, for example, an inertial measurement unit (IMU). The inclination measuring device 124 may be an inclinometer that detects the inclination angle regardless of acceleration and angular velocity. The working machine 100 according to other embodiments may not include the positioning device 123. FIG. 2 shows a state in which the roll angle θr, pitch angle θp, and yaw angle θy of the slewing body 120 are all 0 degrees and the directions of the three axes of XYZ in the global coordinate system coincide with the directions of the three axes of the field coordinate system (local coordinate system). Also, in the present embodiment, the roll angle θr, pitch angle θp, and yaw angle θy of the slewing body 120 are also referred to as the roll angle, pitch angle, and yaw angle of the working machine 100. Also, the X direction is also referred to as the left - right direction, the Y direction as the front - rear direction, and the Z direction as the height direction.

[0030] The hydraulic device 125 includes a hydraulic oil tank, a hydraulic pump, and a flow control valve. The hydraulic pump is driven by the power of an engine or an electric motor (not shown) and supplies hydraulic oil to the boom cylinder 134, the arm cylinder 135, and the bucket cylinder 136 via the flow control valve. The flow control valve has a rod-shaped spool, and adjusts the flow rate of the hydraulic oil supplied to the boom cylinder 134, the arm cylinder 135, and the bucket cylinder 136 according to the position of the spool. The spool is driven based on a control command received from the work machine control device 126. That is, the amount of hydraulic oil supplied to the boom cylinder 134, the arm cylinder 135, and the bucket cylinder 136 is controlled by the work machine control device 126.

[0031] The work machine control device 126 transmits information on the captured image captured by the front camera 122, the position of the revolving body 120 (including information indicating latitude, longitude, and altitude), the azimuth and tilt angle (posture), the boom 131, and the tilt angles of the arm 132 and the bucket 133 to the remote operation room 500 via the communication device 143. Note that the information on the tilt angles of the boom 131, and the arm 132 and the bucket 133 is also referred to as work machine posture information. In the present embodiment, the information measured by various sensors provided in the work machine 100 and transmitted by the work machine control device 126 is also referred to as vehicle body information. The work machine control device 126 receives an operation signal from the remote operation room 500 via the communication device 143. The work machine control device 126 drives the work machine 130, the revolving body 120, or the traveling body 110 based on the received operation signal.

[0032] (Transport vehicle) The transport vehicle 200 is an off-road dump truck equipped with a vessel in the embodiment, and is an example of a transport vehicle. Further, in the present embodiment, the transport vehicle 200 is operated under the control of the unmanned dump truck operation control device 300. The transport vehicle 200 includes a dump truck control device 201, a positioning device 202, and a communication device 203. Similar to the positioning device 123, the positioning device 202 includes two receivers that receive positioning signals from artificial satellites constituting GNSS, and acquires the position of the transport vehicle 200 and the azimuth in which the transport vehicle 200 faces. The communication device 203 transmits and receives a predetermined signal to and from the unmanned dump truck operation control device 300 or the communication device 560 of the remote operation room 500. The dump truck control device 201 moves the transport vehicle 200 between, for example, predetermined points (loading point of excavated material, loading and unloading point, waiting place, etc.) under the control of the unmanned dump truck operation control device 300. Further, the dump truck control device 201 repeatedly provides the position (information including latitude, longitude, and altitude) and azimuth information of the transport vehicle 200 at a predetermined cycle, for example, to the remote operation room 500 via the unmanned dump truck operation control device 300.

[0033] (Control device of remote operation room) The control device 550 is configured using one or more computers such as a microcontroller, and includes a control unit 551 as a functional configuration composed of a combination of hardware such as the computer and peripheral devices of the computer, and software such as a program executed by the computer. Further, the control unit 551 includes an acquisition unit 552, a display unit 553, and a communication unit 554.

[0034] The communication unit 554 receives, from the work machine 100 via the communication device 560, a video signal representing the captured image P1 captured by the front camera 122, the position of the revolving body 120 (information indicating longitude, latitude, and altitude), the azimuth and tilt angles, and information on the tilt angles of the boom 131, the arm 132, and the bucket 133 (also referred to as attitude information). Further, the communication unit 554 receives, from the transport vehicle 200 via the communication device 560, the position of the transport vehicle 200 (information indicating longitude, latitude, and altitude) and azimuth information, either directly or via the unmanned dump truck operation control device 300 or the like. Also, the communication unit 554 transmits an operation signal corresponding to the operation of the operation device 540 to the work machine 100.

[0035] The acquisition unit 552 acquires, via the communication unit 554, a video signal representing the captured image P1 captured by the front camera 122, the position of the revolving body 120 (information indicating longitude, latitude, and altitude) and azimuth, attitude information of the work machine 130 representing the tilt angles of the boom 131, the arm 132, and the bucket 133, and measurement information such as attitude information indicating the roll angle and pitch angle of the work machine 100, and also acquires the position of the transport vehicle 200 (information indicating longitude, latitude, and altitude) and azimuth information. Note that the acquisition of the information indicating the roll angle may be omitted.

[0036] Next, with reference to FIGS. 5 to 13, the display unit 553 will be described. FIG. 5 is a schematic diagram showing an example of an image cut out from the captured image P1 according to the embodiment of the present disclosure. FIG. 6 is a schematic diagram showing an example of an image displayed by the first display device 520 according to the embodiment of the present disclosure. FIG. 7 is a schematic diagram showing an example of the composite image P2 according to the embodiment of the present disclosure. FIGS. 8 and 10 are schematic diagrams for explaining the composite image P2 according to the embodiment of the present disclosure. FIGS. 7, 9, and 11 to 13 are schematic diagrams showing the composite images P2 (P2, P2a to P2d) according to the embodiment of the present disclosure.

[0037] The display unit 553 performs two processes: an image clipping process for dividing the captured image P1 and displaying it on a plurality of displays of the first display device 520, and a display process for superimposing a composite image P2 including a side image or the like, which is an image depicting the side surface of the working machine 100, on one of the clipped images and displaying it on the first display device 520.

[0038] In the image clipping process, for example, as shown in FIG. 5, the display unit 553 clips from the captured image P1 a central image P11 for display on the central display 521, a left image P12 for display on the left display 522, a right image P13 for display on the right display 523, an upper image P14 for display on the upper display 524, and a lower image P15 for display on the lower display 525. When the first display device 520 is composed of one display, it is not necessary to clip the captured image P1.

[0039] In the display process, the display unit 553 generates composite images P2, P2a to P2d (hereinafter, collectively referred to as composite image P2) exemplified in FIGS. 7, 9, and 11 to 13, and superimposes the composite image P2 on one of the clipped images and displays it on the first display device 520 as shown in FIG. 6, for example. In the example shown in FIG. 6, the central image P11 is displayed on the central display 521, the left image P12 is displayed on the left display 522, the right image P13 with the composite image P2 superimposed is displayed on the right display 523, the upper image P14 is displayed on the upper display 524, and the lower image P15 is displayed on the lower display 525. In this embodiment, generating an image includes, for example, creating an image based on one or more images that simulate the whole or each part of the working machine 100 prepared in advance, pasting an image in a predetermined drawing area (for example, a storage area set as an object for drawing), drawing an image in a raster format or a vector format, performing modification processes such as coloring, rotating the image, and changing the size, orientation, or position of the image.

[0040] As shown in FIG. 7, the composite image P2 includes a side image P21, a grid image P22, scale images P23 and P24, a reference image P25, an inclined surface image P26, and an inclination angle image P27. The side image P21 is an image of the working machine 100 viewed from the side. The side image P21 is drawn at a position corresponding to the actual height with reference to the reference image P25. The display unit 553 generates a side image P21 representing the current posture of the working machine 130 based on the inclination angle information of the boom 131, the arm 132, and the bucket 133 acquired by the acquisition unit 552. Further, the display unit 553 generates a side image P21 representing the current posture of the working machine 100 based on the inclination angle of the revolving body 120 acquired by the acquisition unit 552.

[0041] The grid image P22 includes a plurality of scale lines arranged in a grid pattern every 5 m. The scale image P23 includes numerical values representing the distance in the height direction from the reference height (0 m). In the example shown in FIG. 7, the scale image P23 includes numerical values from -10 m to 25 m every 5 m. The scale image P24 includes numerical values representing the distance in the front-rear direction with the swing axis CR (FIG. 2) as 0 m. In the example shown in FIG. 7, the scale image P24 includes numerical values from 0 m to 25 m every 5 m. The scale numbers and numerical values of the scale images P23 and P24 can be set arbitrarily.

[0042] The reference image P25 is an image representing the reference in the height direction (0 m). In the example shown in FIG. 7, it is an image representing a line segment thicker than the thickness of the scale lines of the grid image P22. Note that the reference image P25 may be, for example, an image with the same thickness as the scale lines of the grid image P22 but a different color, or a line type such as a dashed line other than a solid line.

[0043] The inclined surface image P26 is an image representing the inclined surface in front of the working machine 100. For example, as shown in FIG. 8, when the working machine 100 attempts to ascend the inclined surface 2s of the scaffold 2 and the cutting edge 133T contacts the inclined surface 2s, the display unit 553 generates an inclined surface image P26 as shown in FIG. 7. In the example shown in FIG. 7, the inclined surface image P26 includes an image representing a line segment connecting the first position P101 based on the traveling body 110 and the second position P102 based on the working machine 130 based on the posture information. The first position P101 corresponds to, for example, the point where the vertical line (line segment in the Z direction) VL dropped from the tip 100t of the traveling body 110 intersects the ground contact surface 110s of the traveling body 110. The second position P102 is, for example, the position of the cutting edge 133T of the bucket 133.

[0044] The inclination angle image P27 is an image representing the inclination angle θs of the inclined surface image P26. In the example shown in FIG. 7, the inclination angle image P27 represents the characters "Slope: 23°". The inclination angle θs can be, for example, the angle formed by the line segment connecting the first position P101 and the second position P102 and the horizontal line HL passing through the first position P101.

[0045] In the present embodiment, when the display unit 553 causes the first display device 520 to display the composite image P2 including the side surface image P21 of the working machine 100, based on the predetermined measurement information measured by the working machine 100, the inclined surface image P26 representing the inclined surface 2s in front of the working machine 100 and the inclination angle image P27 representing the inclination angle θs of the inclined surface 2s are included in the composite image P2. Here, the measurement information is information measured by a predetermined sensor mounted on the working machine 100, and includes, for example, the posture information representing the posture of the working machine 130 (information such as the inclination angles of the boom 131, the arm 132, and the bucket 133) and the information on the posture of the working machine 100 (information indicating the pitch angle of the working machine 100, etc.). When the working machine 100 includes the traveling body 110 and the working machine 130, the measurement information includes the posture information representing the posture of the working machine 130, and the inclined surface image P26 includes an image connecting the first position P101 based on the traveling body 110 and the second position P102 based on the working machine 130 based on the posture information. When the working machine 130 includes the bucket 133, the second position P102 can be a predetermined position such as the cutting edge 133T of the bucket 133.

[0046]

[0047] For example, when the operator attempts to raise the inclined surface 2s as shown in FIG. 8, the operator can confirm the inclination angle θs of the inclined surface 2s based on the inclined surface image P26 and the inclination angle image P27 by bringing the cutting edge 133T into contact with (or approaching) the inclined surface 2s.

[0048] In addition, the display unit 553 may display the inclined surface image P26 and the inclination angle image P27, for example, when the cutting edge 133T comes into contact with (or approaches) the inclined surface 2s and the operator performs a predetermined input operation, or may always display them regardless of the positional relationship with the inclined surface 2s. That is, the inclined surface image P26 is an image representing the inclination in front of the working machine 100 based on the working implement 130.

[0049] In addition, when the other working machine is the transport vehicle 200, the height with respect to the other working machine can correspond to the height of the traveling path R200 of the transport vehicle 200. Further, as the predetermined height of the surroundings, in addition to the above example, for example, the height of the moving place of the excavated material, the target height of the scaffold when laying the scaffold, etc. can be used. Further, the other working machine is not limited to the transport vehicle, and can be the height with respect to other working machines in general that cooperate. Further, the display unit 533 may switch the height of the reference image P25, for example, in response to a predetermined input operation of the operator.

[0050] For example, the height of the reference image P25 included in the composite image P2 shown in FIG. 7 corresponds to the height of the ground contact surface 110s of the traveling body 110 of the working machine 100 shown in FIG. 8, for example. Further, for example, the height of the reference image P25 included in the composite image P2a shown in FIG. 9 corresponds to the height of the traveling path R200 of the transport vehicle 200 shown in FIG. 10, for example. FIG. 10 schematically shows the positional relationship between the working machine 100 and the transport vehicle 200. In FIG. 10, the working machine 100 is arranged on the scaffold 2. Further, the transport vehicle 200 is located on the traveling path R200 of the transport vehicle 200. The traveling path R200 may be a work site where the transport vehicle 200 stops and the working machine 100 loads the cargo onto the transport vehicle 200. In this case, the height of the traveling path R200 of the transport vehicle 200 is lower than the height of the ground contact surface 110s of the traveling body 110 of the working machine 100. Therefore, the position of the side image P21a included in the composite image P2a shown in FIG. 9 is separated upward from the reference image P25 by the difference in height between the traveling path R200 and the ground contact surface 110s. In this case, the operator can recognize the height of the traveling path R200 of the transport vehicle 200 by checking the composite image P2a.

[0051] Note that the display unit 553 can switch the height of the reference image P25 based on the positional relationship between the working machine 100 and the transport vehicle 200, for example, as follows. That is, the display unit 553 can, for example, as shown in FIG. 10, use the height of the traveling path R200 as a reference when a predetermined position P200 of the transport vehicle 200 is located within a predetermined range (within a diameter D200) from the position (loading point) at the time of loading of the transport vehicle 200, and use a predetermined height of the working machine 100 (for example, the height of the ground contact surface 110s of the traveling body 110) as a reference when the predetermined position P200 of the transport vehicle 200 is not located within the range (within a diameter D200). Note that the position P200 corresponds to, for example, the installation position of the positioning device 202.

[0052] FIGS. 11 to 13 show other examples of the composite image P2 generated and displayed by the display unit 553. The composite image P2b shown in FIG. 11 has a different posture of the working machine in the side image P21b compared to the composite image P2 shown in FIG. 7. The composite image P2c shown in FIG. 12 is an example when the pitch angle θp of the working machine 100 is -15° (the pitch angle θp in FIG. 7 is 0°). The side image P21c is inclined by an amount corresponding to the pitch angle. The composite image P2 shown in FIG. 7 has a pitch angle θp of 0° for the working machine 100 and a tilt angle θs of 23°. That is, the tilt angle θs of 38° shown in FIG. 12 includes the tilt angle θs of 23° when the pitch angle θp of the working machine 100 is 0° and the -15° of the pitch angle θp of the working machine 100. The composite image P2d shown in FIG. 13 is an example when the pitch angle θp of the working machine 100 is +15°. The side image P21d is inclined by an amount corresponding to the pitch angle. Also, the reference image P25 shown in FIGS. 12 and 13 is based on the intermediate height of the traveling body 110 of the working machine 100, but may be based on, for example, the height of the first position P101 or the height of the traveling path R200 of the transport vehicle 200.

[0053] (Operation example of remote operation cab) FIG. 14 is a flowchart showing a display control method by the control device 550 of the remote operation cab according to an embodiment of the present disclosure. When the remote operation of the working machine 100 is started, the control device 550 executes the following display control at a predetermined cycle.

[0054] In the process shown in FIG. 14, first, the acquisition unit 552 acquires the position information of the transport vehicle 200 (step S1). Next, the acquisition unit 552 acquires the captured image P1 captured by the front camera 122 of the work machine 100, the position information, the pitch angle information, and the attitude information representing the attitude of the working machine 130 (step S2). Next, the display unit 553 cuts out a central image P11, a left image P12, a right image P13, an upper image P14, and a lower image P15 from the acquired captured image P1, respectively (step S3). Next, the display unit 553 determines whether the position of the transport vehicle 200 is within a predetermined range from the loading position (step S4). If the position of the transport vehicle 200 is within a predetermined range from the loading position (step S4: YES), the display unit 553 determines the height of the travel path R200 of the transport vehicle 200 as the height of the reference image P25 (step S5). On the other hand, if the position of the transport vehicle 200 is not within a predetermined range from the loading position (step S4: NO), the display unit 553 determines a predetermined height of the work machine 100 as the height of the reference image P25 (step S6).

[0055] Next, the display unit 553 calculates the inclination angle θs of the inclined surface 2s (step S7). Next, the display unit 533 generates a reference image P25, grid images P22, scale images P23 and P24 (step S8). Next, the display unit 533 generates a side image P21 of the working machine 100 based on the height of the reference image P25, the position information of the working machine 100, the pitch angle, and the attitude information representing the attitude of the working unit 130 (step S9). Next, the display unit 533 generates an inclined surface image P26 (step S10). Here, the inclined surface image P26 will be included in the composite image P2 including the reference image P25, the grid images P22, the scale images P23 and P24, and the side image P21. Next, the display unit 533 generates an inclination angle image P27 (step S11). Here, the inclination angle image P27 will be included in the composite image P2. Next, the display unit 533 superimposes the composite image P2 on the cut-out image and displays it on the first display device 520 (step S12), and the process shown in FIG. 14 ends. Note that the composite image P2 may include only the side image 21 and the reference image P25, or may include only the side image 21 and the inclined surface image P26. Note that the generation of the grid images P22, the scale images P23 and P24, and the inclination angle image P27 may be omitted.

[0056] (Function and Effect) As described above, the control device 550 (display control system) according to the embodiment of the present disclosure is a display control system for the first display device 520 (display device), and includes an acquisition unit 552 that acquires measurement information measured by the working machine 100, and a display unit 553 that displays a composite image P2 including a side image P21 of the working machine 100 and a reference image P25 representing a reference in the height direction on the first display device 520 (display device). The display unit 553 draws the side image P21 at a position corresponding to the actual height with reference to the reference image P25 based on the measurement information. According to this configuration, information in the height direction can be supplemented by the composite image P2. That is, in the loading operation for loading the load excavated by the working machine 100 onto the transport vehicle 200 using the remote operation system 1, for example, even if the transport vehicle 200 is at a position lower than the working machine 100, the height of the transport vehicle 200 and the inclination of the working machine 100 can be recognized by the composite image P2.

[0057] (Other embodiments) As described above, one embodiment has been described in detail with reference to the drawings. However, the specific configuration is not limited to the above, and various design changes and the like are possible. For example, in the above-described embodiment, the control device 550 generates a composite image, but it is not limited to this. For example, in other embodiments, the work machine control device 126 of the work machine 100 or an external server device may generate all or part of the composite image P2, and the control device 550 may receive information representing all or part of the composite image P2.

[0058] Also, in the above-described embodiment, the control device 550 calculates the cutting edge position of the bucket 133, but it is not limited to this in other embodiments. For example, the control device 550 according to other embodiments may calculate the lowest bucket height, which is the distance from the ground surface to the lowest point of the bucket, instead of the cutting edge position. The lowest point of the bucket is the point closest to the ground surface among the bucket 133. The control device 550 can identify the lowest point of the bucket 133 by storing in advance the shape of the bucket 133 with respect to the base end portion of the bucket 133 instead of or in addition to the length from the base end portion to the cutting edge of the bucket 133. Both the cutting edge position and the lowest point of the bucket are examples of the position of the bucket 133.

[0059] Also, in the above-described embodiment, the display control system is implemented in the remote operation system 1, but it is not limited to this. For example, in other embodiments, the control unit 551 may be applied to a radio control system that operates the work machine 100 by wireless communication at a position outside the work machine 100 and where the work machine 100 can be visually recognized. When the display control system is applied to the radio control system, the control device may include a display device.

[0060] Alternatively, instead of using the position information acquired by the transport vehicle 200 using the positioning device 202 as a reference, for example, the work machine 100 may be provided with a LiDAR (Light Detection And Ranging), a stereo camera, etc., to acquire three-dimensional point cloud information (three-dimensional position information of a plurality of measurement points) of the runway R200, and based on the acquired three-dimensional point cloud information, the height of the runway R200 may be acquired with reference to the work machine 100. Alternatively, the position information of the transport vehicle 200 may be, for example, the position information acquired by another transport vehicle 200.

[0061] Alternatively, instead of acquiring the information of the slope 2s based on the attitude information of the work implement 130, etc., the work machine 100 may be provided with a LiDAR, a stereo camera, etc., to acquire three-dimensional point cloud information of the slope 2s, and based on the acquired three-dimensional point cloud information, the inclination angle of the slope 2s may be acquired with reference to the work machine 100. In this case, the three-dimensional point cloud information corresponds to the "predetermined measurement information measured by the work machine" according to the present disclosure.

[0062] Also, the reference image P25 may not be an image representing a line segment. For example, an icon such as an arrow or a triangle may be used to represent the reference height.

[0063] Also, the predetermined height of the surroundings may be acquired by recognizing the captured image P1 captured using the above-described LiDAR, stereo camera, monocular camera, etc.

[0064] Also, part or all of the program executed by the computer in the above embodiment can be distributed via a computer-readable recording medium or a communication line.

[0065] (Appendix) The display control system (control device 550) and the remote operation system 1 according to the present disclosure are understood, for example, as follows.

[0066] (1) The display control system (control device 550) according to the first aspect is a display control system for a display device (first display device 520), and includes an acquisition unit 552 that acquires measurement information measured by the work machine 100, and a display unit 553 that displays a composite image P2 including a side image P21 of the work machine 100 and a reference image P25 representing a reference in the height direction on the display device. The display unit 553 uses a predetermined height around the work machine 100 at a height different from the ground contact surface of the work machine 100 as a reference for the reference image P25, and based on the measurement information, draws the side image P21 at a position corresponding to the actual height with reference to the reference image P25. According to this aspect and the following aspects, height direction information can be supplemented by the composite image P2.

[0067] (2) The display control system (control device 550) according to the second aspect is the display control system of (1), and the height of the reference image is switched to either the height based on the work machine or a predetermined height around the work machine to generate the composite image.

[0068] (3) The display control system (control device 550) according to the third aspect is the display control system of (2), and the predetermined height around is the height based on another work machine.

[0069] (4) The display control system (control device 550) according to the fourth aspect is the display control system of (3), and performs the switching based on the positional relationship between the work machine and the other work machine.

[0070] (5) The display control system (control device 550) according to the fifth aspect is the display control system of (3) or (4), and the other work machine is a transport vehicle, and the height based on the transport vehicle corresponds to the height of the traveling path of the other transport vehicle.

[0071] (6) The display control system (control device 550) according to the sixth aspect is the display control system of (1) to (5), and the measurement information includes the pitch angle of the work machine, and based on the pitch angle, the side image is displayed in an inclined manner.

[0072] (7) The display control system (control device 550) according to the seventh aspect is the display control system of (1) to (6), wherein the work machine includes a working machine, the measurement information includes attitude information representing the attitude of the working machine, and based on the attitude information, the side image is displayed.

[0073] (8) The display control system (control device 550) according to the eighth aspect is the display control system of (1) to (7), wherein the work machine includes an imaging device (front camera 122), and the composite image is superimposed on the captured image captured by the imaging device and displayed on the display device.

[0074] (9) The remote operation system according to the ninth aspect includes an acquisition unit that acquires measurement information measured by a work machine including an imaging device, and a display unit that superimposes a composite image including a side image of the work machine and a reference image representing a reference in the height direction on the captured image captured by the imaging device and displays the composite image on a display device provided at a remote location of the work machine. The display unit uses a predetermined height around the work machine at a height different from the ground contact surface of the work machine as a reference for the reference image, and based on the measurement information, draws the side image at a position corresponding to the actual height with reference to the reference image. The remote operation system includes the display control system, the display device, and an operation device of the work machine provided at the remote location. According to this aspect, information in the height direction can be supplemented by the composite image.

Description of reference numerals

[0075] 1…Remote operation system 100…Work machine 130…Working machine 120…Rotating body 110…Traveling body 131…Boom 132…Arm 133…Bucket 134…Boom cylinder 135…Arm cylinder 136…Bucket cylinder 137…Boom angle sensor 138…Arm angle sensor 139…Bucket angle sensor 121…Driver's cab 122…Front camera 123…Positioning device 124…Inclinometer 125…Hydraulic device 126…Work machine control device 200…Transport vehicle 202…Positioning device 500…Remote operation cab 510…Driver's seat 520…First display device 530…Second display device 540…Operating device 550…Control device 551…Control unit 552…Acquisition unit 553…Display unit 554…Communication unit, P2, P2a~P2d…Composite image, P21, P21a~P21d…Side image, P25…Reference image, P26…Slope image, P27…Inclination angle image

Claims

1. A display control system for a display device, comprising: an acquisition unit that acquires measurement information measured by a work machine; a display unit that displays on the display device a composite image including a side image of the work machine and a reference image representing a reference in the height direction; The display control system is provided with: The display unit uses a predetermined height around the work machine at a height different from the ground contact surface of the work machine as a reference for the reference image, and based on the measurement information, draws the side image at a position corresponding to the height of the reference image. Display control system.

2. The height of the reference image is switched to either the height based on the work machine or a predetermined height around the work machine to generate the composite image. The display control system according to claim 1.

3. The predetermined height around is the height based on another work machine. The display control system according to claim 2.

4. The switching is performed based on the positional relationship between the work machine and the other work machine. The display control system according to claim 3.

5. The other work machine is a transport vehicle, The height based on the transport vehicle corresponds to the height of the travel path of the other transport vehicle. The display control system according to claim 3.

6. The measurement information includes the pitch angle of the work machine, and based on the pitch angle, the side image is displayed obliquely. The display control system according to claim 5.

7. The work machine is provided with a working machine, The measurement information includes attitude information representing the attitude of the working machine, and based on the attitude information, the side image is displayed. The display control system according to claim 5 or 6.

8. The work machine is provided with an imaging device, The composite image is superimposed on the captured image captured by the imaging device and displayed on the display device. The display control system according to claim 7.

9. An acquisition unit that acquires measurement information measured by a work machine provided with an imaging device, and a display unit that superimposes a composite image including a side image of the work machine and a reference image representing a reference in the height direction on the captured image captured by the imaging device and displays it on a display device provided at a remote location of the work machine. The display unit uses a predetermined height around the work machine at a height different from the ground contact surface of the work machine as a reference for the reference image, and based on the measurement information, draws the side image at a position corresponding to the height of the reference image. A display control system, The display device, An operation device of the work machine provided at the remote location, A remote operation system comprising:

10. A display control method for a display device, comprising: acquiring measurement information measured by a work machine; displaying, on the display device, a composite image including a side image of the work machine and a reference image representing a reference in the height direction; using a predetermined height around the work machine at a height different from the ground contact surface of the work machine as a reference for the reference image, and drawing the side image at a position corresponding to the height of the reference image based on the measurement information; A display control method including the above steps.

Images