Display control system, display control method, and remote operation system

Abstract

A display control system according to the present invention, which is a display control system for a display device, is provided with: an acquisition unit that acquires measurement information measured by a work machine; and a display unit that displays, on the display device, a composite image including a side surface image of the work machine and a reference image representing a reference in a height direction. The display unit uses a prescribed height around the work machine, which is different from the height of a grounding surface of the work machine, as the reference of the reference image, and draws the side surface image at a position corresponding to an actual height with reference to the reference image on the basis of the measurement information.

Description

Display control system, display control method, and remote operation system

[0001] This application claims priority to Japanese Patent Application No. 2023-201675, filed on November 29, 2023, the contents of which are incorporated herein by reference.

[0002] Patent Document 1 describes a display control system that controls a display device in a remote driver's cab for remotely operating a work vehicle (hereinafter referred to as a work machine). In the display control system described in Patent Document 1, a supplemental image is displayed on the display device to compensate for the lack of sensation obtained when riding the work machine. Here, the supplemental image is, for example, a posture image that includes an image depicting the side of the work machine that shows the current posture of the work machine.

[0003] International Publication No. 2020 / 090985

[0004] When a work machine loads a transport vehicle using a remote control system, it can be difficult for the operator to recognize the height of the transport vehicle from the camera image. Furthermore, in manned work machines, operators who are unfamiliar with the operation can have the same problem. In contrast, the device described in Patent Document 1 has the problem that information about the height of the transport vehicle can sometimes be insufficient.

[0005] In order to solve the above-mentioned problems, the present disclosure aims to provide a display control system, a display control method, and a remote control system that can supplement information in the height direction.

[0006] The display control system of the present disclosure is a display control system for a display device, and includes an acquisition unit that acquires measurement information measured by a work machine, and a display unit that displays on the display device a composite image including a side image of the work machine and a reference image that represents a reference in the height direction, wherein the display unit uses a predetermined height around the work machine that is different from the ground surface of the work machine as a reference for the reference image, and draws the side image at a position that corresponds to the actual height based on the measurement information, using the reference image as a reference.

[0007] The display control method disclosed herein is a display control method for a display device, and includes the steps of acquiring measurement information measured by a work machine, displaying on the display device a composite image including a side image of the work machine and a reference image representing a height reference, and using a predetermined height around the work machine that is different from the ground surface of the work machine as a reference for the reference image, and drawing the side image at a position corresponding to the actual height based on the measurement information, using the reference image as a reference.

[0008] The remote operation system of the present disclosure comprises an acquisition unit that acquires measurement information measured by a work machine equipped with an imaging device, and a display unit that superimposes a composite image including a side image of the work machine and a reference image that represents a height reference on an image captured by the imaging device and displays the superimposed image on a display device provided at a remote location from the work machine, wherein the display unit uses a predetermined height around the work machine that is different from the ground surface of the work machine as a reference for the reference image, and draws the side image at a position that corresponds to the actual height based on the measurement information, the display device, and an operation device for the work machine provided at the remote location.

[0009] According to the display control system, display control method, and remote control system of the present disclosure, information in the height direction can be supplemented by a composite image.

[0010] FIG. 1 is a schematic diagram showing an example configuration of a remote control system according to an embodiment of the present disclosure; FIG. 2 is an external view of a work machine according to an embodiment of the present disclosure; FIG. 3 is a schematic diagram showing an example of a captured image captured by an imaging device of a work machine according to an embodiment of the present disclosure; FIG. 4 is a schematic block diagram showing an example configuration of a remote control system according to an embodiment of the present disclosure; FIG. 5 is a schematic diagram showing an example of an image cut out from a captured image according to an embodiment of the present disclosure; FIG. 6 is a schematic diagram showing an example of an image displayed by a display device according to an embodiment of the present disclosure; FIG. 7 is a schematic diagram for explaining a composite image according to an embodiment of the present disclosure; FIG. 8 is a schematic diagram for explaining a composite image according to an embodiment of the present disclosure; FIG. 9 is a schematic diagram showing an example of a composite image according to an embodiment of the present disclosure; FIG. 10 is a flowchart showing an example operation of a remote control system according to an embodiment of the present disclosure;

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In each drawing, the same or corresponding components are designated by the same reference numerals, and the description thereof will be omitted as appropriate.

[0012] (Remote Control System) FIG. 1 is a schematic diagram showing an example configuration of a remote control system according to an embodiment of the present disclosure. FIG. 4 is a schematic block diagram showing an example configuration of a remote control system 1 according to an embodiment of the present disclosure. The remote control system 1 is a remote control system for remotely controlling a work machine 100 equipped with a work implement 130 having a work tool, and includes the work machine 100 that operates by remote control, a transport vehicle 200 such as a dump truck that is a loading target for loading material excavated by a bucket 133 that is the work tool, and a remote cab 500 for remote control. The work machine 100 operates at a work site (e.g., a mine, a quarry, etc.). The remote cab 500 is provided at a location remote from the work machine 100 (a location remote from the work machine 100; for example, a city, within the work site, etc.). The work machine 100 and the remote cab 500 are connected via communication means such as a mobile communication network or the Internet. The remote control system 1 is a system for operating the work machine 100 using the remote cab 500. The transport vehicle 200 and the remote driver's cab 500 are connected via a communication means such as a mobile communication network or the Internet, and further connected via an unmanned dump truck operation control device, which will be described later.

[0013] The work machine 100 operates in accordance with operation signals received from the remote operator's cab 500. In other words, no operator is on board the work machine 100. The remote operator's cab 500 accepts operations of the work machine 100 through operations by the operator, and transmits operation signals to the work machine 100. The operation signals will be described later.

[0014] (Remote operator's cab) The remote operator's cab 500 includes a driver's seat 510, a first display device 520, a second display device 530, an operation device 540, a control device 550, and a communication device 560. The first display device 520 is disposed in front of the driver's seat 510. The first display device 520 is located in front of the operator when the operator sits in the driver's seat 510. As shown in FIG. 1 , the first display device 520 is composed of a center display 521, a left display 522, a right display 523, an upper display 524, and a lower display 525, which are arranged side by side. The left display 522 is provided to the left of the center display 521. The right display 523 is provided to the right of the center display 521. The upper display 524 is provided above the center display 521. The lower display 525 is provided below the center display 521. The first display device 520 divides and displays, for example, part or all of an image captured by the work machine 100 on five displays.

[0015] In other embodiments, the number of displays constituting first display device 520 is not limited to this. For example, first display device 520 may be constituted by a single display. Furthermore, first display device 520 may project a captured image onto a curved or spherical surface using a projector or the like. Furthermore, first display device 520 is one configuration example of a "display device" according to the present disclosure.

[0016] The second display device 530 is disposed diagonally forward of the driver's seat 510. The second display device 530 displays vehicle body information (remaining fuel, engine water temperature, etc.) transmitted from the work machine 100, notifications of abnormalities in the work machine 100, and the like. Note that in other embodiments, the position of the second display device 530 does not have to be diagonally forward of the driver's seat 510, as long as it is in a position that is visible to the operator. Furthermore, the remote driver's cab 500 according to other embodiments does not need to be equipped with the second display device 530, and preferably the vehicle body information and notifications of abnormalities that are displayed on the second display device 530 may be displayed on the first display device 520.

[0017] The operation device 540 is disposed near the driver's seat 510. The operation device 540 is located within an operable range of the operator when the operator sits in the driver's seat 510. The operation device 540 includes, for example, an electric lever and an electric pedal. When the operator operates the electric lever and the electric pedal, the operation device 540 outputs a work implement operation signal, which is an operation signal for the boom 131, the arm 132, and the bucket 133, a swing operation signal for the swing body 120, and a travel operation signal for the travel body 110. In this embodiment, the work implement operation signal, swing operation signal, and travel operation signal are collectively referred to as operation signals.

[0018] The control device 550 displays the captured images and vehicle body information received from the work machine 100 on the first display device 520. In other words, the control device 550 is an example of a display control system. The control device 550 also transmits operation signals input to the operation device 540 to the work machine 100. The display control system may be a remote operator cab 500 including the control device 550, or may further include a work machine control device 126 of the work machine 100 and a forward camera 122. The display control system includes a positioning device 123 that measures the height of the work machine 100 as measurement information, an acquisition unit 552 acquires the measured measurement information, and a display unit 553 that generates a composite image P2 including a side image P21 of the work machine 100 and a reference image P25 that represents a reference in the height direction, and a display device that generates the side image P21 based on the acquired measurement information and displays the composite image including the side image P21 and the reference image P25.

[0019] The communication device 560 transmits and receives video signals representing captured images, operation signals, signals representing vehicle body information, etc. to and from the communication device 143 of the work machine 100 .

[0020] (Working Machine) Figure 2 is an external view of a working machine 100 according to an embodiment of the present disclosure. The working machine 100 according to an embodiment of the present disclosure is a backhoe excavator, which is a type of hydraulic excavator. Note that the working machine 100 according to other embodiments may be a hydraulic excavator other than a backhoe excavator, such as a face excavator or a rope excavator, or a working machine other than a hydraulic excavator, such as a wheel loader or a bulldozer. The working machine 100 includes a hydraulically driven working implement 130, a revolving unit 120 that supports the working implement 130, and a traveling unit 110 that supports the revolving unit 120. The revolving unit 120 revolves around a revolving axis CR as a central axis of rotation.

[0021] The work machine 130 includes a boom 131, an arm 132, and a bucket 133, which is a work implement. The work machine 130 is driven by extension and contraction of a boom cylinder 134, an arm cylinder 135, and a bucket cylinder 136. A boom angle sensor 137, an arm angle sensor 138, and a bucket angle sensor 139 are attached to the boom 131, the arm 132, and the bucket 133, respectively.

[0022] The base end of the boom 131 is attached to the rotating body 120 via a pin. The arm 132 connects the boom 131 and the bucket 133. The base end of the arm 132 is attached to the tip of the boom 131 via a pin. The bucket 133 is equipped with a cutting edge 133T for excavating earth and sand, and a container for storing the excavated earth and sand. The base end of the bucket 133 is attached to the tip of the arm 132 via a pin.

[0023] The boom cylinder 134 is a hydraulic cylinder for driving the boom 131. A base end of the boom cylinder 134 is attached to the rotating body 120. A tip end of the boom cylinder 134 is attached to the boom 131. The arm cylinder 135 is a hydraulic cylinder for driving the arm 132. A base end of the arm cylinder 135 is attached to the boom 131. A tip end of the arm cylinder 135 is attached to the arm 132. The bucket cylinder 136 is a hydraulic cylinder for driving the bucket 133. A base end of the bucket cylinder 136 is attached to the arm 132. A tip end of the bucket cylinder 136 is attached to the bucket 133.

[0024] The boom angle sensor 137 is attached to, for example, the boom 131 and detects the inclination angle of the boom 131. The arm angle sensor 138 is attached to, for example, the arm 132 and detects the inclination angle of the arm 132. The bucket angle sensor 139 is attached to, for example, the bucket 133 and detects the inclination angle of the bucket 133. The boom angle sensor 137, the arm angle sensor 138, and the bucket angle sensor 139 according to embodiments of the present disclosure detect the inclination angle with respect to the ground plane. Note that angle sensors according to other embodiments are not limited to this, and may detect the inclination angle with respect to another reference plane. For example, in other embodiments, the angle sensors may detect the relative rotation angle using potentiometers provided at the base ends of the boom 131, the arm 132, and the bucket 133, or may detect the inclination angle by measuring the cylinder lengths of the boom cylinder 134, the arm cylinder 135, and the bucket cylinder 136 and converting the cylinder lengths into angles.

[0025] The revolving unit 120 is provided with a cab 121. The cab 121 is provided on the left side of the work machine 130. The cab 121 is provided with a front camera 122. The front camera 122 is installed at the front and upper part of the cab 121. The front camera 122 captures images in front of the cab 121 through a windshield in front of the cab 121. Here, "forward" refers to the direction in which the work machine 130 is mounted on the revolving unit 120, and "rearward" refers to the opposite direction of "forward." "Side" refers to a direction intersecting the fore-and-aft direction (left-right direction). Examples of the front camera 122 include imaging devices using a CCD (Charge Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor. Note that a work machine 100 according to other embodiments may not be equipped with a cab 121. Even in this case, the front camera 122 is installed at a position equivalent to the cab 121 so as to capture images of the front. Furthermore, in other embodiments, the front camera 122 may be configured with two or more cameras. Note that the front camera 122 is one example of an imaging device equipped on the work machine 100.

[0026] Figure 3 is an example of an image captured by an imaging device that is the front camera 122 of the work machine 100 according to an embodiment of the present disclosure. The front camera 122 captures an image of a range that captures a work target in front of the work implement 130 and the cab 121. In other words, the work target in front of the work implement 130 and the cab 121 is captured in image P1 captured by the front camera 122, as shown in Figure 3. Furthermore, because the cab 121 is provided on the left side of the work implement 130, part of the boom 131 is captured in the right portion of image P1.

[0027] 2 or 4, the work machine 100 is equipped with a work machine control device 126, a positioning device 123, a front camera 122, an inclination measuring device 124, a hydraulic device 125, a work machine attitude acquisition device 142, and a communication device 143. The work machine attitude acquisition device 142 includes a boom angle sensor 137, an arm angle sensor 138, and a bucket angle sensor 139.

[0028] The positioning device 123 acquires the position of the revolving unit 120 and the direction in which the revolving unit 120 faces. The positioning device 123 includes two receivers that receive positioning signals from artificial satellites that make up the Global Navigation Satellite System (GNSS). The two receivers are installed at different positions on the revolving unit 120. Based on the positioning signals received by the receivers, the positioning device 123 detects information indicating latitude, longitude, and altitude, as well as the position of a representative point of the revolving unit 120 in the site coordinate system (the origin of the excavator coordinate system). Using the positioning signals received by the two receivers, the positioning device 123 calculates the direction in which the revolving unit 120 faces as a relationship between the installation position of one receiver and the installation position of the other receiver. In other embodiments, the positioning device 123 may detect the direction in which the revolving unit 120 faces based on the measurement values ​​of a rotary encoder or an IMU (Inertial Measurement Unit).

[0029] The inclinometer 124 measures the acceleration and angular velocity of the revolving unit 120 and detects the attitude of the revolving unit 120 (for example, inclination angles such as roll angle, pitch angle, and yaw angle) based on the measurement results. The inclinometer 124 is installed, for example, on the underside of the revolving unit 120. The inclinometer 124 may be, for example, an inertial measurement unit (IMU). The inclinometer 124 may be an inclinometer that detects inclination angles without using acceleration and angular velocity. Note that the work machine 100 according to other embodiments does not need to be equipped with the positioning device 123. Note that FIG. 2 shows a state in which the roll angle θr, pitch angle θp, and yaw angle θy of the revolving unit 120 are all 0 degrees and the directions of the three axes X, Y, and Z of the global coordinate system are aligned with the directions of the three axes of the on-site coordinate system (local coordinate system). In this embodiment, the roll angle θr, pitch angle θp, and yaw angle θy of the revolving unit 120 are also referred to as the roll angle, pitch angle, and yaw angle of the work machine 100. The X direction is also referred to as the left-right direction, the Y direction as the front-rear direction, and the Z direction as the height direction.

[0030] The hydraulic system 125 includes a hydraulic oil tank, a hydraulic pump, and a flow control valve. The hydraulic pump is driven by power from an engine or electric motor (not shown), and supplies hydraulic oil to the boom cylinder 134, the arm cylinder 135, and the bucket cylinder 136 via the flow control valve. The flow control valve has a rod-shaped spool, and adjusts the flow rate of hydraulic oil supplied to the boom cylinder 134, the arm cylinder 135, and the bucket cylinder 136 depending on the position of the spool. The spool is driven based on a control command received from the work machine control device 126. In other words, the amount of hydraulic oil supplied to the boom cylinder 134, the arm cylinder 135, and the bucket cylinder 136 is controlled by the work machine control device 126.

[0031] The work machine control device 126 transmits information about the images captured by the forward camera 122, the position of the revolving unit 120 (including information indicating latitude, longitude, and altitude), orientation and tilt angle (attitude), and tilt angles of the boom 131, arm 132, and bucket 133 to the remote operator's cab 500 via the communication device 143. The information about the tilt angles of the boom 131, arm 132, and bucket 133 is also referred to as work machine attitude information. In this embodiment, information measured by various sensors equipped in the work machine 100 and transmitted by the work machine control device 126 is also referred to as vehicle body information. The work machine control device 126 receives operation signals from the remote operator's cab 500 via the communication device 143. The work machine control device 126 drives the work machine 130, the revolving unit 120, or the traveling unit 110 based on the received operation signals.

[0032] (Haulage Vehicle) In this embodiment, the haulage vehicle 200 is an off-road dump truck equipped with a vessel and is an example of a haulage vehicle. In this embodiment, the haulage vehicle 200 is operated under the control of the unmanned dump truck operation control device 300. The haulage vehicle 200 includes a dump truck control device 201, a positioning device 202, and a communication device 203. Like the positioning device 123, the positioning device 202 includes two receivers that receive positioning signals from artificial satellites that constitute the Global Navigation Satellite System (GNSS), and acquires the position of the haulage vehicle 200 and the direction in which the haulage vehicle 200 is heading. The communication device 203 transmits and receives predetermined signals with the unmanned dump truck operation control device 300 or with the communication device 560 of the remote cab 500. Under the control of the unmanned dump truck operation control device 300, the dump truck control device 201 moves the haulage vehicle 200 between predetermined locations (such as a loading location for excavated material, a loading / unloading location, and a waiting area). In addition, the dump truck control device 201 repeatedly provides the position (including information indicating latitude, longitude, and altitude) and direction information of the transport vehicle 200 at a predetermined period to the remote driver's cab 500, for example, via the unmanned dump truck operation control device 300.

[0033] The control device 550 is configured using one or more computers such as microcontrollers, and includes a control unit 551 as a functional configuration configured by a combination of hardware such as the computer and its peripheral devices, and software such as a program executed by the computer. The control unit 551 also includes an acquisition unit 552, a display unit 553, and a communication unit 554.

[0034] The communication unit 554 receives a video signal representing the captured image P1 captured by the front camera 122, the position (information indicating longitude, latitude, and altitude) of the revolving unit 120, the orientation and tilt angle, and information on the tilt angles (also referred to as attitude information) of the boom 131, arm 132, and bucket 133 from the work machine 100 via the communication device 560. The communication unit 554 also receives information on the position (information indicating longitude, latitude, and altitude) and orientation of the haulage vehicle 200 from the haulage vehicle 200 via the communication device 560 directly or via the unmanned dump truck operation control device 300 or the like. The communication unit 554 also transmits an operation signal corresponding to the operation of the operation device 540 to the work machine 100.

[0035] The acquisition unit 552 acquires, via the communication unit 554, a video signal representing the captured image P1 captured by the front camera 122, and measurement information such as the position (information indicating longitude, latitude, and altitude) and orientation of the revolving unit 120, attitude information of the work implement 130 representing the inclination angles of the boom 131, arm 132, and bucket 133, and attitude information indicating the roll angle and pitch angle of the work machine 100, as well as information on the position (information indicating longitude, latitude, and altitude) and orientation of the transport vehicle 200. Note that acquisition of information indicating the roll angle may be omitted.

[0036] Next, the display unit 553 will be described with reference to FIGS. 5 to 13. FIG. 5 is a schematic diagram showing an example of an image cut out from a captured image P1 according to an embodiment of the present disclosure. FIG. 6 is a schematic diagram showing an example of an image displayed by the first display device 520 according to an embodiment of the present disclosure. FIG. 7 is a schematic diagram showing an example of a composite image P2 according to an embodiment of the present disclosure. FIGS. 8 and 10 are schematic diagrams for explaining the composite image P2 according to an embodiment of the present disclosure. FIGS. 7, 9, and 11 to 13 are schematic diagrams showing composite images P2 (P2, P2a to P2d) according to an embodiment of the present disclosure.

[0037] The display unit 553 performs two processes: an image cropping process for dividing and displaying the captured image P1 on the multiple displays of the first display device 520, and a display process for superimposing a composite image P2 including a side image, which is an image depicting the side of the work machine 100, onto one of the cropped images and displaying it on the first display device 520.

[0038] 5, the display unit 553 cuts out from the captured image P1 a center image P11 to be displayed on the center display 521, a left image P12 to be displayed on the left display 522, a right image P13 to be displayed on the right display 523, an upper image P14 to be displayed on the upper display 524, and a lower image P15 to be displayed on the lower display 525. Note that if the first display device 520 is configured from a single display, it is not necessary to cut out the captured image P1.

[0039] In the display process, the display unit 553 generates composite images P2, P2a to P2d (hereinafter collectively referred to as composite image P2) illustrated in Figures 7, 9, and 11 to 13, and displays the composite image P2 superimposed on one of the cut-out images, for example, as shown in Figure 6, on the first display device 520. In the example shown in Figure 6, a central image P11 is displayed on the central display 521, a left image P12 is displayed on the left display 522, a right image P13 with the composite image P2 superimposed thereon is displayed on the right display 523, an upper image P14 is displayed on the upper display 524, and a lower image P15 is displayed on the lower display 525. In this embodiment, generating an image includes, for example, creating an image based on one or more images that simulate the entire work machine 100 or each part thereof, pasting an image into a specified drawing area (for example, a memory area set as a drawing object), drawing an image in raster or vector format, performing modification processing such as coloring, rotating an image, changing the size, orientation or position of an image, etc.

[0040] 7 , the composite image P2 includes a side image P21, a grid image P22, scale images P23 and P24, a reference image P25, a slope image P26, and an inclination angle image P27. The side image P21 is an image of the work machine 100 viewed from the side. The side image P21 is drawn at a position corresponding to the actual height with respect to the reference image P25. The display unit 553 generates the side image P21 representing the current attitude of the work machine 130 based on the information on the inclination angles of the boom 131, arm 132, and bucket 133 acquired by the acquisition unit 552. The display unit 553 also generates the side image P21 representing the current attitude of the work machine 100 based on the inclination angle of the revolving bed 120 acquired by the acquisition unit 552.

[0041] The grid image P22 includes a plurality of scale lines arranged in a grid pattern every 5 m. The scale image P23 includes numerical values ​​representing the distance in the height direction from a reference height (0 m). In the example shown in FIG. 7, the scale image P23 includes numerical values ​​ranging from -10 m to 25 m every 5 m. The scale image P24 includes numerical values ​​representing the distance in the front-to-rear direction, with the rotation axis CR ( FIG. 2 ) set to 0 m. In the example shown in FIG. 7, the scale image P24 includes numerical values ​​ranging from 0 m to 25 m every 5 m. The number of scales and numerical values ​​of the scale images P23 and P24 can be set arbitrarily.

[0042] The reference image P25 is an image that represents the reference point (0 m) in the height direction. In the example shown in Fig. 7, the reference image P25 is an image that represents line segments that are thicker than the scale lines of the lattice image P22. Note that the reference image P25 may be, for example, an image of the same thickness as the scale lines of the lattice image P22 but in a different color, or may be a line type other than a solid line, such as a dashed line.

[0043] The slope image P26 is an image that represents the slope ahead of the work machine 100. For example, as shown in FIG. 8 , when the work machine 100 attempts to climb the slope 2s of the scaffolding 2 and the cutting edge 133T comes into contact with the slope 2s, the display unit 553 generates a slope image P26 as shown in FIG. 7 . In the example shown in FIG. 7 , the slope image P26 includes an image that represents a line segment connecting a first position P101 based on the traveling body 110 and a second position P102 based on the posture information and based on the work implement 130. The first position P101 corresponds, for example, to the point where a vertical line (a line segment in the Z direction) VL drawn from the tip 100t of the traveling body 110 intersects with the ground contact surface 110s of the traveling body 110. The second position P102 is, for example, the position of the cutting edge 133T of the bucket 133.

[0044] The tilt angle image P27 represents the tilt angle θs of the slope image P26. In the example shown in Fig. 7, the tilt angle image P27 represents the characters "Slope: 23°." Note that the tilt angle θs can be, for example, the angle between a line segment connecting the first position P101 and the second position P102 and a horizontal line HL passing through the first position P101.

[0045] In this embodiment, when the display unit 553 causes the first display device 520 to display a composite image P2 including a side image P21 of the work machine 100, it includes in the composite image P2 a slope image P26 showing the slope 2s in front of the work machine 100 and an inclination angle image P27 showing the inclination angle θs of the slope 2s, based on predetermined measurement information measured by the work machine 100. Here, the measurement information is information measured by a predetermined sensor mounted on the work machine 100, and includes, for example, posture information showing the posture of the work implement 130 (information such as the inclination angles of the boom 131, arm 132, and bucket 133) and posture information of the work machine 100 (information indicating the pitch angle of the work machine 100, etc.). If the work machine 100 is equipped with a traveling body 110 and a work implement 130, the measurement information includes attitude information that represents the attitude of the work implement 130, and the slope image P26 includes an image that connects a first position P101 based on the traveling body 110 and a second position P102 based on the attitude information and based on the work implement 130. If the work implement 130 is equipped with a bucket 133, the second position P102 can be a predetermined position such as the cutting edge 133T of the bucket 133.

[0046] For example, when an operator attempts to climb a slope 2s as shown in Figure 8, the operator can check the slope angle θs of the slope 2s by contacting (or approaching) the cutting edge 133T with the slope 2s using the slope image P26 and the slope angle image P27.

[0047] The display unit 553 may display the slope image P26 and the tilt angle image P27 in response to a predetermined input operation by the operator, for example, when the cutting edge 133T comes into contact with (or approaches) the slope 2s, or may display them constantly, for example, regardless of the positional relationship with the slope 2s. In other words, the slope image P26 is an image that represents the tilt of the front of the work machine 100 based on the work implement 130.

[0048] Furthermore, in this embodiment, when the display unit 553 causes the first display device 520 to display a composite image P2 including a reference image P25 representing a reference in the height direction and a side image P21 drawn at a position corresponding to the actual height with respect to the reference image P25, the display unit 553 generates the composite image P2 by switching the height of the reference image P25 to either a height based on the work machine 100 or a predetermined height around the work machine 100. The actual height is a height measured by various sensors equipped on the work machine 100. Here, the predetermined height around the work machine 100 is a height based on another work machine, such as a transport vehicle 200, for example. For example, the predetermined height around the work machine 100 may be a height based on another work machine, such as a transport vehicle 200, that is located at a different height from the ground contact surface of the work machine 100. Furthermore, the display unit 533 may switch the height of the reference image P25 based on, for example, the positional relationship between the work machine 100 and the other work machine. Here, the positional relationship includes a relationship relating to absolute position and a relationship relating to relative position. Furthermore, an absolute positional relationship includes, for example, a relationship in which one or both of the work machine 100 and the other work machine are located in a predetermined location. Furthermore, a relative positional relationship includes, for example, a relationship in which at least one of the horizontal and vertical distances between the work machine 100 and the other work machine is within a predetermined range.

[0049] If the other work machine is a transport vehicle 200, the height based on the other work machine can correspond to the height of the track R200 of the transport vehicle 200. In addition to the above examples, the predetermined height of the surrounding area can be, for example, the height of the location where excavated materials are moved, or the target height of scaffolding when laying scaffolding. The height of the other work machine is not limited to a transport vehicle, and can be based on other cooperating work machines in general. The display unit 533 may also switch the height of the reference image P25 in response to a predetermined input operation by the operator, for example.

[0050] For example, the height of the reference image P25 included in the composite image P2 shown in Fig. 7 corresponds to the height of the ground contact surface 110s of the traveling body 110 of the work machine 100 shown in Fig. 8, for example. Furthermore, the height of the reference image P25 included in the composite image P2a shown in Fig. 9 corresponds to the height of the track R200 of the haulage vehicle 200 shown in Fig. 10, for example. Fig. 10 schematically shows the positional relationship between the work machine 100 and the haulage vehicle 200. In Fig. 10, the work machine 100 is placed on a scaffolding 2. Furthermore, the haulage vehicle 200 is located on the track R200 of the haulage vehicle 200. The track R200 may be a work site where the haulage vehicle 200 stops and where the work machine 100 loads cargo onto the haulage vehicle 200. In this case, the height of the track R200 of the haulage vehicle 200 is lower than the height of the ground contact surface 110s of the traveling body 110 of the work machine 100. 9 is located above the reference image P25 by the difference in height between the track R200 and the ground contact surface 110s. In this case, the operator can recognize the height of the track R200 of the transport vehicle 200 by checking the composite image P2a.

[0051] The display unit 553 can switch the height of the reference image P25 based on the positional relationship between the work machine 100 and the haulage vehicle 200, for example, as follows. That is, as shown in Fig. 10 , for example, the display unit 553 can use the height of the travel path R200 as the reference when a predetermined position P200 of the haulage vehicle 200 is located within a predetermined range (within a diameter D200) from the position of the haulage vehicle 200 at the time of loading (loading point), and can use a predetermined height of the work machine 100 (for example, the height of the contact surface 110s of the traveling body 110) as the reference when the predetermined position P200 of the haulage vehicle 200 is not located within the range (within the diameter D200). The position P200 corresponds to the installation position of the positioning device 202, for example.

[0052] 11 to 13 show other examples of the composite image P2 generated and displayed by the display unit 553. The composite image P2b shown in FIG. 11 differs from the composite image P2 shown in FIG. 7 in that the attitude of the work machine in the side image P21b is different. The composite image P2c shown in FIG. 12 is an example when the pitch angle θp of the work machine 100 is -15° (the pitch angle θp in FIG. 7 is 0°). The side image P21c is tilted by an amount corresponding to the pitch angle. In the composite image P2 shown in FIG. 7, the pitch angle θp of the work machine 100 is 0° and the tilt angle θs is 23°. In other words, the tilt angle θs of 38° shown in FIG. 12 includes the tilt angle θs of 23° when the pitch angle θp of the work machine 100 is 0°, and the pitch angle θp of -15° of the work machine 100. The composite image P2d shown in FIG. 13 is an example when the pitch angle θp of the work machine 100 is +15°. 12 and 13 is based on the midpoint height of the running body 110 of the work machine 100, but may be based on, for example, the height of the first position P101 or the height of the running path R200 of the transport vehicle 200.

[0053] 14 is a flowchart showing a display control method by the control device 550 of the remote control cab according to an embodiment of the present disclosure. When remote operation of the work machine 100 is started, the control device 550 executes the display control shown below at a predetermined cycle.

[0054] In the process shown in Figure 14, first, the acquisition unit 552 acquires position information of the haulage vehicle 200 (step S1). Next, the acquisition unit 552 acquires an image P1 captured by the front camera 122 of the work machine 100, position information, pitch angle information, and attitude information representing the attitude of the work implement 130 (step S2). Next, the display unit 553 cuts out a center image P11, a left image P12, a right image P13, an upper image P14, and a lower image P15 from the acquired image P1 (step S3). Next, the display unit 553 determines whether the position of the haulage vehicle 200 is within a predetermined range from the loading position (step S4). If the position of the haulage vehicle 200 is within the predetermined range from the loading position (step S4: YES), the display unit 553 determines the height of the travel path R200 of the haulage vehicle 200 as the height of the reference image P25 (step S5). On the other hand, if the position of the transport vehicle 200 is not within the predetermined range from the loading position (step S4: NO), the display unit 553 determines the predetermined height of the work machine 100 as the height of the reference image P25 (step S6).

[0055] Next, the display unit 553 calculates the inclination angle θs of the slope 2s (step S7). Next, the display unit 533 generates a reference image P25, a grid image P22, and scale images P23 and P24 (step S8). Next, the display unit 533 generates a side image P21 of the work machine 100 based on the height of the reference image P25, the position information of the work machine 100, the pitch angle, and posture information indicating the posture of the work implement 130 (step S9). Next, the display unit 533 generates a slope image P26 (step S10). Here, the slope image P26 is included in the composite image P2, which includes the reference image P25, the grid image P22, the scale images P23 and P24, and the side image P21. Next, the display unit 533 generates a slope angle image P27 (step S11). Here, the slope angle image P27 is included in the composite image P2. Next, the display unit 533 superimposes the composite image P2 on the extracted image and displays it on the first display device 520 (step S12), thereby completing the processing shown in Fig. 14. The composite image P2 may include only the side image 21 and the reference image P25, or may include only the side image 21 and the inclined surface image P26. The generation of the grid image P22, the scale images P23 and P24, and the inclination angle image P27 may be omitted.

[0056] (Actions and Effects) As described above, the control device 550 (display control system) according to the embodiment of the present disclosure is a display control system for the first display device 520 (display device), and includes an acquisition unit 552 that acquires measurement information measured by the work machine 100, and a display unit 553 that displays on the first display device 520 (display device) a composite image P2 that includes a side image P21 of the work machine 100 and a reference image P25 that represents a reference in the height direction, and the display unit 553 draws the side image P21 at a position that corresponds to the actual height with respect to the reference image P25, based on the measurement information. With this configuration, it is possible to supplement information in the height direction with the composite image P2. In other words, during loading work to load a load excavated by the work machine 100 onto the haulage vehicle 200 using the remote operation system 1, for example, even if the haulage vehicle 200 is positioned lower than the work machine 100, it is possible to recognize the height of the haulage vehicle 200 and the inclination of the work machine 100 from the composite image P2.

[0057] (Other Embodiments) One embodiment has been described above in detail with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes and the like are possible. For example, in the embodiment described above, the control device 550 generates the composite image, but this is not limited to this. For example, in other embodiments, the work machine control device 126 of the work machine 100 or an external server device may generate all or part of the composite image P2, and the control device 550 may receive information representing all or part of the composite image P2.

[0058] Furthermore, in the embodiment described above, the control device 550 calculates the cutting edge position of the bucket 133, but this is not limited to this in other embodiments. For example, the control device 550 according to other embodiments may calculate the bucket's lowest height, which is the distance from the ground surface to the bucket's lowest point, instead of the cutting edge position. The bucket's lowest point is the point on the bucket 133 that is closest to the ground surface. The control device 550 can identify the bucket's lowest point by storing in advance the shape of the bucket 133 based on the base end of the bucket 133, instead of or in addition to the length from the base end of the bucket 133 to the cutting edge. The cutting edge position and the bucket's lowest point are both examples of the position of the bucket 133.

[0059] Furthermore, in the above-described embodiment, the display control system is implemented in the remote operation system 1, but this is not limited to this. For example, in another embodiment, the control unit 551 may be applied to a radio control system that operates the work machine 100 via wireless communication from a position outside the work machine 100 where the work machine 100 can be seen. When a display control system is applied to a radio control system, the control device may be equipped with a display device.

[0060] Furthermore, instead of using the position information of the track R200 of the transport vehicle 200 as a reference, which is acquired by the transport vehicle 200 using the positioning device 202, for example, a LiDAR (Light Detection and Ranging), a stereo camera, or the like may be provided on the work machine 100 to acquire three-dimensional point cloud information of the track R200 (three-dimensional position information of a plurality of measurement points), and the height of the track R200 may be acquired based on the acquired three-dimensional point cloud information, with the work machine 100 as a reference. Alternatively, the position information of the transport vehicle 200 may use, for example, position information acquired by another transport vehicle 200.

[0061] Furthermore, instead of acquiring information about the slope 2s based on the attitude information of the work machine 130, a LiDAR, stereo camera, or the like may be provided on the work machine 100 to acquire three-dimensional point cloud information of the slope 2s, and the inclination angle of the slope 2s may be acquired based on the acquired three-dimensional point cloud information, with the work machine 100 as the reference. In this case, the three-dimensional point cloud information corresponds to the "predetermined measurement information measured by the work machine" according to the present disclosure.

[0062] Furthermore, the reference image P25 does not have to be an image representing a line segment. For example, the reference height may be represented by an icon such as an arrow or a triangle.

[0063] In addition, the predetermined height of the surrounding area may be obtained by recognizing the captured image P1 captured using the above-mentioned LiDAR, stereo camera, monocular camera, etc.

[0064] Furthermore, part or all of the programs executed by the computer in the above embodiments can be distributed via computer-readable recording media or communication lines.

[0065] (Additional Note) The display control system (control device 550) and the remote operation system 1 according to the present disclosure can be understood, for example, as follows.

[0066] (1) A display control system (control device 550) pertaining to a first aspect is a display control system for a display device (first display device 520), and comprises an acquisition unit 552 that acquires measurement information measured by a work machine 100, and a display unit 553 that displays on the display device a composite image P2 that includes a side image P21 of the work machine 100 and a reference image P25 that represents a reference in the height direction, the display unit 553 uses a predetermined height around the work machine 100 that is different from the ground surface of the work machine 100 as the reference for the reference image P25, and draws the side image P21 at a position that corresponds to the actual height with respect to the reference image P25 based on the measurement information. According to this aspect and the following aspects, height information can be supplemented by the composite image P2.

[0067] (2) A display control system (control device 550) according to a second aspect is the display control system of (1), in which the height of the reference image is switched to either a height based on the work machine or a predetermined height around the work machine to generate the composite image.

[0068] (3) A display control system (control device 550) according to a third aspect is the display control system of (2), in which the predetermined height of the periphery is a height based on other work machines.

[0069] (4) A display control system (control device 550) according to a fourth aspect is the display control system of (3), in which the switching is performed based on the positional relationship between the work machine and the other work machine.

[0070] (5) A display control system (control device 550) according to a fifth aspect is a display control system according to (3) or (4), in which the other work machine is a transport vehicle, and the height based on the transport vehicle corresponds to the height of the road on which the other transport vehicle runs.

[0071] (6) A display control system (control device 550) according to a sixth aspect is a display control system according to any one of (1) to (5), in which the measurement information includes a pitch angle of the work machine, and the side image is displayed at an angle based on the pitch angle.

[0072] (7) A display control system (control device 550) according to a seventh aspect is a display control system according to any one of (1) to (6), in which the work machine is equipped with a work implement, the measurement information includes posture information representing the posture of the work implement, and the side image is displayed based on the posture information.

[0073] (8) A display control system (control device 550) according to an eighth aspect is a display control system according to any one of (1) to (7), in which the work machine is equipped with an imaging device (forward camera 122), and the composite image is superimposed on an image captured by the imaging device and displayed on the display device.

[0074] (9) A remote operation system according to a ninth aspect comprises an acquisition unit that acquires measurement information measured by a work machine equipped with an imaging device, and a display unit that superimposes a composite image including a side image of the work machine and a reference image that represents a reference in the height direction on an image captured by the imaging device and displays the superimposed composite image on a display device provided at a remote location from the work machine, wherein the display unit uses a predetermined height around the work machine that is different from the ground surface of the work machine as a reference for the reference image, and draws the side image at a position that corresponds to the actual height based on the measurement information, the display device, and an operation device for the work machine provided at the remote location. According to this aspect, height information can be supplemented by the composite image.

[0075] According to each aspect of the present disclosure, information in the height direction can be supplemented by a composite image.

[0076] 1...Remote operation system 100...Work machine 130...Work equipment 120...Slewing body 110...Traveling body 131...Boom 132...Arm 133...Bucket 134...Boom cylinder 135...Arm cylinder 136...Bucket cylinder 137...Boom angle sensor 138...Arm angle sensor 139...Bucket angle sensor 121...Driver's cab 122...Forward camera 123...Positioning device 124...Inclination measuring instrument 125...Hydraulic device 126...Work machine control device 200...Transport vehicle 202...Positioning device 500...Remote driver's cab 510...Driver's seat 520...First display device 530...Second display device 540...Operation device 550...Control device 551...Control unit 552...Acquisition unit 553...Display unit 554: Communication unit; P2, P2a to P2d: Composite image; P21, P21a to P21d: Side image; P25: Reference image; P26: Slope image; P27: Tilt angle image

Claims

1. A display control system for a display device, comprising: an acquisition unit that acquires measurement information measured by a work machine; and a display unit that displays on the display device a composite image including a side image of the work machine and a reference image representing a height direction reference, wherein the display unit uses a predetermined height around the work machine, which is different from the ground surface of the work machine, as a reference for the reference image, and draws the side image at a position corresponding to the height of the reference image based on the measurement information.

2. A display control system as described in claim 1, wherein the height of the reference image is switched to either a height based on the work machine or a specified height in the vicinity of the work machine to generate the composite image.

3. A display control system according to claim 2, wherein the predetermined height of the surrounding area is a height based on other work machines.

4. A display control system according to claim 3, wherein the switching is performed based on a positional relationship between the work machine and the other work machine.

5. The display control system according to claim 3, wherein the other work machine is a transport vehicle, and the height based on the transport vehicle corresponds to the height of a road on which the other transport vehicle runs.

6. A display control system according to claim 5, wherein the measurement information includes a pitch angle of the work machine, and the side image is displayed at an inclination based on the pitch angle.

7. A display control system as claimed in claim 5 or 6, wherein the work machine is equipped with a work implement, the measurement information includes attitude information representing an attitude of the work implement, and the side image is displayed based on the attitude information.

8. A display control system according to claim 7, wherein the work machine is equipped with an imaging device, and the composite image is superimposed on an image captured by the imaging device and displayed on the display device.

9. A display control system comprising: an acquisition unit that acquires measurement information measured by a work machine equipped with an imaging device; and a display unit that superimposes a composite image including a side image of the work machine and a reference image representing a height direction reference on an image captured by the imaging device and displays the composite image on a display device provided at a remote location from the work machine, wherein the display unit uses a predetermined height around the work machine that is different from the ground surface of the work machine as a reference for the reference image, and draws the side image at a position corresponding to the height of the reference image based on the measurement information; the display device; and an operating device for the work machine provided at the remote location.

10. A display control method for a display device, comprising the steps of: acquiring measurement information measured by a work machine; displaying on the display device a composite image including a side image of the work machine and a reference image representing a height reference; and setting a predetermined height around the work machine, which is different from the ground surface of the work machine, as a reference for the reference image, and drawing the side image at a position corresponding to the height of the reference image based on the measurement information.

Images