Display control system, remote control system, and display control method

Abstract

The present invention provides a display control system, a remote control system, and a display control method that can appropriately supplement information. [Solution] The display control system is a display control system for a display device, which acquires measurement information measured by a work machine, generates a composite image including a reference image representing a height reference and a ground surface height image representing the height of the ground surface of the work machine, and displays it on the display device, using a predetermined height around the work machine, which is a different height from the ground surface of the work machine, as the reference for the reference image, and generates the composite image based on the measurement information such that the ground surface height image is positioned at a position corresponding to the height of the reference image.

Description

Technical Field

[0001] The present disclosure relates to a display control system, a remote operation system, and a display control method.

Background Art

[0002] Patent Document 1 describes a display control system that controls a display device in a remote cab for remotely operating a work vehicle (hereinafter referred to as a work machine). In the display control system described in Patent Document 1, a supplementary image, which is an image for supplementing the lack of sensation obtained when riding on the work machine, is displayed on the display device. Here, the supplementary image is, for example, a posture image including an image depicting the side surface of the work machine representing the current posture of the work machine.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When a work machine performs a loading operation onto a transport vehicle using a remote operation system, it may be difficult for an operator to recognize the height between the work machine and the transport vehicle and the inclination of the work machine on the camera image. That is, the lack of information such as the posture of the work machine becomes a problem. Also, in a manned work machine, the same problem exists for an operator who is not used to the operation.

[0005] An object of the present disclosure is to provide a display control system, a remote operation system, and a display control method capable of appropriately supplementing information in order to solve the above problems.

Means for Solving the Problems

[0006] The display control system of the present disclosure is a display control system for a display device, which acquires measurement information measured by a work machine, generates a composite image including a reference image representing a height reference and a ground surface height image representing the height of the ground surface of the work machine, and displays the composite image on the display device, using a predetermined height around the work machine, which is a different height from the ground surface of the work machine, as the reference for the reference image, and generates the composite image based on the measurement information such that the ground surface height image is positioned at a position corresponding to the height of the reference image.

[0007] The remote control system of this disclosure comprises the display control system, an operating device for the work machine located at a remote location, and the display device.

[0008] The display control method of the present disclosure is a display control method for a display device, wherein measurement information measured by a work machine is acquired, a composite image is generated including a reference image representing a height reference and a ground surface height image representing the height of the ground surface of the work machine, and the composite image is displayed on the display device, a predetermined height around the work machine which is a different height from the ground surface of the work machine is used as the reference for the reference image, and the composite image is generated such that the ground surface height image is positioned at a position corresponding to the height of the reference image based on the measurement information. [Effects of the Invention]

[0009] According to the display control system, remote control system, and display control method of this disclosure, information can be appropriately supplemented by a composite image. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic diagram showing an example configuration of a remote control system according to the present disclosure. [Figure 2] This is an external view of a work machine according to an embodiment of the present disclosure. [Figure 3] This is a schematic diagram showing an example of an image captured by an imaging device of a work machine according to an embodiment of the present disclosure. [Figure 4] This is a schematic block diagram showing an example configuration of a remote control system according to the present disclosure. [Figure 5] This is a schematic diagram showing an example of an image extracted from an image captured according to the embodiment of this disclosure. [Figure 6] This is a schematic diagram showing an example of an image displayed by a display device according to the embodiment of this disclosure. [Figure 7] This is a schematic diagram showing an example of a composite image according to the embodiments of this disclosure. [Figure 8] This is a schematic diagram illustrating an example of changes in a composite image according to the embodiments of this disclosure. [Figure 9] This is a schematic diagram illustrating a composite image according to an embodiment of the present disclosure. [Figure 10] This is a schematic diagram showing an example of a composite image according to the embodiments of this disclosure. [Figure 11] This is a schematic diagram illustrating an example of changes in a composite image according to the embodiments of this disclosure. [Figure 12] This is a schematic diagram showing an example of a composite image according to the embodiments of this disclosure. [Figure 13] This flowchart shows an example of the operation of a remote control system according to the embodiment of this disclosure. [Modes for carrying out the invention]

[0011] Embodiments of this disclosure will be described below with reference to the drawings. In each drawing, the same or corresponding components are given the same reference numerals, and their descriptions will be omitted as appropriate.

[0012] (Remote control system) FIG. 1 is a schematic diagram showing a configuration example of a remote operation system 1 according to an embodiment of the present disclosure. FIG. 2 is an external view of a work machine 100 according to an embodiment of the present disclosure. FIG. 4 is a schematic block diagram showing a configuration example of the remote operation system 1 according to an embodiment of the present disclosure. The remote operation system 1 is a remote operation system for remotely operating a work machine 100 including a work implement 130 having a working tool, and includes a work machine 100 that operates by remote operation, a transport vehicle 200 such as a dump truck that is a loading target for loading excavated materials excavated by a bucket 133 as a working tool, and a remote operation cab 500 for performing remote operation. The work machine 100 operates at a work site (for example, a mine, a quarry, etc.). The remote operation cab 500 is provided at a location away from the work machine 100 (a remote location of the work machine 100; for example, in a city, within the work site, etc.). The work machine 100 and the remote operation cab 500 are connected via communication means such as a mobile communication network and the Internet. The remote operation system 1 is a system for operating the work machine 100 using the remote operation cab 500. The transport vehicle 200 and the remote operation cab 500 are connected via communication means such as a mobile communication network and the Internet, or further connected via an unmanned dump truck operation control device and the like described later.

[0013] The work machine 100 operates according to an operation signal received from the remote operation cab 500. That is, no operator boards the work machine 100. The remote operation cab 500 receives an operation of the work machine 100 by an operator's operation and transmits an operation signal to the work machine 100. The operation signal will be described later.

[0014] (Remote operation cab) The remote operation cab 500 includes a driver's seat 510, a first display device 520, a second display device 530, an operation device 540, a control device 550, and a communication device 560. The first display device 520 is disposed in front of the driver's seat 510. The first display device 520 is positioned in front of the operator's eyes when the operator sits on the driver's seat 510. As shown in FIG. 1, the first display device 520 is composed of an arranged central display 521, left display 522, right display 523, upper display 524, and lower display 525. The left display 522 is provided on the left side of the central display 521. The right display 523 is provided on the right side of the central display 521. The upper display 524 is provided above the central display 521. The lower display 525 is provided below the central display 521. The first display device 520 divides and displays part or all of the captured image captured by the work machine 100 on five displays.

[0015] In addition, in other embodiments, the number of displays constituting the first display device 520 is not limited to this. For example, the first display device 520 may be composed of one display. Also, the first display device 520 may project the captured image onto a curved surface or a spherical surface by a projector or the like. Further, the first display device 520 is a configuration example of the "display device" according to the present disclosure.

[0016] The second display device 530 is disposed diagonally in front of the driver's seat 510. Vehicle body information (such as remaining fuel, engine water temperature, etc.) transmitted from the work machine 100 and notifications of abnormalities of the work machine 100 are displayed on the second display device 530. In addition, in other embodiments, the position of the second display device 530 does not have to be diagonally in front of the driver's seat 510 as long as it is a position visible to the operator. Also, the remote operation cab 500 according to other embodiments may not include the second display device 530. Preferably, the vehicle body information and abnormality notifications displayed on the second display device 530 may be displayed on the first display device 520.

[0017] The operating device 540 is located near the driver's seat 510. The operating device 540 is located within the operator's reach when the operator is seated in the driver's seat 510. The operating device 540 includes, for example, an electric lever and an electric pedal. When the operator operates the electric lever and electric pedal, the operating device 540 outputs work equipment operation signals, which are operation signals for the boom 131, arm 132, and bucket 133; a slewing operation signal for the slewing body 120; and a travel operation signal for the travel body 110. In this embodiment, the work equipment operation signals, slewing operation signals, and travel operation signals are collectively referred to as operation signals.

[0018] The control device 550 displays the captured images and vehicle body information received from the work machine 100 on the first display device 520. In other words, the control device 550 is an example of a display control system. The control device 550 also transmits the operation signals input to the operation device 540 to the work machine 100. The display control system may be a remote control room 500 including the control device 550, or it may further include the work machine control device 126 and the front camera 122 of the work machine 100. The display control system includes an acquisition unit 552 and a display unit 553. The acquisition unit 552 acquires measurement information such as height information measured by the work machine 100. The display unit 553 generates a composite image P2 including a reference image P25 representing the height reference and a ground surface height image P26 representing the height of the ground surface of the work machine 100, and displays it on the first display device 520. Furthermore, the display unit 553 uses a predetermined height around the work machine 100, which is a different height from the ground contact surface of the work machine 100, as the reference for the reference image P25, and generates a composite image P2 based on the measurement information so that the ground contact surface height image P26 is positioned at a position corresponding to the height of the reference image P25.

[0019] The communication device 560 transmits and receives video signals representing captured images, operation signals, vehicle information, and the like to and from the communication device 143 of the work machine 100.

[0020] (Work machinery) Figure 2 is an external view of a work machine 100 according to an embodiment of the present disclosure. The work machine 100 according to an embodiment of the present disclosure is a backhoe shovel, which is a type of hydraulic excavator. In addition, the work machine 100 according to other embodiments may be other hydraulic excavators such as face shovels or rope shovels, or other work machines such as wheel loaders or bulldozers. The work machine 100 comprises a hydraulically driven work machine 130, a slewing body 120 that supports the work machine 130, and a traveling body 110 that supports the slewing body 120. The slewing body 120 rotates around a slewing axis CR as the center of rotation.

[0021] The work machine 130 comprises a boom 131, an arm 132, and a bucket 133, which is a work tool. The work machine 130 is driven by the extension and retraction of a boom cylinder 134, an arm cylinder 135, and a bucket cylinder 136. The boom 131, arm 132, and bucket 133 are equipped with a boom angle sensor 137, an arm angle sensor 138, and a bucket angle sensor 139, respectively.

[0022] The base end of the boom 131 is attached to the slewing body 120 via a pin. The arm 132 connects the boom 131 and the bucket 133. The base end of the arm 132 is attached to the tip of the boom 131 via a pin. The bucket 133 includes a cutting edge 133T for excavating soil and other materials, and a container for collecting the excavated soil and other materials. The base end of the bucket 133 is attached to the tip of the arm 132 via a pin.

[0023] The boom cylinder 134 is a hydraulic cylinder for driving the boom 131. The base end of the boom cylinder 134 is attached to the slewing body 120. The tip end of the boom cylinder 134 is attached to the boom 131. The arm cylinder 135 is a hydraulic cylinder for driving the arm 132. The base end of the arm cylinder 135 is attached to the boom 131. The tip end of the arm cylinder 135 is attached to the arm 132. The bucket cylinder 136 is a hydraulic cylinder for driving the bucket 133. The base end of the bucket cylinder 136 is attached to the arm 132. The tip end of the bucket cylinder 136 is attached to the bucket 133 via a linkage mechanism.

[0024] The boom angle sensor 137 is attached, for example, to the boom 131 and detects the inclination angle of the boom 131. The arm angle sensor 138 is attached, for example, to the arm 132 and detects the inclination angle of the arm 132. The bucket angle sensor 139 is attached, for example, to the bucket 133 and detects the inclination angle of the bucket 133. The boom angle sensor 137, arm angle sensor 138, and bucket angle sensor 139 according to the embodiments of this disclosure detect the inclination angle with respect to the ground plane. However, angle sensors according to other embodiments are not limited to this and may detect the inclination angle with respect to other reference planes. For example, in other embodiments, the angle sensor may detect the relative rotation angle by potentiometers provided at the base ends of the boom 131, arm 132, and bucket 133, or it may detect the inclination angle by measuring the cylinder lengths of the boom cylinder 134, arm cylinder 135, and bucket cylinder 136 and converting the cylinder lengths into angles.

[0025] The slewing body 120 is equipped with a driver's cab 121. The driver's cab 121 is located to the left of the work machine 130. The driver's cab 121 is equipped with a forward camera 122. The forward camera 122 is installed at the front and top of the driver's cab 121. The forward camera 122 images the area in front of the driver's cab 121 through the windshield at the front of the driver's cab 121. Here, "forward" refers to the direction in which the work machine 130 is mounted on the slewing body 120, and "rear" refers to the opposite direction of "forward". "Side" refers to the direction (left and right) that intersects the front-rear direction. Examples of the forward camera 122 include imaging devices using a CCD (Charge Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor. Note that work machines 100 according to other embodiments do not need to be equipped with a driver's cab 121. In this case as well, the forward camera 122 is installed at a position corresponding to the driver's cab 121 to capture images of the area in front. In other embodiments, the forward camera 122 may consist of two or more cameras. The forward camera 122 is an example of an imaging device provided by the work machine 100.

[0026] Figure 3 shows an example of an image captured by an imaging device, which is a front camera 122, of a work machine 100 according to an embodiment of this disclosure. The front camera 122 captures the area in front of the work machine 130 and the operator's cab 121 that is the work object. In other words, as shown in Figure 3, the image P1 captured by the front camera 122 shows the work machine 130 and the work object in front of the operator's cab 121. Also, since the operator's cab 121 is located to the left of the work machine 130, a part of the boom 131 is captured in the right portion of the image P1.

[0027] As shown in Figure 2 or Figure 4, the work machine 100 includes a work machine control device 126, a positioning device 123, a front camera 122, an inclination measuring instrument 124, a hydraulic system 125, a work machine attitude acquisition device 142, and a communication device 143. The work machine attitude acquisition device 142 includes a boom angle sensor 137, an arm angle sensor 138, and a bucket angle sensor 139.

[0028] The positioning device 123 acquires the position of the rotating body 120 and the direction it is facing. The positioning device 123 includes two receivers that receive positioning signals from artificial satellites constituting the GNSS (Global Navigation Satellite System). The two receivers are installed at different locations on the rotating body 120. Based on the positioning signals received by the receivers, the positioning device 123 detects information indicating latitude, longitude, and altitude, as well as the position of a representative point of the rotating body 120 in the field coordinate system (the origin of the shovel coordinate system). Using the positioning signals received by the two receivers, the positioning device 123 calculates the direction the rotating body 120 is facing as the relationship between the installation position of one receiver and the installation position of the other receiver. In other embodiments, the positioning device 123 may detect the direction the rotating body 120 is facing based on measurements from a rotary encoder or an IMU (Inertial Measurement Unit).

[0029] The inclinometer 124 measures the acceleration and angular velocity of the slewing body 120 and detects the attitude of the slewing body 120 (e.g., inclinometer angles such as roll angle, pitch angle, and yaw angle) based on the measurement results. The inclinometer 124 is installed, for example, on the underside of the slewing body 120. The inclinometer 124 can be, for example, an inertial measuring device (IMU). The inclinometer 124 may also be an inclinometer that detects the inclinometer angle independently of acceleration and angular velocity. Note that the work machine 100 according to other embodiments may not be equipped with a positioning device 123. Figure 2 shows a state in which the roll angle θr, pitch angle θp, and yaw angle θy of the slewing body 120 are all 0 degrees, and the directions of the three axes XYZ of the global coordinate system coincide with the directions of the three axes of the field coordinate system (local coordinate system). In this embodiment, the roll angle θr, pitch angle θp, and yaw angle θy of the slewing body 120 are also referred to as the roll angle, pitch angle, and yaw angle of the work machine 100. The X direction is also referred to as the left-right direction, the Y direction as the front-back direction, and the Z direction as the height direction. The inclination angle of the slewing body 120 is also referred to as the inclination angle of the work machine 100.

[0030] The hydraulic system 125 comprises a hydraulic fluid tank, a hydraulic pump, and a flow control valve. The hydraulic pump is driven by an engine or electric motor (not shown) and supplies hydraulic fluid to the boom cylinder 134, arm cylinder 135, and bucket cylinder 136 via the flow control valve. The flow control valve has a rod-shaped spool, and the position of the spool adjusts the flow rate of hydraulic fluid supplied to the boom cylinder 134, arm cylinder 135, and bucket cylinder 136. The spool is driven based on control commands received from the work machine control device 126. In other words, the amount of hydraulic fluid supplied to the boom cylinder 134, arm cylinder 135, and bucket cylinder 136 is controlled by the work machine control device 126.

[0031] The work machine control device 126 transmits the captured image P1 taken by the forward camera 122, position information representing the position of the slewing body 120 (including information indicating latitude, longitude, and altitude), azimuth and inclination angle (attitude), and inclination angle information of the boom 131, as well as the arm 132 and bucket 133, to the remote control room 500 via the communication device 143. The information regarding the inclination angle of the boom 131, as well as the arm 132 and bucket 133, is also referred to as the attitude information of the work machine 130. In this embodiment, the information measured by various sensors on the work machine 100 and transmitted by the work machine control device 126 is also referred to as vehicle body information. The work machine control device 126 receives operation signals from the remote control room 500 via the communication device 143. Based on the received operation signals, the work machine control device 126 drives the work machine 130, the slewing body 120, and the traveling body 110.

[0032] (Transport vehicle) In this embodiment, the transport vehicle 200 is an off-road dump truck equipped with a vessel, and is an example of the transport vehicle 200. In this embodiment, the transport vehicle 200 is operated under the control of an unmanned dump truck operation control system 300. The transport vehicle 200 comprises a dump truck control device 201, a positioning device 202, and a communication device 203. The positioning device 202, like the positioning device 123, has two receivers that receive positioning signals from satellites constituting the GNSS, and acquires the position of the transport vehicle 200 and the direction in which the transport vehicle 200 is facing. The communication device 203 sends and receives predetermined signals to and from the unmanned dump truck operation control system 300, or to and from the communication device 560 of the remote driver's room 500. The dump truck control device 201 moves the transport vehicle 200 between predetermined points (such as a loading point for excavated material, an unloading point, a waiting area, etc.) under the control of the unmanned dump truck operation control system 300. Furthermore, the dump truck control device 201 repeatedly provides position information and orientation information representing the position of the transport vehicle 200 (including information indicating latitude, longitude, and altitude) at predetermined intervals to the remote control room 500, for example, via the unmanned dump truck operation control device 300.

[0033] (Control device for the remote control room) The control device 550 is configured using one or more computers such as microcontrollers, and includes a control unit 551 as a functional block composed of a combination of hardware such as the computer and its peripheral devices, and software such as programs executed by the computer. The control unit 551 also includes an acquisition unit 552, a display unit 553, and a communication unit 554.

[0034] The communication unit 554 receives from the work machine 100 via the communication device 560 a video signal representing the captured image P1 taken by the forward camera 122, position information representing the position of the slewing body 120 (information indicating longitude, latitude, and altitude), azimuth and inclination angle, and inclination angle information (also called attitude information) of the boom 131, arm 132, and bucket 133. The communication unit 554 also receives position information representing the position of the transport vehicle 200 (information indicating longitude, latitude, and altitude), and azimuth information from the transport vehicle 200 either directly via the communication device 560 or via the unmanned dump truck operation control device 300, etc. The communication unit 554 also transmits operation signals to the work machine 100 in accordance with the operation of the operation device 540.

[0035] The acquisition unit 552 acquires, via the communication unit 554, a video signal representing the captured image P1 taken by the forward camera 122, position information and orientation representing the position (longitude, latitude, and altitude) of the slewing body 120, attitude information of the work machine 130 representing the inclination angles of the boom 131, arm 132, and bucket 133, and measurement information such as the roll angle and pitch angle of the work machine 100. In addition to acquiring the measurement information, the acquisition unit 552 also acquires position information representing the position (longitude, latitude, and altitude) of the transport vehicle 200, as well as measurement information such as orientation. The acquisition of information indicating the roll angle may be omitted.

[0036] Next, the display unit 553 will be described with reference to Figures 5 to 12. Figure 5 is a schematic diagram showing an example of an image extracted from an captured image P1 according to an embodiment of this disclosure. Figure 6 is a schematic diagram showing an example of an image displayed by the first display device 520 according to an embodiment of this disclosure. Figures 7, 10, and 12 are schematic diagrams showing examples of composite images P2 (P2, P2c, and P2f) according to an embodiment of this disclosure. Figures 8 and 11 are schematic diagrams showing examples of changes in composite images P2 (P2a and P2b, and P2d and P2e) when the tilt angle changes.

[0037] The display unit 553 performs image cropping processing to divide and display the captured image P1 on multiple displays of the first display device 520. The display unit 553 also generates a composite image P2 which includes a reference image P25 representing the height reference of the work machine 100, a ground surface height image P26 representing the height of the ground surface of the work machine 100, a side image P21 representing the side of the work machine 100, and an inclination image P27 representing the inclination angle of the work machine 100, and then performs display processing to superimpose the composite image P2 onto one of the cropped images and display it on the first display device 520. In other words, the display unit 553 performs two processes: image cropping processing for dividing and displaying on displays, and display processing to generate a composite image P2 and superimpose the generated composite image P2 onto one of the cropped images and display it on the first display device 520.

[0038] In the image cropping process, the display unit 553 crops the captured image P1 to display the central image P11 on the central display 521, the left image P12 on the left display 522, the right image P13 on the right display 523, the upper image P14 on the upper display 524, and the lower image P15 on the lower display 525, respectively, as shown in Figure 5. Note that if the first display device 520 consists of a single display, cropping of the captured image P1 is not necessary.

[0039] In the display process, the display unit 553 generates composite images P2, P2a to P2f (hereinafter collectively referred to as composite image P2) as illustrated in Figures 7 to 8 and Figures 10 to 12, and displays them on the first display device 520 by superimposing one of the extracted images, for example, as shown in Figure 6. In the example shown in Figure 6, the central image P11 is displayed on the central display 521, the left image P12 is displayed on the left display 522, the right image P13 with composite image P2 superimposed is displayed on the right display 523, the upper image P14 is displayed on the upper display 524, and the lower image P15 is displayed on the lower display 525. In this embodiment, generating an image means, for example, creating an image based on one or more images that simulate the whole or individual parts of a pre-prepared work machine 100, pasting the image onto a predetermined drawing area (for example, a memory area set as a drawing object), drawing the image in raster or vector format, performing modification processing such as coloring, rotating the image, changing the size, orientation or position of the image, and so on.

[0040] The display unit 553 generates a composite image P2 including a reference image P25 representing a height reference and a ground surface height image P26 representing the height of the ground surface 110s of the work machine 100, and displays it on the first display device 520. At this time, the display unit 553 uses a predetermined height around the work machine 100, which is a different height from the ground surface 110s of the work machine 100, as the reference for the reference image P25. In addition, the display unit 553 generates the composite image P2 such that the ground surface height image P26 is positioned at a position corresponding to the height of the reference image P25, based on the measurement information acquired by the acquisition unit 552.

[0041] In this embodiment, the composite image P2, as shown in Figure 7, includes a side view image P21, a grid image P22, scale images P23 and P24, a reference image P25 (see Figure 10, etc.), a ground surface height image P26, an inclination image P27, an inclination angle image P28, and a height image P29. However, in the composite image P2 shown in Figure 7, the reference image P25 and the ground surface height image P26 are displayed overlapping, and for example, only the ground surface height image P26 is visible.

[0042] The side view image P21 is an image of the work machine 100 viewed from the side. The side view image P21 is drawn at a position corresponding to the actual height, based on the reference image P25. In other words, the side view image P21 is displayed at the height indicated by the scale position of the scale image P23, which corresponds to the actual height. The display unit 553 generates a side view image P21 representing the current posture of the work machine 130 based on the tilt angle information of the boom 131, arm 132, and bucket 133 acquired by the acquisition unit 552. The display unit 553 also generates a side view image P21 representing the current posture of the work machine 100 based on the tilt angle of the slewing body 120 acquired by the acquisition unit 552. For example, Figure 7 shows a side view image P21 representing the posture of the work machine 100 when the tilt angle of the slewing body 120 is 0°. Furthermore, Figures 8 and 11 show composite images P2a or P2d, which include a side image P21 representing the posture of the work machine 100 when the tilt angle of the slewing body 120 is 5°, and composite images P2b or P2e, which include a side image P21 representing the posture of the work machine 100 when the tilt angle of the slewing body 120 is -5°. The display unit 553 generates composite image P2 such that a portion of the side image P21 is located outside the rectangular frame line P221 contained within the grid image P22. In other words, a portion of the side image P21 may be displayed within the outer frame of the grid image P22. In this case (when the tilt angle is 0°), the display unit 553 displays the side image P21 such that the vertical line to the right of the rectangular frame line P221 contained within the grid image P22 coincides with the slewing axis CR (Figure 2). Furthermore, the display unit 553 generates a composite image P2 such that the grid image P22, which includes the frame lines P221, is positioned in the forward direction of the side image P21 (in the forward direction of the work machine 100). Alternatively, the side image P21 may be displayed so as to fit within the grid image P22.

[0043] In this embodiment, the grid image P22 includes an image representing a rectangular frame line P221 which is the outer frame of the grid image P22, and a plurality of scale lines P222 arranged in a grid pattern within the frame line P221 at 5m intervals. The scale image P23 includes numerical values ​​representing the distance in the height direction from the reference height (0m). In the example shown in Figure 7, the scale image P23 includes numerical values ​​from -10m to 25m at 5m intervals. The scale image P24 includes numerical values ​​representing the distance in the front-rear direction, with the pivot axis CR as 0m. For example, if the pivot body 120 is tilted as shown in Figure 8, the scale image P24 includes numerical values ​​representing the distance in the front-rear direction, with the intersection point C1 between the pivot axis CR and the ground contact surface 110s of the traveling body 110 as 0m. In the example shown in Figure 7, the scale image P24 includes numerical values ​​from 0m to 25m at 5m intervals. The number of scale divisions and numerical values ​​in the scale images P23 and P24 can be set arbitrarily. In this embodiment, the display unit 553 generates a composite image P2 based on a plurality of grid lines P222.

[0044] Reference image P25 is an image representing the height reference (0m). In the example shown in Figure 10, it is an image representing a line segment thicker than the scale line P222 of the grid image P22. Reference image P25 may also be an image of the same thickness as the scale line of the grid image P22 but a different color, or a line type other than a solid line, such as a dashed line. The reference for reference image P25 can be the height of the ground surface 110s of the work machine 100 (Figures 7 and 8), or a predetermined height around the work machine 100 that is a different height from the ground surface 110s of the work machine 100. A predetermined height around the work machine 100 is, for example, the height of the track R200 of the transport vehicle 200 (Figures 10 to 13).

[0045] The ground contact surface height image P26 is an image representing the height of the ground contact surface 110s of the work machine 100. In the example shown in Figure 7, the ground contact surface height image P26 is shown as a shaded white rectangle. In this case, the ground contact surface height image P26 is an image that extends in the forward direction of the work machine 100. The ground contact surface height image P26 is, for example, an image representing a straight line extending in the horizontal direction. Alternatively, if the work machine 100 includes a slewing body 120, the ground contact surface height image P26 is an image representing a straight line extending horizontally from the intersection point C1 between the pivot axis (slewing axis CR) of the slewing body 120 and the ground contact surface 110s. The display unit 553 generates a composite image P2 based on the measurement information acquired by the acquisition unit 552, such that the ground contact surface height image P26 is positioned at a position corresponding to the height of the reference image P25. Note that the reference image P25, the ground contact surface height image P26, and the inclination image P27 are each displayed in different ways. Here, "appearance" refers to the display color, thickness, whether it's a solid or dashed line, whether it flashes or not, etc.

[0046] The tilt image P27 represents the tilt angle (pitch angle) of the work machine 100. In this embodiment, the tilt image P27 represents a dashed straight line extending from intersection C1 in the forward direction of the work machine 100.

[0047] The tilt angle image P28 is an image that numerically represents the tilt angle of the tilt image P27. In the example shown in Figure 7, the tilt angle image P28 represents the string "Slope:0°". In other words, the tilt angle image P28 shown in Figure 7 indicates that the tilt angle of the rotating body 120 is 0°.

[0048] Height image P29 is an image that numerically represents the difference between the height of the reference image P25 and the height of the ground surface height image P26. In the example shown in Figure 7, height image P29 represents the string "Height: 0m".

[0049] In this embodiment, as described above, the display unit 553 uses a predetermined height around the work machine 100, which is a different height from the ground contact surface 110s of the work machine 100, as the reference for the reference image P25, and generates a composite image based on the measurement information such that the ground contact surface height image P26 is positioned at a position corresponding to the height of the reference image P25. Here, the measurement information is information measured by predetermined sensors mounted on the work machine 100 or predetermined sensors mounted on the transport vehicle 200, and includes, for example, position information and orientation representing the position of the slewing body 120 (information indicating longitude, latitude, and altitude), attitude information of the work machine 130 representing the inclination angles of the boom 131, arm 132, and bucket 133, attitude information representing the roll angle and pitch angle of the work machine 100, and position information and orientation information representing the position of the transport vehicle 200 (information indicating longitude, latitude, and altitude).

[0050] Furthermore, in this embodiment, when the display unit 553 displays a composite image P2 on the first display device 520, which includes a reference image P25 representing a height reference and a ground surface height image P26 drawn at a position corresponding to the actual height based on the reference image P25, the display unit 553 can generate the composite image P2 by switching the height of the reference image P25 to either the height based on the work machine 100 or a predetermined height around the work machine 100. The actual height is the height measured by various sensors provided by the work machine 100. Here, the predetermined height around the work machine 100 is, for example, the height based on other work machines such as a transport vehicle 200. For example, the predetermined height around the work machine 100 may be the height based on other work machines such as a transport vehicle 200 that are located at a different height from the ground surface of the work machine 100. The display unit 553 may also switch the height of the reference image P25 based, for example, on the positional relationship between the work machine 100 and other work machines. Here, the positional relationship includes relationships relating to absolute positions or relationships relating to relative positions. A relationship relating to absolute positions includes, for example, a relationship in which one or both of the work machine 100 and the other work machine are located in a predetermined location. A relationship relating to relative positions includes, for example, a relationship in which at least one of the horizontal or vertical distances between the work machine 100 and the other work machine is within a predetermined range.

[0051] Furthermore, if the other work machine is a transport vehicle 200, the height based on the other work machine can be made to correspond to the height of the transport vehicle 200's track R200. In addition, the predetermined height around the work machine 100 can be, in addition to the above example, the height of the moving location of excavated material, the target height of the scaffolding when laying scaffolding, etc. Furthermore, the other work machine is not limited to a transport vehicle, but can be any other work machine in general that is working in cooperation with the work machine. In addition, the display unit 553 may switch the height of the reference image P25 in response to a predetermined input operation by the operator, for example.

[0052] For example, the height of the reference image P25 (image hidden by the ground surface height image P26) included in the composite image P2 shown in Figure 7 corresponds to the height of the ground surface 110s of the traveling body 110 of the work machine 100 shown in Figure 9. Also, for example, the height of the reference image P25 included in the composite image P2c shown in Figure 10 corresponds to the height of the track R200 of the transport vehicle 200 shown in Figure 9. Figure 9 schematically shows the positional relationship between the work machine 100 and the transport vehicle 200. In Figure 9, the work machine 100 is placed on the scaffolding 2. The transport vehicle 200 is located on the track R200 of the transport vehicle 200. The track R200 may be a work site where the transport vehicle 200 stops and the work machine 100 loads cargo onto the transport vehicle 200. In this case, the height of the track R200 of the transport vehicle 200 is lower than the height of the ground surface 110s of the traveling body 110 of the work machine 100. Therefore, the position of the side view image P21 included in the composite image P2c shown in Figure 10 is located above the reference image P25 by the difference in height between the track R200 and the ground surface 110s. In this case, the operator can recognize the height of the track R200 of the transport vehicle 200 by checking the composite image P2c.

[0053] The display unit 553 can switch the height of the reference image P25 based on the positional relationship between the work machine 100 and the transport vehicle 200, for example, as follows. That is, for example, as shown in Figure 9, the display unit 553 uses the height of the track R200 as the reference when the predetermined position P200 of the transport vehicle 200 is located within a predetermined range (within a diameter D200) from the loading position (loading point) of the transport vehicle 200. Alternatively, if the predetermined position P200 of the transport vehicle 200 is not located within a predetermined range (within a diameter D200) from the loading position of the transport vehicle 200, the display unit 553 can use the predetermined height of the work machine 100 (for example, the height of the ground contact surface 110s of the traveling body 110) as the reference. Note that position P200 corresponds to, for example, the installation position of the positioning device 202.

[0054] As described above, Figures 8 and 10-12 show other examples of the composite image P2 shown in Figure 7, which is generated and displayed by the display unit 553. The composite images P2a and P2b shown in Figure 8 differ from the composite image P2 shown in Figure 7 in that the inclination angle of the work machine 100 in the side view image P21 is different. Figures 10-12 differ from Figures 7 and 8 in that the difference between the height of the reference image P25 and the height of the ground surface height image P26 is different. In the cases of Figures 7 and 8, the height difference is 0m, while in the cases of Figures 10-12, the height difference is 4.5m.

[0055] Furthermore, the shapes of the ground surface height image P26 and the inclination image P27 in Figure 12 differ from those shown in other drawings. In the example shown in Figure 12, when the display unit 553 generates a composite image P2 based on multiple scale lines P222, it generates the composite image P26 such that a part of the ground surface height image P26 (left end) P261 is located outside the frame line P221. Also, the display unit 553 generates the composite image P26 such that a part of the inclination image P27 (left end) P271 is located outside the frame line P221. In other words, the linear images representing the ground surface height image P26 and the inclination image P27 in Figure 12 are longer than those shown in other drawings. Displaying the linear images outside the frame line P221 makes it easier to grasp the values ​​indicated by the scale image P23.

[0056] (Example of operation in the remote control room) Figure 13 is a flowchart showing the display control method by the control device 550 of the remote control room according to an embodiment of the present disclosure. When remote operation of the work machine 100 is started, the control device 550 performs the following display control at predetermined intervals.

[0057] In the process shown in Figure 13, first, the acquisition unit 552 acquires measurement information, which is the position information of the transport vehicle 200 (step S1). Next, the acquisition unit 552 acquires measurement information such as the captured image P1 taken by the front camera 122 of the work machine 100, position information, pitch angle information, and posture information representing the posture of the work machine 130 (step S2). Next, the display unit 553 extracts the center image P11, left image P12, right image P13, upper image P14, and lower image P15 from the acquired captured image P1 (step S3). Next, the display unit 553 determines whether the position of the transport vehicle 200 is within a predetermined range from the loading position (step S4). If the position of the transport vehicle 200 is within a predetermined range from the loading position (step S4: YES), the display unit 553 determines the height of the transport vehicle 200's track R200 to be the height of the reference image P25 (step S5). On the other hand, if the position of the transport vehicle 200 is not within a predetermined range from the loading position (step S4: NO), the display unit 553 determines a predetermined height of the work machine 100 (for example, the height of the ground surface 110s) to be the height of the reference image P25 (step S6).

[0058] Next, the display unit 553 generates a grid image P22, scale images P23 and P24, and a reference image P25 (step S7). Next, the display unit 553 generates a side view image P21 of the work machine 100 based on the height of the reference image P25, the position information of the work machine 100, the pitch angle, and the posture information representing the posture of the work machine 130 (step S8). Next, the display unit 553 generates a ground surface height image P26 and an inclination image P27 based on the measurement information acquired by the acquisition unit 552 (step S9). Next, the display unit 553 generates an inclination angle image P28 and a height image P29 based on the height of the reference image P25, the position information of the work machine 100, and the pitch angle (step S10). Here, the reference image P25, grid image P22, scale images P23 and P24, side view image P21, ground surface height image P26, inclination image P27, inclination angle image P28, and height image P29 are included in the composite image P2. Next, the display unit 553 superimposes the composite image P2 onto the cropped image and displays it on the first display device 520 (step S11), thus ending the process shown in Figure 13.

[0059] (Effects / Actions) As described above, the control device 550 (display control system) according to the embodiment of this disclosure is a display control system for a first display device 520, comprising: an acquisition unit 552 that acquires measurement information measured by the work machine 100; and a display unit 553 that generates a composite image P2 including a reference image P25 representing a height reference and a ground surface height image P26 representing the height of the ground surface 110s of the work machine 100, and displays it on the first display device 520. The display unit 553 uses a predetermined height around the work machine 100, which is a different height from the ground surface 110s of the work machine 100, as the reference for the reference image P25, and generates the composite image P2 such that the ground surface height image P26 is positioned at a position corresponding to the height of the reference image P25 based on the measurement information. According to this embodiment and the following embodiments, at least height information can be appropriately supplemented using the composite image P2.

[0060] Furthermore, in this embodiment, the ground contact surface height image P26 is an image that extends in the forward direction of the work machine 100. In other words, the ground contact surface height image P26 is a straight line image that extends in the forward direction of the work machine 100. With this configuration, it is possible to easily grasp the height information in the forward direction of the work machine 100. That is, the ground contact surface height image P26 is a straight line image that extends in the forward direction of the work machine 100 and extends to the frame line P221, so it is possible to easily grasp the numerical value representing the distance from the height reference shown by the scale image P23.

[0061] Furthermore, in this embodiment, the composite image P2 includes a side view of the work machine 100. With this configuration, it is easier to grasp the height information of the work machine 100's position in relation to the reference image P25, which represents the height reference.

[0062] Furthermore, in this embodiment, the composite image P2 includes an inclination image P27 representing the inclination angle of the work machine 100. This configuration makes it easier to grasp the inclination angle of the work machine 100.

[0063] Furthermore, in this embodiment, the ground contact surface height image P26 is an image representing a straight line extending horizontally. With this configuration, the change in the ground contact surface height image P26 can be limited to horizontal movement, thus maintaining a consistent level of ease of recognition of the ground contact surface height image P26.

[0064] Furthermore, in this embodiment, the ground surface height image P26 is an image representing a straight line extending horizontally from the intersection point C1 between the pivot axis (pivot axis CR) of the pivot body 120 and the ground surface 110s. With this configuration, height information can be supplemented based on the pivot axis.

[0065] Furthermore, in this embodiment, the reference image P25, the ground surface height image P26, and the inclination image P27 are each displayed in different ways. This configuration allows for easy identification of each image.

[0066] Furthermore, in this embodiment, the composite image P2 includes an image (grid image P22) representing a rectangular frame P221 and a plurality of grid lines P222 arranged in a grid pattern within the frame. The display unit 553 generates the composite image P2 based on the plurality of grid lines P222, and generates the composite image P2 such that a part of the ground surface height image P26 (P261) is located outside the frame. This configuration makes it easy to quantitatively grasp the difference between the reference height and the ground surface.

[0067] Furthermore, in this embodiment, the display unit 553 generates a composite image P2 such that a portion of the side image P21 is located outside the frame line P221. This configuration allows for efficient use of the area within the frame line P221.

[0068] Furthermore, in this embodiment, the display unit 553 generates a composite image P2 such that the frame line P221 is positioned in the forward direction of the side image P21. With this configuration, height information can be easily captured for the forward direction of the work machine 100.

[0069] Furthermore, in this embodiment, the predetermined height around the work machine 100, which is used as the reference image P25, can be set to a height based on other work machines. This configuration allows for supplementing height information in relation to other work machines.

[0070] Furthermore, in this embodiment, the work machine 100 is equipped with an imaging device (forward camera 122), and the display unit 553 superimposes a composite image P2 onto the captured image P1 captured by the imaging device and displays it on the first display device 520. With this configuration, the first display device 520 can display the composite image P2 on the captured image P1.

[0071] Furthermore, in this embodiment, the height of the reference image P25 is switched based on the positional relationship between the work machine 100 and other work machines (transport vehicle 200). With this configuration, information can be appropriately supplemented, for example, in work performed in cooperation with other work machines.

[0072] Furthermore, in this embodiment, the height of the reference image P25 is switched according to a predetermined input operation by the operator. With this configuration, information can be appropriately supplemented according to the operator's needs, for example.

[0073] (Other embodiments) Although one embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to that described above, and various design changes are possible. For example, in the embodiment described above, the control device 550 generates the composite image, but this is not the case. For example, in other embodiments, the work machine control device 126 of the work machine 100 or an external server device may generate all or part of the composite image P2, and the control device 550 may receive information representing all or part of the composite image P2.

[0074] Furthermore, while the above-described embodiment includes a display control system in the remote control system 1, the embodiment is not limited to this. For example, in another embodiment, the control unit 551 may be applied to a radio control system that operates the work machine 100 wirelessly from a position outside the work machine 100 where the work machine 100 can be seen. When a display control system is applied to a radio control system, the control device 550 may include a display device. For example, in another embodiment, an operator sitting in the driver's cab 121 of the work machine 100 may view the composite image P2 on a display device provided by the control device 550.

[0075] Furthermore, instead of using the position information of the transport vehicle 200 on the R200 of the track R200 as a reference to the position information acquired by the transport vehicle 200 using the positioning device 202, for example, the work machine 100 may be equipped with LiDAR (Light Detection And Ranging), a stereo camera, etc. to acquire 3D point cloud information (3D position information of multiple measurement points) of the R200 of the track, and the height of the R200 of the track R200 may be acquired based on the acquired 3D point cloud information with the work machine 100 as the reference. Alternatively, the position information of the transport vehicle 200 may be, for example, the position information acquired by another transport vehicle 200.

[0076] Furthermore, images such as the reference image P25, the ground surface height image P26, and the inclination image P27 do not necessarily have to represent line segments. For example, icons such as arrows and triangles may be used to represent height and inclination angles.

[0077] Furthermore, the predetermined height of the surrounding area may be obtained by recognizing the captured image P1 obtained using the LiDAR, stereo camera, monocular camera, etc., as described above.

[0078] Furthermore, some or all of the program executed by the computer in the above embodiment can be distributed via a computer-readable recording medium or communication line.

[0079] (Note) The display control system (control device 550), remote control system 1, and display control method relating to this disclosure can be understood, for example, as follows.

[0080] (1) The display control system according to the first embodiment is a display control system for a display device that acquires measurement information measured by a work machine, generates a composite image including a reference image representing a height reference and a ground surface height image representing the height of the ground surface of the work machine, and displays it on the display device, using a predetermined height around the work machine, which is a different height from the ground surface of the work machine, as the reference for the reference image, and generates the composite image based on the measurement information such that the ground surface height image is positioned at a position corresponding to the height of the reference image. According to this embodiment and each of the following embodiments, information can be appropriately supplemented.

[0081] (2) The display control system according to the second embodiment is the display control system of (1), wherein the ground surface height image is an image that extends in the forward direction of the work machine.

[0082] (3) The display control system according to the third embodiment is the display control system of (1) or (2), wherein the composite image includes a side view of the work machine.

[0083] (4) The display control system according to the fourth embodiment is the display control system of (1) to (3), wherein the composite image includes an inclined image representing the inclination angle of the work machine.

[0084] (5) The display control system according to the fifth embodiment is the display control system of (1) to (4), wherein the ground surface height image is an image representing a straight line extending in the horizontal direction.

[0085] (6) The display control system according to the sixth embodiment is the display control system according to (1) to (5), wherein the work machine includes a slewing body, and the ground surface height image is an image representing a straight line extending horizontally from the intersection of the pivot axis of the slewing body and the ground surface.

[0086] (7) The display control system according to the seventh embodiment is the display control system of (1) to (6), wherein the reference image, the ground surface height image, and the inclination image are each displayed in different manners.

[0087] (8) The eighth display control system is the display control system of (1) to (7), wherein the composite image includes an image representing a rectangular frame and a plurality of grid lines arranged in a grid within the frame, the composite image is generated based on the plurality of grid lines, and the composite image is generated such that a part of the ground surface height image is located outside the frame.

[0088] (9) The display control system according to the ninth aspect is the display control system of (1) to (8), wherein the composite image is generated such that a part of the side image is located outside the frame line.

[0089] (10) The display control system according to the tenth embodiment is the display control system of (1) to (9), wherein the display unit generates the composite image such that the frame line is located in the forward direction of the side image.

[0090] (11) The display control system according to the 11th embodiment is the display control system of (1) to (10), wherein the predetermined height of the periphery is a height based on other work machines.

[0091] (12) A display control system according to the twelfth embodiment is the display control system according to (1) to (11), wherein the work machine is equipped with an imaging device, and the composite image is superimposed on the image captured by the imaging device and displayed on the display device.

[0092] (13) The display control system according to the 13th embodiment is the display control system according to (1) to (12), wherein the height of the reference image is switched based on the positional relationship between the work machine and the other work machine.

[0093] (14) The display control system according to the 14th embodiment is the display control system according to (1) to (13), wherein the height of the reference image is switched in accordance with a predetermined input operation of the operator.

[0094] (15) A remote control system according to the 15th embodiment comprises the display control system of (1) to (14), an operating device for the work machine installed at a remote location, and the display device.

[0095] . (16) A display control method according to the 16th embodiment is a display control method for a display device, wherein measurement information measured by a work machine is acquired, a composite image is generated including a reference image representing a height reference and a ground surface height image representing the height of the ground surface of the work machine, and the composite image is displayed on the display device, a predetermined height around the work machine which is a different height from the ground surface of the work machine is used as the reference for the reference image, and the composite image is generated based on the measurement information such that the ground surface height image is positioned at a position corresponding to the height of the reference image. [Explanation of Symbols]

[0096] 1…Remote control system 100…Work machine 130…Work machine 120…Slewing body 110…Traveling body 131…Boom 132…Arm 133…Bucket 134…Boom cylinder 135…Arm cylinder 136…Bucket cylinder 137…Boom angle sensor 138…Arm angle sensor 139…Bucket angle sensor 121…Operator's cab 122…Forward camera 123…Positioning device 124…Incline measuring device 125…Hydraulic system 126…Work machine control device 200…Transport vehicle 202…Positioning device 500…Remote control cab 510…Driver's seat 520…First display device 530…Second display device 540…Operating device 550…Control device 551…Control unit 552…Acquisition unit 553…Display unit 554...Communication section, P2, P2a~P2f...Composite image, P21...Side view image, P22...Grid image, P25...Reference image, P26...Ground surface height image, P27...Inclined image

Claims

1. A display control system for a display device, We acquire measurement information measured by the work machine, A composite image is generated and displayed on the display device, which includes a reference image representing a height reference and a ground surface height image representing the height of the ground surface of the work machine. A predetermined height around the work machine, which is different from the ground contact surface of the work machine, is used as the reference for the reference image, and the composite image is generated based on the measurement information so that the ground contact surface height image is positioned at a height corresponding to the height of the reference image. Display control system.

2. The aforementioned ground surface height image is an image that extends in the forward direction of the work machine. The display control system according to claim 1.

3. The composite image includes a side view of the work machine. The display control system according to claim 2.

4. The composite image includes an inclined image representing the inclination angle of the work machine. The display control system according to claim 3.

5. The aforementioned image of the ground surface height represents a straight line extending horizontally. The display control system according to claim 1.

6. The aforementioned work machine includes a rotating body, The aforementioned image of the ground contact surface height represents a straight line extending horizontally from the intersection point between the pivot axis of the pivoting body and the ground contact surface. The display control system according to claim 5.

7. The reference image, the ground surface height image, and the inclination image are each displayed in different manner. The display control system according to claim 4.

8. The composite image includes an image representing a rectangular frame and a plurality of grid lines arranged in a grid pattern within the frame. The composite image is generated based on the plurality of grid lines, and the composite image is generated such that a portion of the ground surface height image is located outside the frame line. The display control system according to claim 7.

9. The composite image is generated such that a portion of the side view image is located outside the frame line. The display control system according to claim 8.

10. The composite image is generated such that the frame line is positioned in the forward direction of the side image. The display control system according to claim 9.

11. The aforementioned predetermined height in the surrounding area is a height based on other working machinery. The display control system according to claim 10.

12. The aforementioned work machine is equipped with an imaging device, The composite image is superimposed on the captured image captured by the imaging device and displayed on the display device. The display control system according to claim 11.

13. The height of the reference image is switched based on the positional relationship between the aforementioned work machine and the other work machine. The display control system according to claim 12.

14. The height of the reference image is switched according to a predetermined input operation by the operator. The display control system according to claim 13.

15. A display control system according to any one of claims 1 to 14, An operating device for the work machine located in a remote location, The device comprises the aforementioned display device. Remote control system.

16. A method for controlling the display of a display device, We acquire measurement information measured by the work machine, A composite image is generated and displayed on the display device, which includes a reference image representing a height reference and a ground surface height image representing the height of the ground surface of the work machine. A predetermined height around the work machine, which is different from the ground contact surface of the work machine, is used as the reference for the reference image, and based on the measurement information, the composite image is generated such that the ground contact surface height image is positioned at a position corresponding to the height of the reference image. Display control method.

Images