Display control system, display control method, and remote operation system

Abstract

A display control system according to the present invention, which is a display control system for a display device, is provided with a display unit that displays, on the display device, an image captured by an imaging device provided in a work machine. The display unit displays at least one of a roll angle image and a pitch angle image on the display device so as to be superimposed on the captured image, the roll angle image being rotated on the basis of the roll angle of the work machine about a prescribed reference point serving as the center of rotation, and the pitch angle image being moved in parallel in the vertical direction on the basis of the pitch angle of the work machine.

Description

Display control system, display control method, and remote operation system

[0001] This application claims priority to Japanese Patent Application No. 2023-199217, filed on November 24, 2023, the contents of which are incorporated herein by reference.

[0002] Patent Document 1 describes a remote control system that allows an operator to intuitively recognize the inclination of a work vehicle (hereinafter referred to as a work machine) when remotely operating the work machine. The remote control system described in Patent Document 1 displays a posture image that represents at least one of the roll angle and pitch angle of the work machine. Note that the posture image represents the roll angle or pitch angle by the position of a bubble image included in an image that resembles, for example, an air bubble level.

[0003] Japanese Patent Application Laid-Open No. 2020-72401

[0004] According to the remote control system described in Patent Document 1, an attitude image is displayed on a display device facing the driver's seat, allowing the operator to intuitively recognize the inclination of the work machine. However, depending on the shape and position of the attitude image, there is a problem in that it may be difficult to move the operator's line of sight to the attitude image and instantly recognize the amount of inclination of the work machine while gazing at the captured image during work.

[0005] The present disclosure has been made in consideration of the above circumstances, and aims to provide a display control system, a display control method, and a remote operation system that make it easier to recognize the inclination of a work machine.

[0006] The display control system according to the present disclosure is a display control system for a display device, and includes a display unit that displays an image captured by an imaging device provided in a work machine on the display device, and the display unit displays at least one of a roll angle image that rotates based on the roll angle of the work machine around a predetermined reference point as the center of rotation, and a pitch angle image that moves parallel in the vertical direction based on the pitch angle of the work machine, superimposed on the captured image on the display device.

[0007] A remote operation system according to the present disclosure includes the above-described display control system, and a display device and an operation device that are remotely provided.

[0008] A display control method for a display device according to the present disclosure is a display control method for a display device, in which, when an image captured by an imaging device equipped in a work machine is displayed on the display device, at least one of a roll angle image that rotates based on the roll angle of the work machine around a predetermined reference point as the center of rotation, and a pitch angle image that moves parallel in the vertical direction based on the pitch angle of the work machine is superimposed on the captured image and displayed on the display device.

[0009] The display control system, display control method, and remote operation system disclosed herein can make it easier to recognize the inclination of a work machine.

[0010] FIG. 1 is a schematic diagram showing an example configuration of a remote operation system according to an embodiment of the present disclosure. FIG. 2 is an external view of a work machine according to an embodiment of the present disclosure. FIG. 3 is a schematic diagram showing an example of a captured image captured by an imaging device of a work machine according to an embodiment of the present disclosure. FIG. 4 is a schematic block diagram showing an example configuration of a control device of a remote cab according to an embodiment of the present disclosure. FIG. 5 is a schematic diagram showing an example of an image cut out from a captured image according to an embodiment of the present disclosure. FIG. 6 is a schematic diagram showing an example of an image superimposed on a captured image according to an embodiment of the present disclosure. FIG. 7 is a schematic diagram showing an example of an image displayed by a display device according to an embodiment of the present disclosure. FIG. 8 is a flowchart showing an example operation of a control device of a remote cab according to an embodiment of the present disclosure. FIG. 9 is a schematic diagram showing another example of an image superimposed on a captured image according to an embodiment of the present disclosure.

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In each drawing, the same or corresponding components are designated by the same reference numerals, and the description thereof will be omitted as appropriate.

[0012] (Remote operation system) FIG. 1 is a schematic diagram showing an example configuration of a remote operation system according to an embodiment of the present disclosure. The remote operation system 1 is a remote operation system for remotely operating a work machine, and includes a work machine 100 that operates by remote control, and a remote operator's cab 500 for performing the remote operation. The work machine 100 operates at a work site (e.g., a mine or a quarry). The remote operator's cab 500 is provided at a location remote from the work machine 100 (e.g., in a city or within the work site). The work machine 100 and the remote operator's cab 500 are connected via communication means such as a mobile communication network or the Internet. The remote operation system 1 is a system for operating the work machine 100 using the remote operator's cab 500.

[0013] The work machine 100 operates in accordance with operation signals received from the remote operator's cab 500. In other words, no operator is on board the work machine 100. The remote operator's cab 500 accepts operations of the work machine 100 through operations by the operator, and transmits operation signals to the work machine 100. The operation signals will be described later.

[0014] (Remote operator's cab) The remote operator's cab 500 includes a driver's seat 510, a first display device 520, a second display device 530, an operation device 540, a control device 550, and a communication device 560. The first display device 520 is disposed in front of the driver's seat 510. The first display device 520 is located in front of the operator when the operator sits in the driver's seat 510. As shown in FIG. 1 , the first display device 520 is composed of a center display 521, a left display 522, a right display 523, an upper display 524, and a lower display 525, which are arranged side by side. The left display 522 is provided to the left of the center display 521. The right display 523 is provided to the right of the center display 521. The upper display 524 is provided above the center display 521. The lower display 525 is provided below the center display 521. The first display device 520 divides and displays, for example, part or all of an image captured by the work machine 100 on five displays.

[0015] In other embodiments, the number of displays constituting first display device 520 is not limited to this. For example, first display device 520 may be constituted by a single display. Furthermore, first display device 520 may project an image onto a curved or spherical surface using a projector or the like. Furthermore, first display device 520 is one configuration example of a "display device" according to the present disclosure.

[0016] The second display device 530 is disposed diagonally forward of the driver's seat 510. The second display device 530 displays vehicle body information (inclination angle, remaining fuel level, engine water temperature, etc.) transmitted from the work machine 100, notifications of abnormalities in the work machine 100, and the like. Note that in other embodiments, the position of the second display device 530 does not have to be diagonally forward of the driver's seat 510, as long as it is in a position that is visible to the operator. Furthermore, the remote driver's cab 500 according to other embodiments does not need to be equipped with the second display device 530, and preferably the vehicle body information and notifications of abnormalities that are displayed on the second display device 530 may be displayed on the first display device 520.

[0017] The operation device 540 is disposed near the driver's seat 510. The operation device 540 is located within an operable range of the operator when the operator sits in the driver's seat 510. The operation device 540 includes, for example, an electric lever and an electric pedal. When the operator operates the electric lever and the electric pedal, the operation device 540 outputs a work implement operation signal, which is an operation signal for the boom 131, the arm 132, and the bucket 133, a swing operation signal for the rotating body 120, and a travel operation signal for the traveling body 110. In this embodiment, the work implement operation signal, swing operation signal, and travel operation signal are collectively referred to as operation signals.

[0018] The control device 550 displays the captured images and vehicle body information received from the work machine 100 on the first display device 520. In other words, the control device 550 is an example of a display control system. The display control system may also include an imaging device for acquiring captured images, an inclination measuring device 124 for acquiring the inclination angle of the work machine 100, and a display device that generates an inclination angle display image P101 (described below) based on the acquired inclination angle, and displays an image in which the inclination angle display image P101 is superimposed on the captured image using a display unit 553 that superimposes the generated inclination angle display image P101 on the captured image. The control device 550 also transmits operation signals input to the operation device 540 to the work machine 100. The display control system may be a remote cab 500 that includes the control device 550, or may further include a work machine control device 126 and a forward camera 122 of the work machine 100.

[0019] The communication device 560 transmits and receives operation signals, signals representing vehicle body information, captured images, etc. to and from the communication device 143 of the work machine 100 .

[0020] (Working Machine) Figure 2 is an external view of a working machine according to an embodiment of the present disclosure. The working machine 100 according to an embodiment of the present disclosure is a backhoe excavator, which is a type of hydraulic excavator. Note that the working machine 100 according to other embodiments may be a hydraulic excavator other than a backhoe excavator, such as a face excavator or a rope excavator, or a working machine other than a hydraulic excavator, such as a wheel loader or a bulldozer. The working machine 100 includes a hydraulically driven working implement 130, a rotating unit 120 that supports the working implement 130, and a traveling unit 110 that supports the rotating unit 120.

[0021] The work machine 130 includes a boom 131, an arm 132, and a bucket 133. The work machine 130 is driven by extension and contraction of a boom cylinder 134, an arm cylinder 135, and a bucket cylinder 136. A boom angle sensor 137, an arm angle sensor 138, and a bucket angle sensor 139 are attached to the boom 131, the arm 132, and the bucket 133, respectively.

[0022] The base end of the boom 131 is attached to the rotating body 120 via a pin. The arm 132 connects the boom 131 and the bucket 133. The base end of the arm 132 is attached to the tip of the boom 131 via a pin. The bucket 133 is equipped with a cutting edge 133T for excavating earth and sand, and a container for storing the excavated earth and sand. The base end of the bucket 133 is attached to the tip of the arm 132 via a pin.

[0023] The boom cylinder 134 is a hydraulic cylinder for driving the boom 131. A base end of the boom cylinder 134 is attached to the rotating body 120. A tip end of the boom cylinder 134 is attached to the boom 131. The arm cylinder 135 is a hydraulic cylinder for driving the arm 132. A base end of the arm cylinder 135 is attached to the boom 131. A tip end of the arm cylinder 135 is attached to the arm 132. The bucket cylinder 136 is a hydraulic cylinder for driving the bucket 133. A base end of the bucket cylinder 136 is attached to the arm 132. A tip end of the bucket cylinder 136 is attached to the bucket 133.

[0024] The boom angle sensor 137 is attached to, for example, the boom 131 and detects the inclination angle of the boom 131. The arm angle sensor 138 is attached to, for example, the arm 132 and detects the inclination angle of the arm 132. The bucket angle sensor 139 is attached to, for example, the bucket 133 and detects the inclination angle of the bucket 133. The boom angle sensor 137, the arm angle sensor 138, and the bucket angle sensor 139 according to embodiments of the present disclosure detect the inclination angle with respect to the ground plane. Note that angle sensors according to other embodiments are not limited to this, and may detect the inclination angle with respect to another reference plane. For example, in other embodiments, the angle sensors may detect the relative rotation angle using potentiometers provided at the base ends of the boom 131, the arm 132, and the bucket 133, or may detect the inclination angle by measuring the cylinder lengths of the boom cylinder 134, the arm cylinder 135, and the bucket cylinder 136 and converting the cylinder lengths into angles.

[0025] The revolving unit 120 is provided with a cab 121. The cab 121 is provided on the left side of the work machine 130. The cab 121 is provided with a front camera 122. The front camera 122 is installed at the front and upper part of the cab 121. The front camera 122 captures images in front of the cab 121 through a windshield in front of the cab 121. Here, "forward" refers to the direction in which the work machine 130 is mounted on the revolving unit 120, and "rearward" refers to the opposite direction of "forward." "Side" refers to a direction intersecting the fore-and-aft direction (left-right direction). Examples of the front camera 122 include imaging devices using a CCD (Charge Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor. Note that a work machine 100 according to other embodiments may not be provided with a cab 121. Even in this case, the front camera 122 is installed at a position corresponding to the cab 121 so as to capture images of the front. Furthermore, in other embodiments, the front camera 122 may be configured with two or more cameras.

[0026] Figure 3 is an example of an image captured by an imaging device that is the front camera 122 of the work machine 100 according to an embodiment of the present disclosure. The front camera 122 captures an image of a range that captures a work target in front of the work implement 130 and the cab 121. In other words, the work target in front of the work implement 130 and the cab 121 is captured in image P1 captured by the front camera 122, as shown in Figure 3. Furthermore, because the cab 121 is provided on the left side of the work implement 130, part of the boom 131 is captured in the right portion of image P1.

[0027] Fig. 4 is a schematic block diagram showing an example configuration of a remote operation system 1 according to an embodiment of the present disclosure. As shown in Fig. 2 or 4 , the work machine 100 is equipped with a work machine control device 126, a positioning device 123, a front camera 122, an inclination measuring device 124, a hydraulic device 125, a work machine attitude acquisition device 142, and a communication device 143. The work machine attitude acquisition device 142 includes a boom angle sensor 137, an arm angle sensor 138, and a bucket angle sensor 139.

[0028] The positioning device 123 acquires the position of the revolving unit 120 and the direction in which the revolving unit 120 faces. The positioning device 123 includes two receivers that receive positioning signals from artificial satellites that make up the Global Navigation Satellite System (GNSS). The two receivers are installed at different positions on the revolving unit 120. Based on the positioning signals received by the receivers, the positioning device 123 detects information indicating latitude, longitude, and altitude, as well as the position of a representative point of the revolving unit 120 in the site coordinate system (the origin of the excavator coordinate system). Using the positioning signals received by the two receivers, the positioning device 123 calculates the direction in which the revolving unit 120 faces as a relationship between the installation position of one receiver and the installation position of the other receiver. In other embodiments, the positioning device 123 may detect the direction in which the revolving unit 120 faces based on the measurement values ​​of a rotary encoder or an IMU (Inertial Measurement Unit).

[0029] The inclinometer 124 measures the acceleration and angular velocity of the revolving unit 120 and detects the attitude of the revolving unit 120 (for example, inclination angles such as roll angle, pitch angle, and yaw angle) based on the measurement results. The inclinometer 124 is installed, for example, on the underside of the revolving unit 120. The inclinometer 124 may be, for example, an inertial measurement unit (IMU). The inclinometer 124 may be an inclinometer that detects inclination angles without using acceleration and angular velocity. Note that the work machine 100 according to other embodiments does not need to be equipped with the positioning device 123. Note that FIG. 2 shows a state in which the roll angle θr, pitch angle θp, and yaw angle θy of the revolving unit 120 are all 0 degrees and the directions of the three axes X, Y, and Z of the global coordinate system are aligned with the directions of the three axes of the on-site coordinate system (local coordinate system). In addition, in this embodiment, the roll angle θr, pitch angle θp, and yaw angle θy of the revolving unit 120 are also referred to as the roll angle, pitch angle, and yaw angle of the work machine 100.

[0030] The hydraulic system 125 includes a hydraulic oil tank, a hydraulic pump, and a flow control valve. The hydraulic pump is driven by power from an engine or electric motor (not shown), and supplies hydraulic oil to the boom cylinder 134, the arm cylinder 135, and the bucket cylinder 136 via the flow control valve. The flow control valve has a rod-shaped spool, and adjusts the flow rate of hydraulic oil supplied to the boom cylinder 134, the arm cylinder 135, and the bucket cylinder 136 depending on the position of the spool. The spool is driven based on a control command received from the work machine control device 126. In other words, the amount of hydraulic oil supplied to the boom cylinder 134, the arm cylinder 135, and the bucket cylinder 136 is controlled by the work machine control device 126.

[0031] The work machine control device 126 transmits the images captured by the front camera 122, the rotation speed, position (including information indicating latitude, longitude, and altitude), orientation, and tilt angle (attitude) of the revolving unit 120 to the remote operator's cab 500 via the communication device 143. In this embodiment, information measured by various sensors equipped in the work machine 100 and transmitted by the work machine control device 126 is also referred to as vehicle body information. The work machine control device 126 receives operation signals from the remote operator's cab 500 via the communication device 143. The work machine control device 126 drives the work implement 130, the revolving unit 120, or the traveling unit 110 based on the received operation signals.

[0032] The control device 550 is configured using one or more computers such as microcontrollers, and includes a control unit 551 as a functional configuration configured by a combination of hardware such as the computer and its peripheral devices, and software such as a program executed by the computer. The control unit 551 also includes an acquisition unit 552, a display unit 553, and a communication unit 554.

[0033] The communication unit 554 receives a video signal representing the image captured by the front camera 122, the rotation speed, position (information indicating longitude, latitude, and altitude), orientation, and tilt angle of the rotating unit 120 from the work machine 100 via the communication device 560. The communication unit 554 also transmits an operation signal corresponding to the operation of the operation device 540 to the work machine 100.

[0034] The acquisition unit 552 acquires, via the communication unit 554, a video signal representing the image P1 captured by the front camera 122 and information indicating the roll angle and pitch angle of the work machine 100 based on the inclination angle of the rotating body.

[0035] Next, the display unit 553 will be described with reference to Figs. 5 to 9. Fig. 5 is a schematic diagram showing an example of an image cut out from the captured image P1 according to an embodiment of the present disclosure. Figs. 6 and 7 are schematic diagrams showing an example of an inclination angle display image superimposed on the captured image P1 according to an embodiment of the present disclosure. Figs. 8 and 9 are schematic diagrams showing an example of an image displayed by the first display device 520 according to an embodiment of the present disclosure.

[0036] The display unit 553 performs three processes: an image cropping process for dividing and displaying the captured image P1 on the multiple displays of the first display device 520; a process for generating an image representing the roll angle and pitch angle of the work machine 100; and a display process for superimposing the generated image on the captured image P1 and displaying it on the first display device 520.

[0037] 5, the display unit 553 cuts out from the captured image P1 a center image P11 to be displayed on the center display 521, a left image P12 to be displayed on the left display 522, a right image P13 to be displayed on the right display 523, an upper image P14 to be displayed on the upper display 524, and a lower image P15 to be displayed on the lower display 525. Note that if the first display device 520 is configured from a single display, image cutting may not be performed.

[0038] In the image generation process, the display unit 553 generates, for example, a tilt angle display image P101 shown in Fig. 6. The tilt angle display image P101 is an image that is superimposed on the central image P11. The tilt angle display image P101 has, for example, the same image size as the central image P11. The tilt angle display image P101 shown in Fig. 6 includes a roll angle image I1 that rotates in the direction of arrow A1 based on the roll angle of the work machine 100, with a predetermined reference point BP as the center of rotation. The tilt angle display image P101 shown in Fig. 6 also includes roll angle scale images I3 and I4 that indicate the value of the roll angle of the work machine 100. In other words, the tilt angle display image P101 shown in Fig. 6 includes roll angle scale images I3 and I4 that indicate the value of the roll angle of the work machine 100 in accordance with their positional relationship with the roll angle image I1. For example, based on the roll angle of the work machine 100, the roll angle image I1 rotates in the direction of the arrow A1 with a predetermined reference point BP as the center of rotation, and on the scales displayed by the roll angle scale images I3 and I4, the roll angle image I1 rotates to the scale indicating the roll angle of the work machine 100.

[0039] Furthermore, the inclination angle display image P101 shown in Fig. 6 includes a pitch angle image I2 that moves parallel to the up and down direction indicated by arrow A2 based on the pitch angle of the work machine 100. The inclination angle display image P101 shown in Fig. 6 also includes pitch angle scale images I5 and I6 that indicate the value of the pitch angle. That is, the inclination angle display image P101 shown in Fig. 6 includes pitch angle scale images I5 and I6 that indicate the value of the pitch angle in accordance with their positional relationship with the pitch angle image I2. For example, based on the pitch angle of the work machine 100, the pitch angle image I2 moves in the direction of arrow A2, with zero indicated by the pitch angle scale images I5 and I6 as the reference. The pitch angle image I2 moves to the scales on the pitch angle scale images I5 and I6 that indicate the pitch angle of the work machine 100.

[0040] The roll angle scale images I3 and I4 include graduations in 1-degree increments from -10 degrees to +10 degrees and numbers in 5-degree increments. The graduations in the roll angle scale images I3 and I4 are point-symmetric. In this embodiment, the graduations in the roll angle scale images I3 and I4 are point-symmetric with a predetermined reference point BP as the center of symmetry. For example, the graduations and numerical values ​​in the roll angle scale images I3 and I4 can be set arbitrarily. The pitch angle scale images I5 and I6 include graduations in 1-degree increments from -20 degrees to +20 degrees and numbers in 5-degree increments. The graduations in the pitch angle scale images I5 and I6 are graduations that have the same value in the up and down directions (the direction of arrow A2). For example, the graduations and numerical values ​​in the pitch angle scale images I5 and I6 can be set arbitrarily. In the example shown in FIG. 6, the roll angle is approximately 3 degrees and the pitch angle is approximately -7.5 degrees. The roll angle is positive when the revolving unit 120 is tilted to the right, and the pitch angle is negative when the revolving unit 120 is tilted forward. The scales of the roll angle scale images I3 and I4 and the roll angle image I1 are connected to the top and bottom ends of the inclination angle display image P101 (i.e., the top and bottom ends of the display screen of the first display device 520). The scales of the pitch angle scale images I5 and I6 and the pitch angle image I2 are connected to the left and right ends of the inclination angle display image P101 (i.e., the left and right ends of the display screen of the first display device 520). One of the roll angle scale images I3 and I4 or one of the pitch angle scale images I5 and I6 may be omitted. For example, the predetermined reference point BP may be located at the center of the inclination angle display image P101.

[0041] In the example shown in FIG. 6 , the roll angle image I1 and the pitch angle image I2 are linear images. In this embodiment, linear images are images shaped like long, thin lines. The width of the linear images is preferably limited to a width that is easily visible to the operator but does not unnecessarily reduce the visibility of the captured image P1. Depending on the resolution of the first display device 520, the width may be, for example, several pixels. The lengths of the roll angle image I1 and the pitch angle image I2 are not limited to those connected to the ends of the tilt angle display image P101 (e.g., those spanning the entire display screen of the first display device 520 or the display screen of the central display 521). For example, the lengths of the roll angle image I1 and the pitch angle image I2 may be connected to a portion of the roll angle scale image I3 or I4 or a portion of the pitch angle scale image I5 or I6. Furthermore, the lengths may be any length, such as an image having a length at least half the vertical or horizontal length of the display screen. Furthermore, the type (shape) of the figures of the roll angle image I1 and the pitch angle image I2 is not limited to a rectangle, but may be an ellipse, a polygon other than a rectangle, a polygon with rounded edges, etc. In this embodiment, the difference in the type of figures of the roll angle image I1 and the pitch angle image I2 also includes, for example, the difference in the type of line (solid line, dashed line, chain line, wavy line, etc.) in the case of linear images.

[0042] The roll angle image I1 and the pitch angle image I2 may have the same shape and color, or may have different shapes and colors. The roll angle image I1 may have the same color as the roll angle scale images I3 and I4, or a different color. The pitch angle image I2 may have the same color as the pitch angle scale images I5 and I6, or a different color. In the tilt angle display image P102 shown in FIG. 7, the roll angle is approximately −3 degrees and the pitch angle is approximately +7.5 degrees.

[0043] Furthermore, in the display processing, the display unit 553 generates the display center image P11e shown in Fig. 8 by superimposing, for example, the inclination angle display image P101 shown in Fig. 6 on the center image P11 shown in Fig. 5. Furthermore, as shown in Fig. 9, the display center image P11e is displayed on the center display 521, the left image P12 is displayed on the left display 522, the right image P13 is displayed on the right display 523, the upper image P14 is displayed on the upper display 524, and the lower image P15 is displayed on the lower display 525. In the example shown in Fig. 9, the display center image P11e includes the work implement 130, a roll angle image I1, and a pitch angle image I2, and it is possible to instantly recognize the inclination of the work machine 100, for example, by gazing at the captured image.

[0044] 10 is a flowchart showing a display control method by the control device 550 of the remote control cab according to an embodiment of the present disclosure. When remote operation of the work machine 100 is started, the control device 550 executes the following display control at a predetermined cycle. For example, the control is executed for each frame of the video signal.

[0045] 10 , first, the acquisition unit 552 acquires a captured image P1 captured by the front camera 122 of the work machine 100 via the communication unit 554 (step S1). Next, the acquisition unit 552 acquires information representing the roll angle and pitch angle of the work machine 100 from the work machine 100 via the communication unit 554 (step S2). Next, the display unit 553 cuts out a center image P11, a left image P12, a right image P13, an upper image P14, and a lower image P15 from the acquired captured image P1 (step S3).

[0046] Next, the display unit 553 generates an inclination angle display image P101 including the roll angle image I1, the pitch angle image I2, the roll angle scale images I3 and I4, and the pitch angle scale images I5 and I6 (step S4).

[0047] Next, the display unit 553 generates a display center image P11e by superimposing the tilt angle display image P101, which includes the roll angle image I1, the pitch angle image I2, the roll angle scale images I3 and I4, and the pitch angle scale images I5 and I6, on the center image P11, and displays the display center image P11e on the center display 521. In addition, the display unit 553 displays a left image P12 on the left display 522, a right image P13 on the right display 523, an upper image P14 on the upper display 524, and a lower image P15 on the lower display 525 (step S5), thereby ending the processing shown in FIG.

[0048] (Actions and Effects) As described above, the remote operation system 1 according to the embodiment of the present disclosure is a remote operation system 1 for remotely operating the work machine 100 by displaying on the first display device 520 an image P1 captured by the front camera 122 equipped on the work machine 100, and at least one of a roll angle image I1 that rotates based on the roll angle of the work machine 100 around a predetermined reference point BP as the center of rotation, and a pitch angle image I2 that moves parallel in the up and down direction based on the pitch angle of the work machine 100, is superimposed on the image P1 and displayed on the first display device 520. With this configuration, for example, the roll angle image I1 or the pitch angle image I2 can be easily superimposed on the captured image of the work implement 130, and according to this embodiment, the inclination of the work machine can be easily recognized. Furthermore, according to this embodiment, the amount of inclination can be easily recognized with a small amount of eye movement.

[0049] (Other Embodiments) One embodiment has been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design modifications and the like are possible.

[0050] For example, the inclination angle display image may be as shown in FIG. 11 . FIG. 11 is a schematic diagram showing another example of an inclination angle display image superimposed on a captured image according to an embodiment of the present disclosure. The inclination angle display image P103 shown in FIG. 11 is, for example, an image superimposed on the center image P11. The inclination angle display image P103 has, for example, the same image size as the center image P11. The inclination angle display image P103 shown in FIG. 11 includes a roll angle image I1a that rotates in the direction of arrow A1a based on the roll angle of the work machine 100 around a predetermined reference point BPa as the rotation center. The inclination angle display image P103 shown in FIG. 11 also includes a pitch angle image I2a that moves in parallel in the up-down direction indicated by arrow A2a based on the pitch angle of the work machine 100. In this case, the roll angle image I1a is depicted as a solid line, and the pitch angle image I2a is depicted as a dashed line. Furthermore, the roll angle image I1a is, for example, green, and the pitch angle image I2a is, for example, white. The reference point BPa is always located on the pitch angle image I2a and moves in the direction of the arrow A2a as the pitch angle image I2a moves in the direction of the arrow A2a. In this configuration, changes in the pitch angle are represented by the up and down movement of a white dashed line, and changes in the roll angle are represented by the rotation of a green line from the white dashed line. The tilt angle display image P103 includes a roll angle numeric image I7, which represents the value of the roll angle, and a pitch angle numeric image I8, which represents the value of the pitch angle. The roll angle numeric image I7 includes a number I71 indicating the value of the roll angle and an icon I72 indicating the value of the roll angle. The pitch angle numeric image I8 includes a number I81 indicating the value of the pitch angle and an icon I82 indicating the value of the pitch angle. When the tilt angle display image P103 shown in FIG. 11 is used, the roll angle scale image I3 or I4 or the pitch angle scale image I5 or I6 shown in FIG. 6 can be omitted. The roll angle image I1a can be an image representing 0 degrees in the horizontal position.

[0051] 6 includes a roll angle image I1 and a pitch angle image I2, but it is not necessary to include either one of them. Also, for example, the tilt angle display image P101 shown in Fig. 6 includes roll angle scale images I3 and I4 and pitch angle scale images I5 and I6, but it is not necessary to include either one or both of them.

[0052] Furthermore, for example, the roll angle scale images I3 and I4 and the pitch angle scale images I5 and I6 included in the tilt angle display image P101 shown in FIG. 6 may have different colors for the roll angle or pitch angle scale depending on the value, in order to visually notify the operator of dangerous areas, for example. For example, the roll angle scale may be red when the absolute value of the roll angle is 5 degrees or greater, and white when it is less than 5 degrees. For example, the pitch angle scale may be red when the absolute value of the pitch angle is 15 degrees or greater, and white when it is less than 15 degrees. Note that the numerical values ​​used as boundaries for changing the color are merely examples.

[0053] Furthermore, for example, one or both of the roll angle image I1 and pitch angle image I2 included in the tilt angle display image P101 shown in FIG. 6 may have different display modes, such as color, type of graphic, whether or not to blink, the blinking interval, and transparency value, depending on the magnitude of the roll angle and the magnitude of the pitch angle. For example, the roll angle image I1 may be colored red when the absolute value of the roll angle is 5 degrees or more, and green when it is less than 5 degrees. For example, the pitch angle image I2 may be colored red when the absolute value of the pitch angle is 15 degrees or more, and green when it is less than 15 degrees. Note that the numerical values ​​of the boundaries for changing the color are merely examples. Furthermore, the colors may be changed to three or more colors.

[0054] The first display device 520 may also be a display device worn by an operator, such as a head-mounted display.

[0055] Furthermore, part or all of the programs executed by the computer in the above embodiments can be distributed via computer-readable recording media or communication lines.

[0056] (Additional Note) The display control system (control device 550) and the remote operation system 1 according to the present disclosure can be understood, for example, as follows.

[0057] (1) A display control system (control device 550) according to a first aspect is a display control system for a display device (first display device 520), and includes a display unit 553 that displays, on the display device, a captured image P1 captured by an imaging device (front camera 122) provided on a work machine 100, and the display unit 553 displays, on the display device, at least one of a roll angle image I1 that rotates based on the roll angle of the work machine 100 around a predetermined reference point BP as the center of rotation, and a pitch angle image I2 that moves parallel in the vertical direction based on the pitch angle of the work machine 100, superimposed on the captured image P1. According to this aspect and each of the following aspects, it is possible to make it easier to recognize the inclination of the work machine.

[0058] (2) A display control system (control device 550) according to a second aspect is the display control system of (1), in which the roll angle image I1 or the pitch angle image I2 is a linear image. According to this aspect, it is possible to suppress a decrease in visibility of the captured image due to the superimposition of the roll angle image I1 or the pitch angle image I2.

[0059] (3) A display control system (control device 550) according to a third aspect is the display control system of (1) or (2), in which the roll angle image is connected to the top and bottom edges of the display screen of the display device, or the pitch angle image is connected to the left and right edges of the display screen of the display device. According to this aspect, it is possible to improve the distinguishability between the roll angle image and the pitch angle image.

[0060] (4) A display control system (control device 550) according to a fourth aspect is the display control system of any one of (1) to (3), in which (when the roll angle image is displayed), the display unit further superimposes at least one of a roll angle scale image representing the value of the roll angle and a roll angle numeric image representing the value of the roll angle on the captured image in accordance with a positional relationship with the roll angle image, and displays the superimposed image on the display device. According to this aspect, the value of the roll angle can be accurately recognized.

[0061] (5) A display control system (control device 550) according to a fifth aspect is the display control system of any one of (1) to (4), in which (when the pitch angle image is displayed), the display unit further superimposes at least one of a pitch angle scale image representing the value of the pitch angle and a pitch angle numeric image representing the value of the pitch angle on the captured image in accordance with a positional relationship with the pitch angle image, and displays the superimposed image on the display device. According to this aspect, the pitch angle value can be accurately recognized.

[0062] (6) A display control system (control device 550) according to a sixth aspect is the display control system of any one of (1) to (5), in which the display mode of the roll angle image changes depending on the magnitude of the roll angle of the work machine. According to this aspect, changes in the magnitude of the roll angle can be easily recognized.

[0063] (7) A display control system (control device 550) according to a seventh aspect is the display control system of any one of (1) to (6), in which the display mode of the pitch angle image changes depending on the magnitude of the pitch angle of the work machine. According to this aspect, changes in the magnitude of the pitch angle can be easily recognized.

[0064] (8) The remote operation system 1 according to the eighth aspect includes the display control system (control device 550) described in (1) to (7), a remotely provided display device (first display device 520), and an operation device 540. According to this aspect, it is possible to make it easier to recognize the inclination of the work machine.

[0065] According to each aspect of the present disclosure, it is possible to make it easier to recognize the inclination of a work machine.

[0066] 1...Remote operation system 100...Work machine 130...Work equipment 120...Slewing body 110...Traveling body 131...Boom 132...Arm 133...Bucket 134...Boom cylinder 135...Arm cylinder 136...Bucket cylinder 137...Boom angle sensor 138...Arm angle sensor 139...Bucket angle sensor 121...Driver's cab 122...Forward camera 123...Positioning device 124...Inclination measuring instrument 125...Hydraulic device 126...Work machine control device 500...Remote driver's cab 510...Driver's seat 520...First display device 530...Second display device 540...Operation device 550...Control device 551...Control unit 552...Acquisition unit 553...Display unit 554...Communication unit P1...Captured image I1...Roll angle image I2...Pitch angle image I3, I4...Roll angle scale image I5, I6...Pitch angle scale image I7...Roll angle numerical image I8...Pitch angle numerical image

Claims

1. A display control system for a display device, comprising: a display unit that displays an image captured by an imaging device equipped in a work machine on the display device, and the display unit displays on the display device at least one of a roll angle image that rotates based on the roll angle of the work machine around a predetermined reference point as the center of rotation, and a pitch angle image that moves parallel in the vertical direction based on the pitch angle of the work machine, superimposed on the captured image.

2. The display control system according to claim 1, wherein the roll angle image or the pitch angle image is a linear image.

3. The display control system according to claim 2, wherein the roll angle image is connected to the top and bottom edges of the display screen of the display device, or the pitch angle image is connected to the left and right edges of the display screen of the display device.

4. The display control system of claim 2, wherein the display unit further superimposes at least one of a roll angle scale image representing the value of the roll angle depending on a positional relationship with the roll angle image and a roll angle numerical image representing the value of the roll angle on the captured image and displays it on the display device.

5. A display control system as described in claim 2 or claim 4, wherein the display unit further superimposes at least one of a pitch angle scale image representing the value of the pitch angle and a pitch angle numerical image representing the value of the pitch angle in accordance with a positional relationship with the pitch angle image on the captured image and displays it on the display device.

6. A display control system according to claim 4, wherein the display manner of the roll angle image changes according to the magnitude of the roll angle of the work machine.

7. A display control system according to claim 5, wherein the display manner of the pitch angle image changes according to the magnitude of the pitch angle of the work machine.

8. A remote operation system comprising: the display control system according to claim 7; and a display device and an operation device provided remotely.

9. A display control method for a display device, comprising the steps of: when displaying an image captured by an imaging device equipped in a work machine on the display device, superimposing at least one of a roll angle image that rotates based on the roll angle of the work machine around a predetermined reference point as the center of rotation and the roll angle image that moves in parallel in the vertical direction based on the pitch angle of the work machine on the captured image and displaying it on the display device.

Images