Display control system, display control method and remote operation system

Abstract

To provide a control system and a control method which make it easy to recognize inclination of a work machine.SOLUTION: A display control system of a display device includes a display for displaying an image captured by an imaging device provided in a work machine, on the display device, wherein the display displays at least one of a roll angle image which is rotated with a predetermined reference point as a center of rotation on the basis of a roll angle of the work machine, and a pitch angle image which is moved in parallel in a vertical direction on the basis of a pitch angle of the work machine, on the display device while superimposing the one on the captured image.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present disclosure relates to a display control system, a display control method, and a remote operation system. [Background technology]

[0002] Patent Document 1 describes a remote control system that allows an operator to intuitively recognize the inclination of a work vehicle (hereinafter referred to as a work machine) when remotely operating the work machine. In the remote control system described in Patent Document 1, a posture image that represents at least one of the roll angle and pitch angle of the work machine is displayed. Note that the posture image represents the roll angle or pitch angle by the position of a bubble image included in an image that imitates, for example, an air bubble level. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-72401 A Summary of the Invention [Problem to be solved by the invention]

[0004] According to the remote control system described in Patent Document 1, a posture image is displayed on a display device facing the driver's seat, allowing the operator to intuitively recognize the inclination of the work machine. However, depending on the shape and position of the posture image, there is a problem in that it may be difficult to move the line of sight to the posture image and instantly recognize the amount of inclination of the work machine while gazing at the captured image during work.

[0005] The present disclosure has been made in consideration of the above circumstances, and aims to provide a display control system, a display control method, and a remote operation system that make it easier to recognize the inclination of a work machine. [Means for solving the problem]

[0006] The display control system according to the present disclosure is a display control system for a display device, and includes a display unit that displays an image captured by an imaging device equipped in a work machine on the display device, and the display unit displays on the display device at least one of a roll angle image that rotates based on the roll angle of the work machine around a predetermined reference point as a rotation center, and a pitch angle image that moves parallel in the vertical direction based on the pitch angle of the work machine, superimposed on the captured image.

[0007] A remote operation system according to the present disclosure includes the above-mentioned display control system, and a display device and an operation device that are remotely provided.

[0008] A display control method for a display device according to the present disclosure is a display control method for a display device, which, when displaying an image captured by an imaging device equipped in a work machine on the display device, displays on the display device at least one of a roll angle image that rotates based on the roll angle of the work machine around a predetermined reference point as a rotation center, and a pitch angle image that moves parallel in the vertical direction based on the pitch angle of the work machine, superimposed on the captured image. Effect of the Invention

[0009] The display control system, display control method, and remote operation system disclosed herein can make it easier to recognize the inclination of a work machine. [Brief description of the drawings]

[0010] [Figure 1] 1 is a schematic diagram illustrating a configuration example of a remote control system according to an embodiment of the present disclosure. [Diagram 2] 1 is an external view of a work machine according to an embodiment of the present disclosure. FIG. [Diagram 3] 2 is a schematic diagram showing an example of an image captured by an imaging device of a work machine according to an embodiment of the present disclosure. FIG. [Figure 4] FIG. 2 is a schematic block diagram illustrating a configuration example of a control device for a remote operator cab according to an embodiment of the present disclosure. [Diagram 5] 1 is a schematic diagram illustrating an example of an image cut out from a captured image according to an embodiment of the present disclosure. [Figure 6] 1A and 1B are schematic diagrams illustrating examples of images superimposed on a captured image according to an embodiment of the present disclosure. [Figure 7] 1A and 1B are schematic diagrams illustrating examples of images superimposed on a captured image according to an embodiment of the present disclosure. [Figure 8] 1 is a schematic diagram illustrating an example of an image displayed by a display device according to an embodiment of the present disclosure. [Figure 9] 1 is a schematic diagram illustrating an example of an image displayed by a display device according to an embodiment of the present disclosure. [Figure 10] 4 is a flowchart illustrating an example of an operation of the control device of the remote cab according to the embodiment of the present disclosure. [Figure 11] 11A and 11B are schematic diagrams illustrating other examples of images superimposed on a captured image according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In each drawing, the same or corresponding components are designated by the same reference numerals and the description thereof will be omitted as appropriate.

[0012] (Remote Control System) FIG. 1 is a schematic diagram showing a configuration example of a remote operation system according to an embodiment of the present disclosure. The remote operation system 1 is a remote operation system for remotely operating a work machine, and includes a work machine 100 that operates by remote operation, and a remote cab 500 for performing the remote operation. The work machine 100 operates at a work site (e.g., a mine, a quarry). The remote cab 500 is provided at a location away from the work machine 100 (e.g., in a city, within the work site). The work machine 100 and the remote cab 500 are connected via a communication means such as a mobile communication network or the Internet. The remote operation system 1 is a system for operating the work machine 100 using the remote cab 500.

[0013] The work machine 100 operates in accordance with an operation signal received from the remote operator's cab 500. In other words, no operator boards the work machine 100. The remote operator's cab 500 accepts operations of the work machine 100 by the operator, and transmits operation signals to the work machine 100. The operation signals will be described later.

[0014] (Remote Control Cabin) The remote operator's cab 500 includes an operator's seat 510, a first display device 520, a second display device 530, an operation device 540, a control device 550, and a communication device 560. The first display device 520 is disposed in front of the operator's seat 510. The first display device 520 is positioned in front of the operator when the operator sits in the operator's seat 510. As shown in FIG. 1, the first display device 520 is configured by a center display 521, a left display 522, a right display 523, an upper display 524, and a lower display 525 arranged side by side. The left display 522 is provided on the left side of the center display 521. The right display 523 is provided on the right side of the center display 521. The upper display 524 is provided on the upper side of the center display 521. The lower display 525 is provided on the lower side of the center display 521. The first display device 520 divides a part or all of an image captured by the work machine 100, for example, and displays it on five displays.

[0015] In other embodiments, the number of displays constituting the first display device 520 is not limited to this. For example, the first display device 520 may be composed of one display. Also, the first display device 520 may project an image onto a curved or spherical surface by a projector or the like. Also, the first display device 520 is one configuration example of the "display device" according to the present disclosure.

[0016] The second display device 530 is disposed diagonally forward of the driver's seat 510. Vehicle body information (inclination angle, remaining fuel, engine water temperature, etc.) transmitted from the work machine 100, notifications of abnormalities in the work machine 100, etc. are displayed on the second display device 530. Note that in other embodiments, the position of the second display device 530 does not have to be diagonally forward of the driver's seat 510 as long as it is a position where the operator can view it. Furthermore, the remote driver's cab 500 according to other embodiments does not need to be equipped with the second display device 530, and preferably the vehicle body information displayed on the second display device 530, notifications of abnormalities, etc. may be displayed on the first display device 520.

[0017] The operation device 540 is disposed near the driver's seat 510. The operation device 540 is located within an operable range of the operator when the operator sits in the driver's seat 510. The operation device 540 includes, for example, an electric lever and an electric pedal. When the operator operates the electric lever and the electric pedal, the operation device 540 outputs a work implement operation signal which is an operation signal for the boom 131, the arm 132, and the bucket 133, a rotation operation signal for the rotating body 120, and a travel operation signal for the traveling body 110. Note that in this embodiment, the work implement operation signal, the rotation operation signal, and the travel operation signal are collectively referred to as operation signals.

[0018] The control device 550 displays the captured image and vehicle body information received from the work machine 100 on the first display device 520. That is, the control device 550 is an example of a display control system. The display control system may further include an imaging device for acquiring captured images, an inclination measuring device 124 for acquiring the inclination angle of the work machine 100, and a display device that generates an inclination angle display image P101 (described later) based on the acquired inclination angle, and displays an image in which the inclination angle display image P101 is superimposed on the captured image by a display unit 553 that superimposes the generated inclination angle display image P101 on the captured image. The control device 550 transmits an operation signal input to the operation device 540 to the work machine 100. The display control system may be a remote cab 500 including the control device 550, and may further include a work machine control device 126 of the work machine 100 and a front camera 122.

[0019] The communication device 560 transmits and receives operation signals, signals representing vehicle body information, captured images, and the like, to and from the communication device 143 of the work machine 100.

[0020] (Work Machinery) 2 is an external view of a work machine according to an embodiment of the present disclosure. The work machine 100 according to an embodiment of the present disclosure is a backhoe shovel, which is a type of hydraulic excavator. Note that the work machine 100 according to other embodiments may be a hydraulic excavator other than a backhoe shovel, such as a face shovel or a rope shovel, or a work machine other than a hydraulic excavator, such as a wheel loader or a bulldozer. The work machine 100 includes a hydraulically driven work implement 130, a rotating body 120 that supports the work implement 130, and a traveling body 110 that supports the rotating body 120.

[0021] The work machine 130 includes a boom 131, an arm 132, and a bucket 133. The work machine 130 is driven by extension and contraction of a boom cylinder 134, an arm cylinder 135, and a bucket cylinder 136. A boom angle sensor 137, an arm angle sensor 138, and a bucket angle sensor 139 are attached to the boom 131, the arm 132, and the bucket 133, respectively.

[0022] A base end of the boom 131 is attached to the rotating body 120 via a pin. The arm 132 connects the boom 131 and the bucket 133. A base end of the arm 132 is attached to a tip of the boom 131 via a pin. The bucket 133 is equipped with a cutting edge 133T for excavating earth and sand, and a container for storing the excavated earth and sand. A base end of the bucket 133 is attached to the tip of the arm 132 via a pin.

[0023] The boom cylinder 134 is a hydraulic cylinder for driving the boom 131. A base end of the boom cylinder 134 is attached to the rotating body 120. A tip end of the boom cylinder 134 is attached to the boom 131. The arm cylinder 135 is a hydraulic cylinder for driving the arm 132. A base end of the arm cylinder 135 is attached to the boom 131. A tip end of the arm cylinder 135 is attached to the arm 132. The bucket cylinder 136 is a hydraulic cylinder for driving the bucket 133. A base end of the bucket cylinder 136 is attached to the arm 132. A tip end of the bucket cylinder 136 is attached to the bucket 133.

[0024] The boom angle sensor 137 is attached to, for example, the boom 131 and detects the inclination angle of the boom 131. The arm angle sensor 138 is attached to, for example, the arm 132 and detects the inclination angle of the arm 132. The bucket angle sensor 139 is attached to, for example, the bucket 133 and detects the inclination angle of the bucket 133. The boom angle sensor 137, the arm angle sensor 138, and the bucket angle sensor 139 according to the embodiment of the present disclosure detect the inclination angle with respect to the horizontal plane. Note that the angle sensor according to other embodiments is not limited to this, and may detect the inclination angle with respect to another reference plane. For example, in other embodiments, the angle sensor may detect the relative rotation angle by a potentiometer provided at the base end of the boom 131, the arm 132, and the bucket 133, or may detect the inclination angle by measuring the cylinder lengths of the boom cylinder 134, the arm cylinder 135, and the bucket cylinder 136 and converting the cylinder lengths into angles.

[0025] The revolving body 120 is provided with a cab 121. The cab 121 is provided on the left side of the work machine 130. The cab 121 is provided with a front camera 122. The front camera 122 is installed at the front and upper part of the cab 121. The front camera 122 captures an image in front of the cab 121 through a windshield in front of the cab 121. Here, "forward" refers to the direction in which the work machine 130 is mounted on the revolving body 120, and "rearward" refers to the opposite direction to "forward". "Side" refers to a direction intersecting the front-rear direction (left-right direction). An example of the front camera 122 is an imaging device using, for example, a CCD (Charge Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor. Note that the work machine 100 according to other embodiments may not be provided with the cab 121. Even in this case, the front camera 122 is installed so as to capture an image of the front at a position equivalent to the driver's cab 121. In other embodiments, the front camera 122 may be configured with two or more cameras.

[0026] Fig. 3 is an example of a captured image captured by an imaging device that is the front camera 122 of the work machine 100 according to an embodiment of the present disclosure. The front camera 122 captures an image of a range in which the work target in front of the work implement 130 and the cab 121 is captured. That is, the work target in front of the work implement 130 and the cab 121 is captured in the captured image P1 captured by the front camera 122, as shown in Fig. 3. In addition, because the cab 121 is provided on the left side of the work implement 130, a part of the boom 131 is captured in the right side of the captured image P1.

[0027] Fig. 4 is a schematic block diagram showing an example configuration of the remote operation system 1 according to an embodiment of the present disclosure. As shown in Fig. 2 or 4, the work machine 100 includes a work machine control device 126, a positioning device 123, a front camera 122, an incline measuring instrument 124, a hydraulic device 125, a work machine attitude acquisition device 142, and a communication device 143. The work machine attitude acquisition device 142 includes a boom angle sensor 137, an arm angle sensor 138, and a bucket angle sensor 139.

[0028] The positioning device 123 acquires the position of the revolving body 120 and the direction in which the revolving body 120 faces. The positioning device 123 includes two receivers that receive positioning signals from artificial satellites that constitute the Global Navigation Satellite System (GNSS). The two receivers are installed at different positions on the revolving body 120. The positioning device 123 detects information indicating latitude, longitude, and altitude, and the position of a representative point of the revolving body 120 in the site coordinate system (the origin of the shovel coordinate system) based on the positioning signals received by the receivers. The positioning device 123 calculates the direction in which the revolving body 120 faces as the relationship between the installation position of one receiver and the installation position of the other receiver, using the positioning signals received by the two receivers. Note that in another embodiment, the positioning device 123 may detect the direction in which the revolving body 120 faces based on the measurement value of a rotary encoder or an Inertial Measurement Unit (IMU).

[0029] The inclination measuring device 124 measures the acceleration and angular velocity of the revolving unit 120, and detects the attitude of the revolving unit 120 (for example, inclination angles such as roll angle, pitch angle, yaw angle, etc.) based on the measurement results. The inclination measuring device 124 is installed, for example, on the underside of the revolving unit 120. For example, an inertial measurement unit (IMU) can be used as the inclination measuring device 124. The inclination measuring device 124 may be an inclinometer that detects the inclination angle without using the acceleration and angular velocity. Note that the work machine 100 according to other embodiments does not need to be equipped with the positioning device 123. Note that FIG. 2 shows a state in which the roll angle θr, pitch angle θp, and yaw angle θy of the revolving unit 120 are all 0 degrees, and the directions of the three axes XYZ of the global coordinate system coincide with the directions of the three axes of the on-site coordinate system (local coordinate system). In this embodiment, the roll angle θr, pitch angle θp, and yaw angle θy of the rotating body 120 are also referred to as the roll angle, pitch angle, and yaw angle of the work machine 100.

[0030] The hydraulic device 125 includes a hydraulic oil tank, a hydraulic pump, and a flow control valve. The hydraulic pump is driven by the power of an engine or an electric motor (not shown), and supplies hydraulic oil to the boom cylinder 134, the arm cylinder 135, and the bucket cylinder 136 via the flow control valve. The flow control valve has a rod-shaped spool, and adjusts the flow rate of hydraulic oil supplied to the boom cylinder 134, the arm cylinder 135, and the bucket cylinder 136 depending on the position of the spool. The spool is driven based on a control command received from the work machine control device 126. In other words, the amount of hydraulic oil supplied to the boom cylinder 134, the arm cylinder 135, and the bucket cylinder 136 is controlled by the work machine control device 126.

[0031] The work machine control device 126 transmits the captured image captured by the front camera 122, the rotation speed, position (including information indicating latitude, longitude, and altitude), orientation, and tilt angle (attitude) of the revolving body 120 to the remote operator's cab 500 via the communication device 143. In this embodiment, information measured by various sensors equipped in the work machine 100 and transmitted by the work machine control device 126 is also referred to as vehicle body information. The work machine control device 126 receives an operation signal from the remote operator's cab 500 via the communication device 143. The work machine control device 126 drives the work implement 130, the revolving body 120, or the traveling body 110 based on the received operation signal.

[0032] (Remote cab control device) The control device 550 is configured using one or more computers such as a microcontroller, and includes a control unit 551 as a functional configuration configured by a combination of hardware such as the computer and peripheral devices of the computer, and software such as a program executed by the computer. The control unit 551 also includes an acquisition unit 552, a display unit 553, and a communication unit 554.

[0033] The communication unit 554 receives a video signal representing an image captured by the front camera 122, the rotation speed, position (information indicating longitude, latitude, and altitude), orientation, and tilt angle of the rotating body 120 from the work machine 100 via the communication device 560. In addition, the communication unit 554 transmits an operation signal corresponding to the operation of the operation device 540 to the work machine 100.

[0034] The acquisition unit 552 acquires, via the communication unit 554, a video signal representing the image P1 captured by the front camera 122 and information indicating the roll angle and pitch angle of the work machine 100 based on the inclination angle of the rotating bed.

[0035] Next, the display unit 553 will be described with reference to Figs. 5 to 9. Fig. 5 is a schematic diagram showing an example of an image cut out from a captured image P1 according to an embodiment of the present disclosure. Figs. 6 and 7 are schematic diagrams showing an example of an inclination angle display image superimposed on the captured image P1 according to an embodiment of the present disclosure. Figs. 8 and 9 are schematic diagrams showing an example of an image displayed by the first display device 520 according to an embodiment of the present disclosure.

[0036] The display unit 553 performs three processes: an image cropping process for dividing and displaying the captured image P1 on multiple displays of the first display device 520; a generation process for an image representing the roll angle and pitch angle of the work machine 100; and a display process for superimposing the generated image on the captured image P1 and displaying it on the first display device 520.

[0037] 5, for example, from the captured image P1, the display unit 553 cuts out a center image P11 for display on the center display 521, a left image P12 for display on the left display 522, a right image P13 for display on the right display 523, an upper image P14 for display on the upper display 524, and a lower image P15 for display on the lower display 525. Note that when the first display device 520 is configured from a single display, it is not necessary to cut out the image.

[0038] In the image generation process, the display unit 553 generates, for example, a tilt angle display image P101 shown in FIG. 6. The tilt angle display image P101 is an image superimposed on the central image P11. The tilt angle display image P101 has, for example, the same image size as the central image P11. The tilt angle display image P101 shown in FIG. 6 includes a roll angle image I1 that rotates in the direction of the arrow A1 based on the roll angle of the work machine 100 with a predetermined reference point BP as the rotation center. The tilt angle display image P101 shown in FIG. 6 also includes roll angle scale images I3 and I4 that indicate the value of the roll angle of the work machine 100. That is, the tilt angle display image P101 shown in FIG. 6 includes roll angle scale images I3 and I4 that indicate the value of the roll angle of the work machine 100 according to the positional relationship with the roll angle image I1. For example, based on the roll angle of the work machine 100, the roll angle image I1 rotates in the direction of the arrow A1 with a predetermined reference point BP as the center of rotation, and in the scales displayed by the roll angle scale images I3 and I4, the roll angle image I1 rotates to the scale indicating the roll angle of the work machine 100.

[0039] The inclination angle display image P101 shown in Fig. 6 also includes a pitch angle image I2 that moves in parallel in the up-down direction indicated by the arrow A2 based on the pitch angle of the work machine 100. The inclination angle display image P101 shown in Fig. 6 also includes pitch angle scale images I5 and I6 that indicate the value of the pitch angle. That is, the inclination angle display image P101 shown in Fig. 6 includes pitch angle scale images I5 and I6 that indicate the value of the pitch angle according to their positional relationship with the pitch angle image I2. For example, based on the pitch angle of the work machine 100, the pitch angle image I2 moves in the direction of the arrow A2 based on 0 indicated by the pitch angle scale images I5 and I6. The pitch angle image I2 moves to the scales indicating the pitch angle of the work machine 100 in the pitch angle scale images I5 and I6.

[0040] The roll angle scale images I3 and I4 include scales every 1 degree from -10 degrees to +10 degrees and numbers every 5 degrees. The scales of the roll angle scale images I3 and I4 are point-symmetric scales. In this embodiment, the scales of the roll angle scale images I3 and I4 are point-symmetric scales with a predetermined reference point BP as the center of symmetry. For example, the scales and numerical values ​​of the roll angle scale images I3 and I4 can be set arbitrarily. The pitch angle scale images I5 and I6 include scales every 1 degree from -20 degrees to +20 degrees and numbers every 5 degrees. The scales of the pitch angle scale images I5 and I6 are scales that have the same value in the up and down directions (the direction of the arrow A2). For example, the scales and numerical values ​​of the pitch angle scale images I5 and I6 can be set arbitrarily. In the example shown in FIG. 6, the roll angle is about 3 degrees and the pitch angle is about -7.5 degrees. The roll angle is a positive value when the rotating body 120 is tilted to the right, and the pitch angle is a negative value when the rotating body 120 is tilted forward. The scales of the roll angle scale images I3 and I4 and the roll angle image I1 are connected to the upper and lower ends of the inclination angle display image P101 (i.e., the upper and lower ends of the display screen of the first display device 520). The scales of the pitch angle scale images I5 and I6 and the pitch angle image I2 are connected to the left and right ends of the inclination angle display image P101 (i.e., the left and right ends of the display screen of the first display device 520). One of the roll angle scale images I3 and I4 or one of the pitch angle scale images I5 and I6 may be omitted. For example, the predetermined reference point BP may be located at the center of the inclination angle display image P101.

[0041] In the example shown in FIG. 6, the roll angle image I1 and the pitch angle image I2 are linear images. In this embodiment, a linear image is an image shaped like a long thin line. It is desirable that the width of the linear image is a width that is easily visible to the operator and does not unnecessarily reduce the visibility of the captured image P1, and the upper limit is limited to a width. Although it depends on the resolution of the first display device 520, the width can be, for example, about several pixels. The length of the roll angle image I1 and the length of the pitch angle image I2 are not limited to those connected to the end of the tilt angle display image P101 (for example, those that cover the entire display screen of the first display device 520 or the display screen of the central display 521), but may be, for example, those connected to a part of the roll angle scale image I3 or I4 or a part of the pitch angle scale image I5 or I6, and may be any length, such as an image having a length at least half the vertical or horizontal length of the display screen. Furthermore, the type (shape) of the figures of the roll angle image I1 and the pitch angle image I2 is not limited to a rectangle, but may be an ellipse, a polygon other than a rectangle, a polygon with rounded ends, etc. In this embodiment, the difference in the type of figure of the roll angle image I1 and the pitch angle image I2 also includes, for example, the difference in the type of line (solid line, dashed line, chain line, wavy line, etc.) in the case of a linear image.

[0042] The roll angle image I1 and the pitch angle image I2 may have the same shape and color, or may have different shapes and colors. The roll angle image I1 may have the same color as the roll angle scale images I3 and I4, or may have a different color. The pitch angle image I2 may have the same color as the pitch angle scale images I5 and I6, or may have a different color. In the tilt angle display image P102 shown in FIG. 7, the roll angle is about -3 degrees and the pitch angle is about +7.5 degrees.

[0043] In the display process, the display unit 553 generates a display center image P11e shown in Fig. 8 by superimposing, for example, the tilt angle display image P101 shown in Fig. 6 on the center image P11 shown in Fig. 5. In addition, as shown in Fig. 9, the display center image P11e is displayed on the center display 521, the left image P12 is displayed on the left display 522, the right image P13 is displayed on the right display 523, the upper image P14 is displayed on the upper display 524, and the lower image P15 is displayed on the lower display 525. In the example shown in Fig. 9, the display center image P11e includes the work machine 130, the roll angle image I1, and the pitch angle image I2, and, for example, the inclination of the work machine 100 can be instantly recognized while gazing at the captured image.

[0044] (Example of remote cab operation) 10 is a flowchart showing a display control method by the control device 550 of the remote operator's cab according to the embodiment of the present disclosure. When remote operation of the work machine 100 is started, the control device 550 executes the display control shown below at a predetermined cycle. For example, the control is executed for each frame of the video signal.

[0045] 10, first, the acquisition unit 552 acquires a captured image P1 captured by the front camera 122 of the work machine 100 via the communication unit 554 (step S1). Next, the acquisition unit 552 acquires information indicating the roll angle and pitch angle of the work machine 100 from the work machine 100 via the communication unit 554 (step S2). Next, the display unit 553 cuts out a center image P11, a left image P12, a right image P13, an upper image P14, and a lower image P15 from the acquired captured image P1 (step S3).

[0046] Next, the display unit 553 generates an inclination angle display image P101 including the roll angle image I1, the pitch angle image I2, the roll angle scale images I3 and I4, and the pitch angle scale images I5 and I6 (step S4).

[0047] Next, the display unit 553 generates a display center image P11e by superimposing the tilt angle display image P101, which includes the roll angle image I1, the pitch angle image I2, the roll angle scale images I3 and I4, and the pitch angle scale images I5 and I6, on the center image P11, and displays the display center image P11e on the center display 521. In addition, it displays the left image P12 on the left display 522, the right image P13 on the right display 523, the upper image P14 on the upper display 524, and the lower image P15 on the lower display 525 (step S5), and the process shown in FIG.

[0048] (Action and effect) As described above, the remote operation system 1 according to the embodiment of the present disclosure is a remote operation system 1 for remotely operating the work machine 100 by displaying the captured image P1 captured by the front camera 122 equipped in the work machine 100 on the first display device 520, and at least one of the roll angle image I1 that rotates based on the roll angle of the work machine 100 around a predetermined reference point BP as the rotation center and the pitch angle image I2 that moves in parallel in the vertical direction based on the pitch angle of the work machine 100 is superimposed on the captured image P1 and displayed on the first display device 520. According to this configuration, for example, the roll angle image I1 and the pitch angle image I2 can be easily superimposed on the captured image of the work machine 130, and according to this embodiment, the inclination of the work machine can be easily recognized. Furthermore, according to this embodiment, the amount of inclination can be easily recognized with a small movement of the line of sight.

[0049] (Other embodiments) Although one embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design modifications and the like are possible.

[0050] For example, the inclination angle display image may be as shown in FIG. 11. FIG. 11 is a schematic diagram showing another example of the inclination angle display image superimposed on the captured image according to the embodiment of the present disclosure. The inclination angle display image P103 shown in FIG. 11 is, for example, an image superimposed on the central image P11. The inclination angle display image P103 has, for example, the same image size as the central image P11. The inclination angle display image P103 shown in FIG. 11 includes a roll angle image I1a that rotates in the direction of the arrow A1a based on the roll angle of the work machine 100 with a predetermined reference point BPa as the rotation center. In addition, the inclination angle display image P103 shown in FIG. 11 includes a pitch angle image I2a that moves in parallel in the up-down direction indicated by the arrow A2a based on the pitch angle of the work machine 100. In this case, the type of figure of the roll angle image I1a is a solid line, and the type of figure of the pitch angle image I2a is a broken line. Also. The roll angle image I1a is, for example, green, and the pitch angle image I2a is, for example, white. The reference point BPa is always located on the pitch angle image I2a, and moves in the direction of the arrow A2a with the movement of the pitch angle image I2a in the direction of the arrow A2a. In this configuration, the change in pitch angle is represented by the up and down movement of the white dashed line, and the change in roll angle is represented by the rotation of the green line from the white dashed line. The inclination angle display image P103 includes a roll angle numerical image I7 which is an image representing the value of the roll angle, and a pitch angle numerical image I8 which is an image representing the value of the pitch angle. The roll angle numerical image I7 includes a number I71 indicating the value of the roll angle and an icon I72 indicating the value of the roll angle. The pitch angle numerical image I8 includes a number I81 indicating the value of the pitch angle and an icon I82 indicating the value of the pitch angle. When the inclination angle display image P103 shown in FIG. 11 is used, the roll angle scale image I3 or I4 or the pitch angle scale image I5 or I6 shown in FIG. 6 and the like can be omitted. The roll angle image I1a can be an image representing 0 degrees in a horizontal state.

[0051] For example, the tilt angle display image P101 shown in Fig. 6 includes a roll angle image I1 and a pitch angle image I2, but it is not necessary to include either one of them. For example, the tilt angle display image P101 shown in Fig. 6 includes roll angle scale images I3 and I4 and pitch angle scale images I5 and I6, but it is not necessary to include either one or both of them.

[0052] In addition, for example, the roll angle scale images I3 and I4 and the pitch angle scale images I5 and I6 included in the tilt angle display image P101 shown in Fig. 6 may have different colors of the roll angle or pitch angle scale depending on the value, for example, to visually inform the operator of dangerous areas. For example, the roll angle scale may be red when the absolute value of the roll angle is 5 degrees or more, and white when it is less than 5 degrees. In addition, for example, the pitch angle scale may be red when the absolute value of the pitch angle is 15 degrees or more, and white when it is less than 15 degrees. Note that the numerical values ​​of the boundaries when changing the colors are merely examples.

[0053] In addition, for example, one or both of the roll angle image I1 and pitch angle image I2 included in the tilt angle display image P101 shown in Fig. 6 may change the display mode such as color, type of figure, whether or not to blink, blink interval, transparency value, etc. according to the magnitude of the roll angle and the magnitude of the pitch angle. For example, the roll angle image I1 is red when the absolute value of the roll angle is 5 degrees or more, and green when it is less than 5 degrees. Also, for example, the pitch angle image I2 is red when the absolute value of the pitch angle is 15 degrees or more, and green when it is less than 15 degrees. Note that the numerical value of the boundary for changing the color is one example. Also, it may be changed to multiple colors of three or more colors.

[0054] Furthermore, the first display device 520 may be, for example, a display device worn by an operator, such as a head-mounted display.

[0055] Furthermore, a part or all of the programs executed by the computer in the above-described embodiments can be distributed via a computer-readable recording medium or a communication line.

[0056] (Additional Note) The display control system (control device 550) and the remote control system 1 according to the present disclosure may be understood, for example, as follows.

[0057] (1) A display control system (control device 550) according to a first aspect is a display control system for a display device (first display device 520), and includes a display unit 553 that displays, on the display device, a captured image P1 captured by an imaging device (front camera 122) equipped on a work machine 100, and the display unit 553 displays, on the display device, at least one of a roll angle image I1 that rotates based on the roll angle of the work machine 100 around a predetermined reference point BP as the center of rotation, and a pitch angle image I2 that moves in parallel in the vertical direction based on the pitch angle of the work machine 100, superimposed on the captured image P1. According to this aspect and each of the following aspects, it is possible to make it easier to recognize the inclination of the work machine.

[0058] (2) A display control system (control device 550) according to a second aspect is the display control system of (1), in which the roll angle image I1 or the pitch angle image I2 is a linear image. According to this aspect, it is possible to suppress a decrease in visibility of a captured image caused by superimposing the roll angle image I1 or the pitch angle image I2.

[0059] (3) A display control system (control device 550) according to a third aspect is the display control system according to (1) or (2), in which the roll angle image is connected to the upper and lower ends of the display screen of the display device, or the pitch angle image is connected to the left and right ends of the display screen of the display device. According to this aspect, it is possible to improve the distinguishability between the roll angle image and the pitch angle image.

[0060] (4) A display control system (control device 550) according to a fourth aspect is the display control system of (1) to (3), in which (when displaying the roll angle image), the display unit further superimposes at least one of a roll angle scale image representing the value of the roll angle and a roll angle numeric image representing the value of the roll angle on the captured image in accordance with a positional relationship with the roll angle image, and displays the image on the display device. According to this aspect, the value of the roll angle can be accurately recognized.

[0061] (5) A display control system (control device 550) according to a fifth aspect is the display control system of (1) to (4), in which (when displaying the pitch angle image), the display unit further superimposes at least one of a pitch angle scale image representing the value of the pitch angle and a pitch angle numerical image representing the value of the pitch angle on the captured image in accordance with a positional relationship with the pitch angle image, and displays the image on the display device. According to this aspect, the pitch angle value can be accurately recognized.

[0062] (6) A display control system (control device 550) according to a sixth aspect is the display control system of (1) to (5), in which the display mode of the roll angle image changes according to the magnitude of the roll angle of the work machine. According to this aspect, it is easy to recognize the change in the magnitude of the roll angle.

[0063] (7) A display control system (control device 550) according to a seventh aspect is the display control system of (1) to (6), in which the pitch angle image changes in display mode depending on the magnitude of the pitch angle of the work machine. According to this aspect, it is easy to recognize the change in the magnitude of the pitch angle.

[0064] (8) The remote operation system 1 according to the eighth aspect includes the display control system (control device 550) of (1) to (7), a remotely provided display device (first display device 520), and an operation device 540. According to this aspect, it is possible to easily recognize the inclination of the work machine. [Explanation of symbols]

[0065] 1...Remote operation system 100...Work machine 130...Working equipment 120...Rotating body 110...Traveling body 131...Boom 132...Arm 133...Bucket 134...Boom cylinder 135...Arm cylinder 136...Bucket cylinder 137...Boom angle sensor 138...Arm angle sensor 139...Bucket angle sensor 121...Driver's cab 122...Forward camera 123...Positioning device 124...Inclination measuring device 125...Hydraulic device 126...Working machine control device 500...Remote driver's cab 510...Driver's seat 520...First display device 530...Second display device 540...Operation device 550...Control device 551...Control unit 552...Acquisition unit 553...Display unit 554...Communication unit P1...Captured image I1...Roll angle image I2...Pitch angle image I3, I4...Roll angle scale image I5, I6…Pitch angle scale image I7…Roll angle numerical image I8…Pitch angle numerical image

Claims

1. A display control system for a display device, comprising: a display unit that displays an image captured by an imaging device provided in the work machine on the display device, The display unit displays on the display device at least one of a roll angle image that rotates based on the roll angle of the work machine around a predetermined reference point as a rotation center and a pitch angle image that translates in the vertical direction based on the pitch angle of the work machine by superimposing it on the captured image. Display control system.

2. The roll angle image or the pitch angle image is a linear image. The display control system according to claim 1 .

3. The roll angle image is connected to the top and bottom edges of the display screen of the display device, or the pitch angle image is connected to the left and right edges of the display screen of the display device. The display control system according to claim 2 .

4. The display unit displays at least one of a roll angle scale image representing a value of the roll angle in accordance with a positional relationship with the roll angle image and a roll angle numeric image representing the value of the roll angle, The captured image is further superimposed on the captured image and displayed on the display device. The display control system according to claim 2 .

5. The display unit displays at least one of a pitch angle scale image representing a value of the pitch angle and a pitch angle numerical image representing the value of the pitch angle according to a positional relationship with the pitch angle image, The captured image is further superimposed on the captured image and displayed on the display device. The display control system according to claim 2 or 4.

6. The display mode of the roll angle image changes according to the magnitude of the roll angle of the work machine. The display control system according to claim 4 .

7. The display manner of the pitch angle image changes according to the magnitude of the pitch angle of the work machine. The display control system according to claim 5 .

8. A display control system according to claim 7; A display device and an operating device provided remotely; Equipped with Remote control system.

9. A display control method for a display device, comprising: When an image captured by an imaging device provided in a work machine is displayed on the display device, at least one of a roll angle image that rotates based on the roll angle of the work machine around a predetermined reference point as a rotation center and a pitch angle image that moves in parallel in the up and down direction based on the pitch angle of the work machine is superimposed on the captured image and displayed on the display device. Display control method.

Images