Display control device and display method

Abstract

To provide a display control device capable of suppressing a decrease in work efficiency when working with a work machine.SOLUTION: A captured image acquisition unit acquires a captured image including a working unit from a camera installed on the work machine. A cutting-edge image generation unit generates an image of a cutting edge obtained by projecting the cutting edge of the work machine vertically to the projection surface. A display image generation unit generates a display image in which the captured image, the image of the cutting edge, a left line, and a right line are superimposed. The left line extends in the front-back direction of the working unit along the projection plane passing through the left edge of the cutting edge image. The right line extends in the front-rear direction of the working unit along the projection surface passing through the right edge of the cutting edge image. The display control unit outputs display signals for displaying the display images.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a display control device and a display method. [Background technology]

[0002] There is known technology for remotely operating a work machine. A remotely operated work machine is equipped with a camera, which captures images of the work site where the machine is operating. The captured images are transmitted to a remote location and displayed on a display device located at the remote location. An operator at the remote location remotely operates the work machine while viewing the captured images displayed on the display device. Because the captured images displayed on the display device are two-dimensional, it is difficult to give the operator a sense of perspective. Patent Document 1 discloses a technique for displaying a mesh-like line image on the surface of a work target shown in a captured image in order to give an operator a sense of perspective. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-035645 A Summary of the Invention [Problem to be solved by the invention]

[0004] When operating a work machine while viewing a two-dimensional captured image, it is difficult for an operator to recognize an area of ​​the work target that can be worked on by the work machine, which may result in a decrease in the operator's work efficiency. An object of the present disclosure is to provide a display control device and a display method that can suppress a decrease in work efficiency when working using a work machine. [Means for solving the problem]

[0005] According to one aspect of the present invention, a display control device is a display control device that displays an image used for operating a work machine equipped with a work implement, and includes an image acquisition unit that acquires an image of the work machine from a camera provided on the work machine, a blade tip shadow generation unit that generates a blade tip shadow projected in a vertical direction onto a projection surface, a display image generation unit that generates a display image that superimposes the image, the blade tip shadow, a left line that passes through the left end of the blade tip shadow and extends in the fore-and-aft direction of the work machine along the projection surface, and a right line that passes through the right end of the blade tip shadow and extends in the fore-and-aft direction of the work machine along the projection surface, and a display control unit that outputs a display signal for displaying the display image. Effect of the Invention

[0006] According to the above aspect, it is possible to suppress a decrease in work efficiency when working using a work machine. [Brief description of the drawings]

[0007] [Figure 1] 1 is a schematic diagram showing a configuration of a working system according to a first embodiment. [Diagram 2] 1 is an external view of a work machine according to a first embodiment. [Diagram 3] 1 is a schematic block diagram showing a configuration of a remote control device according to a first embodiment. [Figure 4] FIG. 2 is a diagram showing an example of a display image according to the first embodiment. [Diagram 5] 4 is a side view showing the relationship between a cutting edge shadow image and a cutting edge reach gauge image according to the first embodiment. FIG. [Figure 6] 5 is a flowchart showing a display control process by the remote control device according to the first embodiment. [Figure 7] FIG. 4 is an external view of a work machine according to a second embodiment. [Figure 8] FIG. 11 is a schematic block diagram showing the configuration of a remote control device according to a second embodiment. [Figure 9] FIG. 11 is a diagram showing an example of a display image according to the second embodiment. [Figure 10]13 is a side view showing the relationship between a cutting edge shadow image and a cutting edge reach gauge image according to the second embodiment. FIG. [Figure 11] FIG. 11 is a schematic block diagram showing the configuration of a remote control device according to a third embodiment. [Figure 12] FIG. 13 is a side view showing the relationship between a cutting edge shadow image and a cutting edge reach gauge image according to the third embodiment. [Figure 13] FIG. 13 is a diagram showing an example of a display image according to another embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] First Embodiment <Work System 1> FIG. 1 is a schematic diagram showing the configuration of a work system 1 according to a first embodiment. The work system 1 includes a work machine 100 and a remote operator's cab 500. The work machine 100 operates at a work site. Examples of work sites include mines and quarries. The remote operator's cab 500 is provided in a remote location away from the work site. Examples of remote locations include cities and within the work site. In other words, an operator remotely operates the work machine 100 from a distance where the operator cannot see the work machine 100.

[0009] The work machine 100 is remotely operated based on an operation signal transmitted from a remote cab 500. The remote cab 500 is connected to the work machine 100 via an access point 300 provided at the work site. An operation signal indicating an operation by an operator received at the remote cab 500 is transmitted to the work machine 100 via the access point 300. The work machine 100 operates based on the operation signal received from the remote cab 500. In other words, the work system 1 includes a remote operation system composed of the work machine 100 and the remote cab 500. In addition, the work machine 100 captures an image of a work target, and the image is displayed in the remote cab 500. In other words, the work system 1 is an example of a display control system.

[0010] "Work Machine 100" FIG. 2 is an external view of the work machine 100 according to the first embodiment. The work machine 100 according to the first embodiment is a loading shovel (face shovel). Note that the work machine 100 according to other embodiments may be another work machine such as a backhoe, a wheel loader, or a bulldozer. The work machine 100 includes a traveling body 110, a rotating body 120 supported by the traveling body 110, and a working implement 130 that is hydraulically operated and supported by the rotating body 120. The rotating body 120 is supported so as to be freely rotatable about a central rotation axis O. The working implement 130 is provided at the front of the rotating body 120.

[0011] The work machine 130 includes a boom 130A, an arm 130B, and a bucket 130C. The base end of the boom 130A is attached to the rotating body 120 via a pin. The arm 130B connects the boom 130A and the bucket 130C. A base end of the arm 130B is attached to a tip end of the boom 130A via a pin. Bucket 130C includes cutting edge 130D for excavating earth and sand, and a container for storing the excavated earth and sand. A base end of bucket 130C is attached to a tip of arm 130B via a pin.

[0012] The work machine 130 is driven by the movements of a boom cylinder 131A, an arm cylinder 131B, and a bucket cylinder 131C. Hereinafter, the boom cylinder 131A, the arm cylinder 131B, and the bucket cylinder 131C are collectively referred to as hydraulic cylinders 131. The boom cylinder 131A is a hydraulic cylinder for operating the boom 130A. A base end of the boom cylinder 131A is attached to the rotating body 120. A tip end of the boom cylinder 131A is attached to the boom 130A. The arm cylinder 131B is a hydraulic cylinder for driving the arm 130B. A base end of the arm cylinder 131B is attached to the boom 130A. A tip end of the arm cylinder 131B is attached to the arm 130B. The bucket cylinder 131C is a hydraulic cylinder for driving the bucket 130C. A base end of the bucket cylinder 131C is attached to the boom 130A. A tip end of the bucket cylinder 131C is attached to the bucket 130C.

[0013] A boom attitude sensor 132A, an arm attitude sensor 132B, and a bucket attitude sensor 132C are attached to the work machine 130 to detect the attitudes of the boom 130A, the arm 130B, and the bucket 130C. Hereinafter, the boom attitude sensor 132A, the arm attitude sensor 132B, and the bucket attitude sensor 132C are collectively referred to as the attitude sensor 132. The attitude sensor 132 according to the first embodiment is a stroke sensor attached to the hydraulic cylinder 131. That is, the attitude sensor 132 detects the stroke length of the hydraulic cylinder 131. The stroke length is the moving distance of the rod from the stroke end of the hydraulic cylinder 131. The stroke end refers to the end of the movable range of the rod. That is, the stroke end refers to the position of the rod when the hydraulic cylinder 131 is most contracted, or the position of the rod when the hydraulic cylinder 131 is most extended.

[0014] Boom posture sensor 132A is provided on boom cylinder 131A and detects the stroke length of boom cylinder 131A. The arm posture sensor 132B is provided on the arm cylinder 131B and detects the stroke length of the arm cylinder 131B. The bucket attitude sensor 132C is provided in the bucket cylinder 131C and detects the stroke length of the bucket cylinder 131C.

[0015] Note that the attitude sensor 132 according to other embodiments is not limited to this. For example, in other embodiments, the attitude sensor 132 may detect the relative rotation angle by a potentiometer provided at the base end of the boom 130A, the arm 130B, and the bucket 130C, may detect the rotation angle with respect to the vertical direction by an IMU, or may detect the rotation angle with respect to the vertical direction by an inclinometer.

[0016] The revolving body 120 is provided with a cab 121. The cab 121 is provided with a camera 122. The camera 122 is installed at the front and upper part of the cab 121. The camera 122 captures an image in front of the cab 121 through a windshield in front of the cab 121. Here, "forward" refers to the direction in which the work machine 130 is mounted on the revolving body 120, and "rearward" refers to the opposite direction of "forward". "Side" refers to a direction intersecting the front-rear direction (left-right direction). Examples of the camera 122 include an imaging device using a CCD (Charge Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor. In other embodiments, the camera 122 does not necessarily have to be provided in the cab 121, and may be provided at a position where at least the construction target and the work machine 130 can be imaged. In other words, the imaging range of the camera 122 includes at least a part of the work machine 130.

[0017] The work machine 100 is equipped with a camera 122 , a position and orientation calculator 123 , an incline measuring device 124 , a hydraulic device 125 , and a vehicle control device 126 .

[0018] The position and orientation calculator 123 calculates the position of the rotating body 120 and the orientation in which the rotating body 120 faces. The position and orientation calculator 123 includes two receivers that receive positioning signals from satellites that constitute the GNSS. The two receivers are installed at different positions on the rotating body 120. The position and orientation calculator 123 detects the position of a representative point (the origin of the vehicle body coordinate system) of the rotating body 120 in the on-site coordinate system based on the positioning signals received by the receivers. The position and orientation calculator 123 calculates the orientation in which the rotating body 120 faces as the relationship between the installation position of one receiver and the installation position of the other receiver using the positioning signals received by the two receivers. Note that in other embodiments, the position and orientation calculator 123 may detect the orientation in which the rotating body 120 faces based on the measurement values ​​of a rotary encoder or an IMU.

[0019] The inclination measuring device 124 measures the acceleration and angular velocity of the rotating body 120, and detects the attitude (e.g., roll angle, pitch angle) of the rotating body 120 based on the measurement results. The inclination measuring device 124 is installed, for example, on the underside of the rotating body 120. The inclination measuring device 124 can use, for example, an inertial measurement unit (IMU).

[0020] The hydraulic device 125 supplies hydraulic oil to the hydraulic cylinder 131. The flow rate of the hydraulic oil supplied to the hydraulic cylinder 131 is controlled based on a control command received from the vehicle control device 126.

[0021] The vehicle control device 126 transmits the images captured by the camera 122, the rotation speed, position, direction, and tilt angle of the revolving unit 120, the attitude of the work unit 130, and the traveling speed of the running unit 110 to the remote operator's cab 500. In addition, the vehicle control device 126 receives an operation signal from the remote operator's cab 500, and drives the work unit 130, the revolving unit 120, and the running unit 110 based on the received operation signal.

[0022] Remote Control Cabin 500 The remote operator's cab 500 includes an operator's seat 510 , a display device 520 , an operating device 530 , and a remote control device 540 . Display device 520 is disposed in front of driver's seat 510. Display device 520 is located in front of the operator when the operator sits in driver's seat 510. Display device 520 may be configured with a plurality of displays arranged in a row, or may be configured with one large display as shown in Fig. 1. Display device 520 may also be one that projects an image onto a curved or spherical surface by a projector or the like.

[0023] The operation device 530 is an operation device for the remote operation system. In response to the operation of the operator, the operation device 530 generates an operation signal for the boom cylinder 131A, an operation signal for the arm cylinder 131B, an operation signal for the bucket cylinder 131C, a swing operation signal for the left and right of the revolving body 120, and a travel operation signal for moving the traveling body 110 forward and backward, and outputs them to the remote control device 540. The operation device 530 is composed of, for example, a lever, a knob switch, and a pedal (not shown). The operation device 530 is disposed near the driver's seat 510. The operation device 530 is located within a range that the operator can operate when the operator sits in the driver's seat 510.

[0024] The remote control device 540 generates a display image based on the data received from the work machine 100, and displays the display image on the display device 520. The remote control device 540 also transmits an operation signal representing the operation of the operation device 530 to the work machine 100. The remote control device 540 is an example of a display control device.

[0025] FIG. 3 is a schematic block diagram showing the configuration of a remote control device 540 according to the first embodiment. The remote control device 540 is a computer including a processor 610, a main memory 630, a storage 650, and an interface 670. The storage 650 stores a program. The processor 610 reads the program from the storage 650, loads it in the main memory 630, and executes processing according to the program. The remote control device 540 is connected to a network via the interface 670.

[0026] Examples of the storage 650 include a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. The storage 650 may be an internal medium directly connected to a common communication line of the remote control device 540, or an external medium connected to the remote control device 540 via an interface 670. The storage 650 is a non-transitory tangible storage medium.

[0027] The processor 610, by executing a program, is provided with a data acquisition unit 611, a posture identification unit 612, a blade tip shadow generation unit 613, a display image generation unit 614, a display control unit 615, an operation signal input unit 616, and an operation signal output unit 617. In another embodiment, the remote control device 540 may include a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to or instead of the above configuration. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, some or all of the functions realized by the processor 610 may be realized by the integrated circuit. Such an integrated circuit is also included in an example of a processor.

[0028] The data acquisition unit 611 acquires data from the work machine 100 indicating images captured by the camera 122 , the rotation speed, position, direction and tilt angle of the rotating body 120 , the attitude of the work implement 130 , and the traveling speed of the traveling body 110 .

[0029] The attitude identifying unit 612 identifies the attitude of the work machine 100 in the vehicle body coordinate system and the attitude in the site coordinate system based on the data acquired by the data acquiring unit 611. The vehicle body coordinate system is a local coordinate system that has an origin at the intersection of the rotation center axis O of the rotating unit 120 and the bottom surface of the running unit 110 and is defined by three axes of the rotating unit 120, namely, the front-rear axis, the left-right axis, and the up-down axis. The site coordinate system is a global coordinate system that has an origin at a specified point (such as a reference station) at the work site and is defined by three axes, namely, the latitude axis, the longitude axis, and the vertical axis. The attitude identifying unit 612 identifies the positions of the tip of the boom 130A, the tip of the arm 130B, and both the left and right ends of the cutting edge 130D in the vehicle body coordinate system and the site coordinate system. A specific method of identifying the position of each part by the data acquiring unit 611 will be described later.

[0030] The blade tip shadow generating unit 613 generates a blade tip shadow image representing a blade tip shadow projected vertically onto a projection plane based on the positions of both ends of the blade tip 130D in the site coordinate system identified by the posture identifying unit 612. The projection plane according to the first embodiment is a plane passing through the bottom surface of the traveling body 110. Specifically, the blade tip shadow generating unit 613 generates a blade tip shadow image in the following procedure. The blade tip shadow generating unit 613 specifies the position of the blade tip shadow projected onto the projection plane in the site coordinate system by rewriting the values ​​of the vertical axis components of the positions of both ends of the blade tip 130D to zero. The blade tip shadow generating unit 613 converts the position of the blade tip shadow in the site coordinate system to a position in the image coordinate system based on known camera parameters indicating the relationship between the image coordinate system, which is a two-dimensional orthogonal coordinate system related to the image captured by the camera 122, and the site coordinate system. The blade tip shadow generating unit 613 generates a blade tip shadow image by drawing a line segment representing the blade tip 130D at the converted position.

[0031] The display image generating unit 614 generates a display image by superimposing the cutting edge shadow image G1 and the cutting edge reach gauge image G2 on the captured image acquired by the data acquiring unit 611. Fig. 4 is a diagram showing an example of a display image according to the first embodiment. The cutting edge reach gauge image G2 includes a left line G21, a right line G22, a maximum reach line G23, a scale line G24, a scale value G25, and a reference range figure G26.

[0032] The left line G21 is a line that indicates the reachable range of the left end of the blade tip 130D. As shown in Fig. 4, the left line G21 passes through the left end of the blade tip shadow image G1. The right line G22 is a line that indicates the reachable range of the right end of the blade tip 130D. As shown in Fig. 4, the right line G22 passes through the right end of the blade tip shadow image G1.

[0033] The maximum reach line G23 is a line indicating the front edge of the reachable range of the cutting edge 130D. The maximum reach line G23 connects the front end of the left line G21 and the front end of the right line G22. The scale line G24 is a line indicating the distance from the rotation center axis O of the rotating body 120. The scale lines G24 are provided at equal intervals. In the example shown in Fig. 4, the scale lines G24 are provided at intervals of 2 meters. Each scale line G24 is provided so as to connect the left line G21 and the right line G22. The maximum reach line G23 and the scale line G24 are parallel to the blade tip shadow image G1. The scale value G25 is provided corresponding to the scale line G24 and represents the distance indicated by the scale line G24 as a numerical value. In the example shown in FIG. 4, the scale value G25 is provided near the right end of the scale line G24.

[0034] The reference range figure G26 is a figure showing the reachable range of the blade tip 130D on the projection surface. The reference range figure G26 according to the first embodiment is a rectangle surrounded by the left line G21, the right line G22, the leading edge of the reachable range on the projection surface, and the trailing edge of the reachable range on the projection surface. The reachable range of the blade tip 130D on the projection surface is the reachable range of the blade tip 130D under the condition that the projection surface and the blade tip 130D are in contact with each other. The reference range figure G26 is highlighted by hatching, coloring, or the like.

[0035] The maximum reach line G23 and the front ends of the left line G21 and right line G22 represent the front edge of the reachable range of the blade tip 130D when no condition is imposed for the projection surface to come into contact with the blade tip 130D. The maximum reach line G23, the left line G21, and the right line G22 are an example of a reachable range figure projecting the reachable range of the blade tip when no condition is imposed.

[0036] 5 is a side view showing the relationship between the cutting edge shadow image G1 and the cutting edge arrival gauge image G2 according to the first embodiment. The cutting edge shadow image G1 and the cutting edge arrival gauge image G2 according to the first embodiment are drawn on the projection plane F1, which is a plane passing through the bottom surface of the traveling body 110. Therefore, when the cutting edge shadow image G1 and the cutting edge arrival gauge image G2 are superimposed on the captured image, the cutting edge shadow image G1 and the cutting edge arrival gauge image G2 appear to be sunk into the ground surface F2 in a portion of the ground surface F2 that is higher than the projection plane F1. The cutting edge shadow image G1 and the cutting edge arrival gauge image G2 appear to be floating in relation to the ground surface F2 in a portion of the ground surface F2 that is lower than the projection plane F1.

[0037] 5, the leading edge of the cutting edge reach gauge image G2, i.e., the maximum reach line G23, is projected onto the projection plane F1 at a position within the reachable range R of the cutting edge 130D that is farthest from the pivot axis O. Therefore, no matter what posture the cutting edge 130D is in, the cutting edge shadow image G1 is always located in front of the maximum reach line G23. As shown in FIG. 5, a reference range figure G26 indicates the range where the reachable range of the blade tip 130D and the projection surface overlap.

[0038] In addition, since the camera 122 is fixed to the revolving unit 120, the reachable range of the cutting edge 130D on the projection plane in the image coordinate system does not change regardless of the rotation of the revolving unit 120 or the travel of the traveling unit 110. In other words, the cutting edge reach gauge image G2 is constant regardless of the position and posture of the work machine 100. Therefore, the display image generating unit 614 according to the first embodiment generates a display image by superimposing a previously prepared cutting edge reach gauge image G2 on the captured image.

[0039] The display control unit 615 outputs a display signal for displaying the display image generated by the display image generation unit 614 to the display device 520 . The operation signal input unit 616 receives an operation signal from the operation device 530 . The operation signal output unit 617 transmits the operation signal received by the operation signal input unit 616 to the work machine 100.

[0040] How to identify posture Here, a description will be given of a method for identifying the attitude by the attitude identifying unit 612. The attitude identifying unit 612 identifies the positions in the vehicle body coordinate system and the positions in the site coordinate system of the tip (pin at the tip) of the boom 130A, the tip (pin at the tip) of the arm 130B, and both ends of the cutting edge 130D, in the following procedure.

[0041] The attitude identifying unit 612 identifies the angle of the boom 130A relative to the revolving unit 120, i.e., the angle relative to the front-rear axis of the vehicle body coordinate system, based on the stroke length of the boom cylinder 131A. The attitude identifying unit 612 identifies a boom vector extending from the base end (base end pin) of the boom 130A to the tip (tip pin) of the boom 130A in the vehicle body coordinate system, based on the angle of the boom 130A and the known length of the boom 130A. The attitude identifying unit 612 identifies the position vector of the tip (tip pin) of the boom 130A in the vehicle body coordinate system by adding the known position vector of the base end (base end pin) of the boom 130A in the vehicle body coordinate system and the boom vector.

[0042] The posture identification unit 612 identifies the angle of the arm 130B relative to the boom 130A based on the stroke length of the arm cylinder 131B. The posture identification unit 612 identifies the angle of the arm 130B relative to the front-rear axis by adding the identified angle of the arm 130B and the angle of the boom 130A relative to the front-rear axis of the vehicle body coordinate system. The posture identification unit 612 identifies an arm vector extending from the base end (base end pin) of the arm 130B to the tip (tip pin) of the arm 130B in the vehicle body coordinate system based on the angle of the arm 130B and the known length of the arm 130B. The posture identification unit 612 identifies the position vector of the tip (tip pin) of the arm 130B in the vehicle body coordinate system by adding the position vector of the tip (tip pin) of the boom 130A in the vehicle body coordinate system and the arm vector.

[0043] The posture identification unit 612 identifies the angle of the bucket 130C relative to the arm 130B based on the stroke length of the bucket cylinder 131C. The posture identification unit 612 identifies the angle of the bucket 130C relative to the front-rear axis by adding the identified angle of the bucket 130C and the angle of the arm 130B relative to the front-rear axis of the vehicle body coordinate system. The posture identification unit 612 identifies a right bucket vector and a left bucket vector based on the angle of the bucket 130C, a known length from the base end (pin at the base end) of the bucket 130C to the cutting edge 130D, and a known width of the cutting edge 130D. The right bucket vector is a vector that extends from the base end (pin at the base end) of the bucket 130C to the right end of the cutting edge 130D in the vehicle body coordinate system. The left bucket vector is a vector that extends from the base end of the bucket 130C to the left end of the cutting edge 130D. Posture identification unit 612 adds the position vector of the tip (pin at the tip) of arm 130B in the vehicle body coordinate system to the left bucket vector, thereby identifying the position vector of the left end of cutting edge 130D in the vehicle body coordinate system. Also, posture identification unit 612 adds the position vector of the tip (pin at the tip) of arm 130B in the vehicle body coordinate system to the right bucket vector, thereby identifying the position vector of the right end of cutting edge 130D in the vehicle body coordinate system.

[0044] The attitude identification unit 612 can identify the position of each part in the site coordinate system by translating the position of each part in the vehicle body coordinate system based on the position of the work machine 100 in the site coordinate system, and rotating it based on the orientation (yaw angle) of the rotating body 120 and the roll angle and pitch angle of the work implement 130.

[0045] Display control method Fig. 6 is a flowchart showing a display control process by the remote control device 540 according to the first embodiment. When the operator starts remote operation of the work machine 100 using the remote operator's cab 500, the remote control device 540 executes the display control process shown in Fig. 6 at regular time intervals.

[0046] The data acquisition unit 611 acquires data indicating the image captured by the camera 122, the rotation speed, position, direction and tilt angle of the rotating body 120, the attitude of the work implement 130, and the traveling speed of the traveling body 110 from the vehicle control device 126 of the work machine 100 (step S1). Next, the attitude identification unit 612 identifies the positions of both ends of the cutting edge 130D in the vehicle body coordinate system based on the data acquired in step S1 (step S2).

[0047] The cutting edge shadow generating unit 613 specifies the position of the cutting edge shadow projected on the projection surface in the vehicle body coordinate system by rewriting the values ​​of the vertical axis components of the positions of both ends of the cutting edge 130D in the vehicle body coordinate system specified in step S2 to zero (step S3). The cutting edge shadow generating unit 613 converts the position of the cutting edge shadow in the vehicle body coordinate system to a position in the image coordinate system based on the camera parameters (step S4). The cutting edge shadow generating unit 613 generates a cutting edge shadow image G1 by drawing a line segment at the converted position (step S5).

[0048] The display image generating unit 614 generates a display image by superimposing the cutting edge shadow image G1 generated in step S5 and the cutting edge arrival gauge image G2 prepared in advance on the captured image acquired in step S1 (step S6). The display control unit 615 outputs a display signal for displaying the display image generated in step S6 to the display device 520 (step S7). As a result, the display image as shown in FIG.

[0049] Actions and Effects Thus, according to the first embodiment, the remote control device 540 causes the display device 520 to display a display image in which the captured image showing the work machine 130, the cutting edge shadow image G1 in which the cutting edge 130D is projected vertically onto the projection surface, and the left line G21 and the right line G22 that pass through both ends of the cutting edge shadow image G1 and extend in the front-rear direction along the projection surface are superimposed. This allows the operator to easily recognize the range of the work object to be excavated by the work machine 130. In other words, the operator recognizes that the part of the work object shown in the captured image between the left line G21 and the right line G22 will be excavated, and can estimate the amount of soil to be excavated. Therefore, the remote control device 540 can suppress a decrease in work efficiency when working using the work machine 100.

[0050] The display image according to the first embodiment includes a reference range figure G26 that indicates a reachable range under the condition that the cutting edge 130D is brought into contact with the projection surface F1. This allows the operator to recognize the range in which the piston of the hydraulic cylinder 131 may hit the stroke end when the cutting edge 130D is moved on the projection surface F1. Therefore, the operator can reduce the possibility that the piston of the hydraulic cylinder 131 will hit the stroke end by operating the operation device 530 while recognizing the positional relationship between the cutting edge shadow image G1 and the reference range figure G26.

[0051] In the display image according to the first embodiment, the maximum reach line G23 is displayed at the position farthest from the turning center axis O of the work machine 100 within the reachable range of the cutting edge 130D. This allows the operator to determine whether or not the excavation target ahead can be excavated from the current position by visually checking the display image. Note that in other embodiments, the same effect can be achieved even if the maximum reach line G23 is not displayed and the left line G21 and the right line G22 simply extend to the front edge of the reachable range. Also, in other embodiments, as long as the maximum reach line G23 is displayed, the same effect can be achieved even if the left line G21 and the right line G22 extend to infinity.

[0052] In addition, the left line G21 and the right line G22 included in the display image according to the first embodiment extend to the position farthest from the central axis O of the turning machine 100 in the reachable range of the cutting edge 130D. In addition, the maximum reach line G23 is displayed at the position farthest from the central axis O of the turning machine 100 in the reachable range of the cutting edge 130D. This allows the operator to determine whether or not the excavation target ahead can be excavated from the current position by visually checking the display image. Note that in other embodiments, the same effect can be achieved even if the maximum reach line G23 is not displayed and the left line G21 and the right line G22 simply extend to the front edge of the reachable range. In other embodiments, as long as the maximum reach line G23 is displayed, the same effect can be achieved even if the left line G21 and the right line G22 extend to infinity.

[0053] The display image according to the first embodiment includes a scale line G24 and a scale value G25 indicating the distance from the pivot axis O for each of a plurality of positions away from the pivot axis O. This allows the operator to recognize the position of the cutting edge 130D in the depth direction by visually checking the display image. Note that in other embodiments, the same effect can be achieved even if either the scale line G24 or the scale value G25 is not displayed.

[0054] Second embodiment The cutting edge shadow image G1 and cutting edge arrival gauge image G2 according to the first embodiment are images projected onto a projection plane F1, which is a plane passing through the bottom surface of the traveling object 110. In contrast, the cutting edge shadow image G1 and cutting edge arrival gauge image G2 according to the second embodiment are projected onto the ground surface F2. That is, the projection plane according to the second embodiment is the ground surface F2.

[0055] "Work Machine 100" FIG. 7 is an external view of a work machine 100 according to a second embodiment. The work machine 100 according to the second embodiment further includes a depth detection device 127 in addition to the configuration of the first embodiment. The depth detection device 127 is provided near the camera 122, and detects the depth in the same direction as the imaging direction of the camera 122. The depth is the distance from the depth detection device 127 to an object. Examples of the depth detection device 127 include a LiDAR device, a radar device, and a stereo camera. The detection range of the depth detection device 127 is approximately the same as the imaging range of the camera 122.

[0056] Remote control device 540 8 is a schematic block diagram showing the configuration of a remote control device 540 according to the second embodiment. In addition to the configuration according to the first embodiment, the remote control device 540 according to the second embodiment further includes a terrain update unit 618 and a gauge generation unit 619. Furthermore, the remote control device 540 according to the second embodiment differs from the first embodiment in the processing of the blade tip shadow generation unit 613.

[0057] The terrain update unit 618 updates the terrain data indicating the three-dimensional shape of the work target in the site coordinate system based on the depth data acquired by the data acquisition unit 611 from the depth detection device 127. Specifically, the terrain update unit 618 updates the terrain data in the following procedure.

[0058] The terrain update unit 618 converts the depth data into three-dimensional data related to the vehicle body coordinate system. Since the depth detection device 127 is fixed to the revolving body 120, the conversion function between the depth data and the vehicle body coordinate system can be obtained in advance. The terrain update unit 618 removes a portion of the generated three-dimensional data in which the work implement 130 is shown, based on the attitude of the work implement 130 in the vehicle body coordinate system identified by the attitude identification unit 612. The terrain update unit 618 converts the three-dimensional data in the vehicle body coordinate system into three-dimensional data in the site coordinate system, based on the position and attitude of the vehicle body acquired by the data acquisition unit 611. The terrain update unit 618 updates the terrain data already recorded in the main memory 630 using the newly generated three-dimensional data. That is, the terrain update unit 618 replaces the portion of the already recorded terrain data that overlaps with the new three-dimensional data with the value of the new three-dimensional data. This allows the terrain update unit 618 to always record the latest terrain data in the main memory 630.

[0059] The gauge generating unit 619 generates a cutting edge reach gauge image G2 projected on the ground surface F2 based on the terrain data. For example, the gauge generating unit 619 generates the cutting edge reach gauge image G2 in the following procedure. Based on the position and posture of the vehicle body acquired by the data acquiring unit 611, the gauge generating unit 619 converts the part of the terrain data included in the imaging range into the vehicle body coordinate system. Using the terrain data in the vehicle body coordinate system, the gauge generating unit 619 projects the known reach range of the cutting edge 130D and a plurality of lines that divide the reach range at equal intervals onto the ground surface F2. As a result, the gauge generating unit 619 specifies the positions of the left line G21, the right line G22, the maximum reach line G23, and the scale line G24 in the vehicle body coordinate system.

[0060] Next, the gauge generating unit 619 specifies a surface where the reachable range R of the known cutting edge 130D and the topographical data in the vehicle body coordinate system overlap as a reference range graphic G26 that represents a reachable range under the condition that the cutting edge 130D is brought into contact with the ground surface F2. Next, the gauge generating unit 619 converts the left line G21, the right line G22, the maximum reach line G23, the scale line G24, and the reference range graphic G26 into an image based on the camera parameters of the camera 122. The gauge generating unit 619 assigns a scale value G25 to the vicinity of each scale line G24 of the converted image. This allows the gauge generating unit 619 to generate a cutting edge reach gauge image G2 projected onto the ground surface F2. The blade tip shadow generating unit 613, like the gauge generating unit 619, generates a blade tip shadow image G1 by projecting the blade tip 130D onto the ground surface F2 based on the topographical data.

[0061] The display image generating unit 614 generates a display image by superimposing the cutting edge shadow image G1 and the cutting edge reach gauge image G2 on the captured image acquired by the data acquiring unit 611. Fig. 9 is a diagram showing an example of a display image according to the second embodiment. The cutting edge reach gauge image G2 includes a left line G21, a right line G22, a maximum reach line G23, a scale line G24, a scale value G25, and a reference range figure G26.

[0062] 10 is a side view showing the relationship between a cutting edge shadow image G1 and a cutting edge arrival gauge image G2 according to the second embodiment. The cutting edge shadow image G1 and the cutting edge arrival gauge image G2 according to the second embodiment are drawn along the ground surface F2 detected by the depth detection device 127. Therefore, when the cutting edge shadow image G1 and the cutting edge arrival gauge image G2 are superimposed on the captured image, the cutting edge shadow image G1 and the cutting edge arrival gauge image G2 appear to be stuck on the ground surface F2.

[0063] The reference range figure G26 according to the second embodiment represents a reachable range under the condition that the blade tip 130D is in contact with the ground surface F2, but is not limited thereto. For example, the reference range figure G26 according to another embodiment may represent a reachable range under the condition that the blade tip 130D is in contact with a plane passing through the bottom surface of the running object 110, as in the first embodiment. In this case, the gauge generating unit 619 generates the reference range figure G26 by projecting the reachable range under the condition that the blade tip 130D is in contact with a plane passing through the bottom surface of the running object 110 onto the ground surface F2.

[0064] Third embodiment The reference range figure G26 generated by the remote control device 540 according to the first and second embodiments represents a reachable range under a condition where the cutting edge 130D is brought into contact with a projection plane (a plane passing through the bottom surface of the traveling body 110 or the ground surface). In contrast, the remote control device 540 according to the third embodiment represents a reachable range of the cutting edge 130D under a condition where only the arm 130B is driven. This is because, as a mode of use of the loading shovel, the excavation operation of the work target is often performed by pushing the arm 130B, and the piston of the arm cylinder 131B is more likely to hit the stroke end than the boom cylinder 131A and the bucket cylinder 131C. The configuration of the work system 1 according to the third embodiment is basically the same as that of the first embodiment.

[0065] Remote control device 540 11 is a schematic block diagram showing the configuration of a remote control device 540 according to the third embodiment. In addition to the configuration according to the first embodiment, the remote control device 540 according to the third embodiment further includes a reference range specifying unit 620. The reference range specifying unit 620 calculates the reachable range of the cutting edge 130D when the boom 130A and the bucket 130C are fixed and only the arm 130B is driven, based on the postures of the boom 130A and the bucket 130C specified by the posture specifying unit 612.

[0066] FIG. 12 is a side view showing the relationship between the cutting edge shadow image G1 and the cutting edge reach gauge image G2 according to the third embodiment. Specifically, the reference range identifying unit 620 identifies the rotation center P (pin center) of the arm 130B based on the posture of the boom 130A, and identifies the length L from the rotation center to the cutting edge 130D based on the posture of the bucket 130C. Then, the reference range identifying unit 620 calculates the reachable range R1 of the cutting edge 130D when only the arm 130B is driven based on the known rotation range of the arm 130B. The reference range identifying unit 620 generates a reference range figure G26 by projecting the calculated reachable range R1 onto the projection plane F1 from the vertical direction. The reference range figure G26 generated by the reference range identifying unit 620 changes every time the posture of at least one of the boom 130A and the bucket 130C changes.

[0067] This allows the operator to remotely operate the work machine 100 so that the piston of the arm cylinder 131B does not hit the stroke end by controlling the work machine 130 so that the cutting edge shadow image G1 does not hit the end of the reference range figure G26.

[0068] <<Variation>> The cutting edge arrival gauge image G2 according to the third embodiment is a shape projected onto the projection surface F1, but is not limited thereto. For example, the cutting edge arrival gauge image G2 according to other embodiments may be a shape projected onto the ground surface F2 as in the second embodiment.

[0069] Other Embodiments Although one embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes are possible. That is, in other embodiments, the order of the above-mentioned processes may be changed as appropriate. Also, some of the processes may be executed in parallel.

[0070] The remote control device 540 according to the embodiment described above may be configured by a single computer, or the configuration of the remote control device 540 may be divided and arranged among multiple computers, and the multiple computers may function as the remote control device 540 by working together. In this case, some of the computers constituting the remote control device 540 may be provided inside the remote operator's cab 500, and other computers may be provided outside the remote operator's cab 500. For example, some of the computers constituting the remote control device 540 may be provided in the work machine 100.

[0071] FIG. 13 is a diagram showing an example of a display image according to another embodiment. The cutting edge reach gauge image G2 according to the above-mentioned embodiment includes a left line G21 and a right line G22, so that the operator can recognize the range excavated by the work machine 130. On the other hand, the cutting edge reach gauge image G2 according to another embodiment may include a center line G27 instead of the left line G21 and the right line G22 in the display image, as shown in FIG. 13. The center line G27 passes through the center point of the cutting edge 130D and extends in the front-rear direction along the projection surface. In this case, too, the operator can recognize the position of the cutting edge 130D in the depth direction by at least one of the end point of the center line G27, the maximum reach line G23, the scale line G24, the scale value G25, and the reference range figure G26.

[0072] The reference range figure G26 according to the embodiment described above shows the leading edge and trailing edge of the reachable range of the cutting edge 130D under a predetermined condition, but is not limited thereto in other embodiments. For example, when the work machine 100 is a loading shovel, the cutting edge 130D of the bucket 130C faces forward, so that excavation work is usually performed by the pushing action of the arm 130B. Therefore, the leading edge is more likely to hit the stroke end than the trailing edge of the reachable range. Therefore, the reference range figure G26 according to other embodiments may show only the leading edge of the reachable range of the cutting edge 130D under a predetermined condition. On the other hand, when the work machine 100 is a backhoe, the cutting edge 130D of the bucket 130C faces backward, so that excavation work is usually performed by the pulling action of the arm 130B. Therefore, the trailing edge is more likely to hit the stroke end than the leading edge of the reachable range. Therefore, the reference range figure G26 according to other embodiments may show only the trailing edge of the reachable range of the cutting edge 130D under a predetermined condition. [Explanation of symbols]

[0073] 1...Work system 100...Work machine 110...Traveling body 120...Rotating body 121...Operator's cab 122...Camera 130...Work machine 130A...Boom 130B...Arm 130C...Bucket 130D...Cutting tip 500...Remote operator's cab 510...Operator's seat 520...Display device 530...Operation device 540...Remote control device 611...Data acquisition unit 612...Attitude determination unit 613...Cutting tip shadow generation unit 614...Display image generation unit 615...Display control unit 616...Operation signal input unit 617...Operation signal output unit 618...Terrain update unit 619...Gauge generation unit 620...Reference range determination unit G1...Cutting tip shadow image G2...Cutting tip reach gauge image G21...Left line G22...Right line G23...Maximum reach line G24...Scale line G25...Scale value G26…Reference range diagram

Claims

1. A display control device that displays an image used for operating a work machine having a work implement, an image acquisition unit that acquires an image showing the work machine from a camera provided on the work machine; a display image generating unit that generates a display image by superimposing an image representing the end position of a reachable range of the cutting edge of the work machine under a condition where the projection surface and the cutting edge of the work machine are in contact with each other, and a line that passes through a point where a specific part of the cutting edge is projected onto the projection surface and extends in the front-rear direction of the work machine along the projection surface; a display control unit that outputs a display signal for displaying the display image; A display control device comprising:

2. The display image includes a maximum reach line that is a projection of a position that is farthest from a center position of the work machine within the reachable range of the blade tip onto the projection surface. The display control device according to claim 1 .

3. The display image includes a distance image showing the distance from a center position of the work machine to each of a plurality of positions that are in the vicinity of the line and away from the center position of the work machine. The display control device according to claim 1 or 2.

4. The distance image includes a plurality of lines spaced apart from the central position by a unit distance. The display control device according to claim 3 .

5. The distance image includes a numerical value indicating a distance from the center position. The display control device according to claim 3 or 4.

6. The projection plane is a plane passing through the contact surface of the work machine. The display control device according to any one of claims 1 to 5.

7. A display method for displaying an image used for operating a work machine having a work implement, comprising: acquiring an image showing the work machine from a camera provided on the work machine; generating a display image by superimposing an image representing the end position of the reachable range of the blade tip under a condition where the projection surface and the blade tip of the work machine are in contact with each other, and a line passing through a point where a specific part of the blade tip is projected onto the projection surface and extending in the front-rear direction of the work machine along the projection surface; displaying the display image; A display method comprising:

Images