A display system for a work machine, a remote control system for a work machine, a work machine, and a method for displaying a work machine.

Abstract

To suppress lowering of work efficiency.SOLUTION: A display system of a work machine includes: a three-dimensional data acquisition part for acquiring three-dimensional data of a work object of a ripper work machine which is connected to a vehicle body of the work machine; an image data acquisition part for acquiring a work object image indicating an image of the work object; a contact position calculation part for calculating a contact position with which a ripper point is brought into contact in a work object, on the basis of the three-dimensional data, an initial position of a ripper blade edge provided on a shank of the ripper work machine and a moving route of the ripper point; a contact position image generation part for generating a contact position image indicating the contact position; and a display control part for synthesizing the work object image with the contact position image, and displaying the synthesized image on a display device.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to a display system for a work machine, a remote control system for a work machine, a work machine, and a display method for a work machine.

Background Art

[0002] In the technical field related to display systems for work machines, a display system as disclosed in Patent Document 1 is known.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In remote control of a work machine, if it becomes difficult for an operator to recognize the relative position between the work machine and a work target, work efficiency may decrease. Further, even when an operator aboard the work machine operates the work implement, it may be difficult to grasp the distance between the work implement and the work target depending on the skill level of the operator, and work efficiency may also decrease.

[0005] An object of the present disclosure is to suppress a decrease in work efficiency.

Means for Solving the Problems

[0006] According to this disclosure, a display system for a work machine is provided, comprising: a 3D data acquisition unit that acquires 3D data of a work target of a ripper work machine connected to the body of the work machine; an image data acquisition unit that acquires a work target image showing an image of the work target; a contact position calculation unit that calculates the contact position where the ripper point contacts the work target based on the 3D data, the initial position of the ripper point provided on the shank of the ripper work machine, and the movement path of the ripper point; a contact position image generation unit that generates a contact position image showing the contact position; and a display control unit that combines the work target image and the contact position image and displays them on a display device. [Effects of the Invention]

[0007] According to this disclosure, the decline in work efficiency is suppressed. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a schematic diagram showing a remote control system for a work machine according to the first embodiment. [Figure 2] Figure 2 is a side view showing a work machine according to the first embodiment. [Figure 3] Figure 3 is a rear view showing the work machine according to the first embodiment. [Figure 4] Figure 4 is a block diagram showing a remote control system including a display system according to the first embodiment. [Figure 5] Figure 5 is a diagram illustrating the method for calculating the contact position according to the first embodiment. [Figure 6] Figure 6 is a diagram illustrating the method for calculating the contact position according to the first embodiment. [Figure 7] Figure 7 shows an example of display data shown on the display device according to the first embodiment. [Figure 8] Figure 8 is a flowchart showing the method for generating 3D map data according to the first embodiment. [Figure 9] Figure 9 is a flowchart showing the display method during operation of a ripper work machine according to the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating a work target image and a contact position image displayed on the display device according to the first embodiment. [Figure 11] FIG. 11 is a diagram illustrating a contact position image according to a modification of the first embodiment. [Figure 12] FIG. 12 is a diagram illustrating a contact position image according to a modification of the first embodiment. [Figure 13] FIG. 13 is a diagram illustrating a contact position image according to a modification of the first embodiment. [Figure 14] FIG. 14 is a block diagram illustrating a remote operation system including a display system according to the second embodiment. [Figure 15] FIG. 15 is a diagram illustrating a work target image and a contact position image displayed on the display device according to the second embodiment. [Figure 16] FIG. 16 is a diagram illustrating a contact position image according to a modification of the second embodiment. [Figure 17] FIG. 17 is a block diagram illustrating a computer system according to an embodiment. [Description of Embodiments]

[0009] Hereinafter, embodiments according to the present disclosure will be described with reference to the drawings, and the present disclosure is not limited to the embodiments. The constituent elements of the embodiments described below may be combined as appropriate. In addition, some constituent elements may not be used.

[0010] In the embodiments, the positional relationship of each part will be described using the terms "left", "right", "front", "rear", "upper", and "lower". These terms indicate relative positions or directions referenced to the center of the work machine 10.

[0011] [First Embodiment] The first embodiment will be described.

[0012] <Remote Operation System> FIG. 1 is a schematic diagram illustrating a remote control system 100 for a work machine 10 according to the present embodiment. The remote control system 100 remotely operates the work machine 10 that performs work at a work site.

[0013] At least a part of the remote control system 100 is disposed in a remote control room 200. The remote control room 200 is installed at a remote location away from the work site. The remote control system 100 includes a remote control device 140, a display device 120, and a controller 110.

[0014] The remote control device 140 is disposed in the remote control room 200. The remote control device 140 is operated by an operator in the remote control room 200. The operator performs an operation for generating an operation signal that moves a ripper work implement 11 up, down, forward, and backward. The operator performs an operation for generating an operation signal that causes a traveling body 3 to travel in a forward direction and a reverse direction. The operator performs an operation for generating an operation signal that operates a lift cylinder 6 to move an excavating blade 2 in the up-down direction. The operator performs an operation for generating an operation signal that operates a tilt cylinder 5 to change the inclination of the excavating blade. The operator can operate the remote control device 140 while being seated on a control seat 150.

[0015] The display device 120 is disposed in the remote control room 200. The display device 120 displays an image of the work site. The operator in the remote control room 200 can visually recognize the situation of the work site via the display device 120.

[0016] The operator operates the remote control device 140 while viewing the image of the work site displayed on the display device 120. The work machine 10 is remotely operated by the remote control device 140.

[0017] The controller 110 is disposed in the remote control room 200. The controller 110 includes a computer system.

[0018] The work machine 10 includes a controller 80. The controller 80 includes a computer system.

[0019] Controller 110 and Controller 80 communicate via a communication system 400. Examples of the communication system 400 include the internet, a local area network (LAN), a mobile phone network, or a satellite network. The communication system 400 may also include relay stations that relay the data being communicated.

[0020] <Working machinery> Figure 2 is a side view showing the work machine 10 according to this embodiment. Figure 3 is a rear view showing the work machine 10 according to this embodiment. In this embodiment, the work machine 10 is assumed to be a bulldozer. In the following description, the work machine 10 will be referred to as bulldozer 10 as appropriate.

[0021] As shown in Figures 2 and 3, the bulldozer 10 comprises a vehicle body 1, an excavation work machine 20, a vehicle body 3, a ripper work machine 11, a position sensor 30, a 3D sensor 40, a camera 50, and a controller 80.

[0022] The vehicle body 1 has a cab 9 and an engine compartment 8. The cab 9 is located at the top and rear of the vehicle body 1. The engine compartment 8 is located in front of the cab 9. The engine 21 is housed in the engine compartment 8. The engine 21 is the power source for the bulldozer 10. An engine hood 22 is located above the engine 21. A fuel tank 26 is located at the rear of the vehicle body 1. The fuel tank 26 is located behind the cab 9. Fuel is stored inside the fuel tank 26. Fuel is supplied from the fuel tank 26 to the engine 21.

[0023] A workbench 23 is positioned above the fuel tank 26. The workbench 23 is located behind the cab 9. A guardrail 24 is attached to the workbench 23. Workers can perform tasks on the workbench 23. Examples of tasks that workers may perform include cleaning the windows of the cab 9 or replacing the filter of the fuel tank 26.

[0024] The excavating machine 20 performs excavation or leveling work on the target area. The excavating machine 20 is connected to the vehicle body 1. At least a portion of the excavating machine 20 is positioned in front of the vehicle body 1. The excavating machine 20 has an excavation blade 2, a frame 4, a tilt cylinder 5, and a lift cylinder 6.

[0025] The digging blade 2 is positioned in front of the vehicle body 1. The digging blade 2 has a cutting edge 2C. The frame 4 supports the digging blade 2. One end of the frame 4 is connected to the rear of the digging blade 2 via a pivot mechanism. The other end of the frame 4 is connected to the side of the vehicle body 3 via a pivot mechanism. The frame 4 includes a first frame 4 connected to the left side of the rear of the digging blade 2 and a second frame 4 connected to the right side of the rear of the digging blade 2.

[0026] The tilt cylinder 5 and the lift cylinder 6 each operate the drilling blade 2. The tilt cylinder 5 is driven to tilt the drilling blade 2. The lift cylinder 6 is driven to move the drilling blade 2 up and down. One end of the tilt cylinder 5 is connected to the back of the drilling blade 2 via a pivot mechanism. The other end of the tilt cylinder 5 is connected to the top surface of the frame 4. The tilt angle of the drilling blade 2 changes as the tilt cylinder 5 extends and retracts. One end of the lift cylinder 6 is connected to the back of the drilling blade 2 via a pivot mechanism. The middle part of the lift cylinder 6 is connected to the side of the vehicle body 1. The drilling blade 2 moves up and down as the lift cylinder 6 extends and retracts.

[0027] The vehicle body 3 supports the vehicle body 1 and moves. The vehicle body 3 has a pair of tracks 3C. The bulldozer 10 moves as the tracks 3C rotate.

[0028] The ripper implement 11 performs ripping operations, such as cutting or crushing the workpiece. For example, it can be used for crushing rocks. The ripper implement 11 is connected to the vehicle body 1. At least a portion of the ripper implement 11 is positioned at the rear of the vehicle body 1. The ripper implement 11 includes a shank 12, a ripper point 13, a ripper arm 14, a tilt cylinder 15, a lift cylinder 16, and a beam 17.

[0029] The shank 12 is located at the rear of the vehicle body 1. The shank 12 has a ripper point 13. The ripper point 13 is located at the tip (lower end) of the shank 12, including the cutting edge position of the ripper. The ripper arm 14 supports the shank 12. The ripper arm 14 connects the vehicle body 1 and the shank 12. One end of the ripper arm 14 is connected to the rear of the vehicle body 1 via a pivot mechanism. The other end of the ripper arm 14 is connected to a beam 17. The beam 17 is rotatably connected to the ripper arm 14. The shank 12 is connected to the ripper arm 14 via the beam 17.

[0030] The tilt cylinder 15 and the lift cylinder 16 each operate the shank 12. The tilt cylinder 15 and the lift cylinder 16 are each connected to the vehicle body 1. The tilt cylinder 15 drives the shank 12 to tilt. The lift cylinder 6 drives the shank 12 to move up and down. One end of the tilt cylinder 15 is connected to the beam 17 via a pivot mechanism. The other end of the tilt cylinder 15 is connected to the rear of the vehicle body 1. As the tilt cylinder 15 extends and retracts, the tilt angle of the shank 12 changes. The tilt cylinder 15 moves the ripper point 13 in the front-rear direction. One end of the lift cylinder 16 is connected to the beam 17 via a pivot mechanism. The other end of the lift cylinder 16 is connected to the rear of the vehicle body 1. As the lift cylinder 16 extends and retracts, the shank 12 moves in the up-and-down direction. The lift cylinder 16 moves the shank 12, including the ripper point 13, in the up-and-down direction.

[0031] The ripper implement 11 pierces the workpiece with its ripper point 13. While the traveling body 3 is stopped, the ripper point 13 comes into contact with the workpiece, and as the traveling body 3 moves with the ripper point 13 embedded in the workpiece, the workpiece is cut or crushed. While the traveling body 3 is moving, the ripper implement 11 is moved in the up, down, forward, and backward directions, bringing it into contact with the workpiece, and cutting or crushing occurs as the traveling body 3 moves with the workpiece embedded in it.

[0032] The position sensor 30 detects the position of the bulldozer 10. The position sensor 30 uses the Global Navigation Satellite System (GNSS) to detect the position of the bulldozer 10 in a global coordinate system. The position sensor 30 includes a GNSS receiver installed on the vehicle body 1.

[0033] The 3D sensor 40 detects the distance to the surface of the object to be detected. The 3D sensor 40 detects 3D data of the surface of the object by detecting the relative distance to each of the multiple detection points on the surface of the object. The 3D data includes point cloud data consisting of multiple detection points. The 3D data includes the relative distance and relative position between the 3D sensor 40 and each of the multiple detection points defined on the object to be detected. The 3D sensor 40 detects 3D data that shows the 3D shape of the object to be detected. An example of the 3D sensor 40 is a laser sensor (LIDAR: Light Detection and Ranging) that detects the object by emitting laser light. The 3D sensor 40 may also be an infrared sensor that detects the object by emitting infrared light or a radar sensor (RADAR: Radio Detection and Ranging) that detects the object by emitting radio waves. The 3D sensor 40 may also be a 3D camera such as a stereo camera.

[0034] The 3D sensor 40 detects 3D data of the bulldozer 10's work site. The 3D data of the work site includes 3D data of the work site's terrain (shape of the ground). The 3D data of the work site also includes 3D data of the target of the ripper work machine 11.

[0035] Camera 50 captures images of the object to be imaged. Camera 50 is a two-dimensional camera, such as a monocular camera. Camera 50 may be a visible light camera or an infrared camera. The images acquired by camera 50 may be videos or still images. If the three-dimensional sensor 40 is a stereo camera, the stereo camera may be designated as camera 50.

[0036] Camera 50 captures images of the work site of the bulldozer 10 and acquires images of the work site. The images of the work site include images of the terrain (shape of the ground surface) of the work site. The images of the work site include images of the work target of the ripper implement 11. The images of the work site include images of at least a part of the ripper implement 11.

[0037] Multiple 3D sensors 40 are arranged on the bulldozer 10. In this embodiment, four 3D sensors 40 are arranged on the bulldozer 10. The 3D sensors 40 include a first 3D sensor 41, a second 3D sensor 42, a third 3D sensor 43, and a fourth 3D sensor 44.

[0038] Multiple cameras 50 are arranged on the bulldozer 10. In this embodiment, four cameras 50 are arranged on the bulldozer 10. Camera 50 includes a first camera 51, a second camera 52, a third camera 53, and a fourth camera 54.

[0039] The 3D sensor 41 detects objects in front of the vehicle body 1. The camera 51 captures images of objects in front of the vehicle body 1. At least a portion of the detection range of the 3D sensor 41 and the imaging range of the camera 51 overlap. The 3D sensor 41 and the camera 51 may be arranged adjacent to each other or at separate locations. In this embodiment, the 3D sensor 41 is supported by a support member 7 that protrudes upward from the engine hood 22. The camera 51 is located in the center of the vehicle width of the vehicle body 1 and is installed above the cab 9.

[0040] The 3D sensor 42 detects objects to be detected behind the vehicle body 1. The camera 52 captures images of objects to be imaged behind the vehicle body 1. At least a portion of the detection range of the 3D sensor 42 and the imaging range of the camera 52 overlap. The 3D sensor 42 and the camera 52 are arranged adjacent to each other. In this embodiment, the 3D sensor 42 and the camera 52 are located at the rear of the cab 9.

[0041] The 3D sensor 43 detects at least a part of the ripper workpiece 11 and the workpiece that the ripper workpiece 11 is working on. The 3D sensor 43 detects at least the ripper point 13 and the workpiece into which the ripper point 13 penetrates. The camera 53 images at least a part of the ripper workpiece 11 and the workpiece that the ripper workpiece 11 is working on. The camera 53 images at least the ripper point 13 and the workpiece into which the ripper point 13 penetrates. The 3D sensor 43 and the camera 53 are arranged adjacent to each other. In this embodiment, the 3D sensor 43 and the camera 53 are located at the rear of the vehicle body 1. The 3D sensor 43 and the camera 53 are located in the center of the vehicle body 1 in the left-right direction.

[0042] The 3D sensor 44 detects at least a portion of the ripper workpiece 11 and the workpiece the ripper workpiece 11 is working on. The 3D sensor 44 detects at least the ripper point 13 and the workpiece into which the ripper point 13 penetrates. The camera 54 images at least a portion of the ripper workpiece 11 and the workpiece the ripper workpiece 11 is working on. The camera 54 images at least the ripper point 13 and the workpiece into which the ripper point 13 penetrates. The detection range of the 3D sensor 44 and the imaging range of the camera 54 overlap by at least a portion of each other. The 3D sensor 44 and the camera 54 are positioned adjacent to each other. The 3D sensor 44 detects at least a portion of the ripper workpiece 11 and the workpiece the ripper workpiece 11 is working on from a different direction than the 3D sensor 43. The camera 54 detects at least a portion of the ripper workpiece 11 and the workpiece the ripper workpiece 11 is working on from a different direction than the camera 53. In this embodiment, the 3D sensor 44 and the camera 54 are positioned at the rear left side of the vehicle body 1.

[0043] <Display System> Figure 4 is a block diagram showing a remote control system 100 including a display system 300 according to this embodiment. As shown in Figure 4, the remote control system 100 includes a communication device 401 located in a remote control room 200, a controller 110 connected to the communication device 401, a remote control device 140 connected to the controller 110, and a display device 120 connected to the controller 110. The remote control system 100 also includes a communication device 402 located on the bulldozer 10, a controller 80 connected to the communication device 402, a position sensor 30 connected to the controller 80, a three-dimensional sensor 40 connected to the controller 80, a camera 50 connected to the controller 80, a traveling body 3 controlled by the controller 80, an excavation work machine 20 controlled by the controller 80, and a ripper work machine 11 controlled by the controller 80.

[0044] In the example shown in Figure 4, the remote control device 140 and the display device 120 are connected to the same controller 110, but they may be separate controllers. Similarly, for the on-board controller 80, the controllers connected to the position sensor 30, the 3D sensor 40, and the camera 50 may be separate controllers from the controllers that control the vehicle 3, the excavating machine 20, and the ripper machine 11.

[0045] The controller 80 includes a vehicle control unit 81, an excavation work machine control unit 82, a ripper work machine control unit 83, a position data transmission unit 84, a 3D data transmission unit 85, and an image data transmission unit 86.

[0046] The vehicle control unit 81 receives operation signals from the remote control device 140 transmitted from the controller 110. Based on the operation signals from the remote control device 140, the vehicle control unit 81 outputs a control signal to control the vehicle 3.

[0047] The excavation work machine control unit 82 receives operation signals from the remote control device 140 transmitted from the controller 110. Based on the operation signals from the remote control device 140, the excavation work machine control unit 82 outputs control signals to control the excavation work machine 20. The control signals to control the excavation work machine 20 include control signals to control the tilt cylinder 5 and the lift cylinder 6.

[0048] The ripper work machine control unit 83 receives operation signals from the remote control device 140 transmitted from the controller 110. Based on the operation signals from the remote control device 140, the ripper work machine control unit 83 outputs control signals to control the ripper work machine 11. The control signals to control the ripper work machine 11 include control signals to control the tilt cylinder 15 and the lift cylinder 16.

[0049] The position data transmission unit 84 transmits the position of the bulldozer 10 detected by one or more position sensors 30 to the controller 110.

[0050] The 3D data transmission unit 85 transmits the 3D data detected by the 3D sensor 40 to the controller 110.

[0051] The image data transmission unit 86 transmits the image captured by the camera 50 to the controller 110.

[0052] Communication device 402 communicates with communication device 401 via communication system 400. Communication device 402 receives operation signals from the remote control device 140 transmitted from the controller 110 via communication device 401 and outputs them to the controller 80. Communication device 402 transmits the position received from the position data transmission unit 84, the 3D data received from the 3D data transmission unit 85, and the image data received from the image data transmission unit 86 to the remote communication device 401.

[0053] Communication device 401 communicates with communication device 402 via communication system 400. Communication device 401 transmits operation signals generated when the remote control device 140 is operated to communication device 402. Communication device 401 receives position data, 3D data, and images transmitted from controller 80 via communication device 402 and outputs them to controller 110.

[0054] The controller 110 includes an operation signal transmission unit 101, a position data acquisition unit 102, a 3D data acquisition unit 103, an image data acquisition unit 104, an initial position determination unit 105, a contact position calculation unit 106, a contact position image generation unit 107, a map data generation unit 108, a display control unit 109, and a map data storage unit 111.

[0055] The map data generation unit 108 may also be mounted on the in-vehicle controller 80.

[0056] The operation signal transmission unit 101 transmits an operation signal to remotely control the bulldozer 10. When the remote control device 140 is operated by the operator, an operation signal to remotely control the bulldozer 10 is generated. The operation signal transmission unit 101 transmits the operation signal from the remote control device 140 to the controller 80.

[0057] The position data acquisition unit 102 acquires the position of the bulldozer 10 detected by the position sensor 30 from the controller 80.

[0058] The 3D data acquisition unit 103 acquires 3D data of the work target of the ripper work machine 11, detected by the 3D sensor 40, from the controller 80. The 3D data may be data acquired in the past before the start of work, or 3D data acquired in advance by a drone or the like.

[0059] The image data acquisition unit 104 acquires from the controller 80 an image of the work target of the ripper work machine 11, which has been captured by the camera 50.

[0060] The initial position determination unit 105 determines the initial position of the ripper point 13 provided on the shank 12 of the ripper work implement 11. The initial position of the ripper point 13 includes the initial position of the tip 13T of the ripper point 13. In this embodiment, the initial position determination unit 105 determines the initial position of the ripper point 13 based on a predetermined initial posture of the ripper work implement 11. In this embodiment, the initial posture of the ripper work implement 11 is set to the posture in which the lift cylinder 16 is fully extended within the range of motion of the lift cylinder 16 and the tilt cylinder 15 is fully retracted within the range of motion of the tilt cylinder 15. That is, in this embodiment, the initial posture of the ripper work implement 11 is set to the posture in which the tip 13T of the ripper point 13 is positioned as far upward and as far back as possible within the range of motion of the ripper point 13. In the following description, the initial posture of the ripper work implement 11 will be appropriately referred to as the maximum tilt-back posture.

[0061] The contact position calculation unit 106 calculates the contact position at which the tip 13T of the ripper point 13 makes contact with the work object, based on the 3D data of the work object, the initial position of the ripper point 13 determined by the initial position determination unit 105, and the movement path of the tip 13T of the ripper point 13.

[0062] Figures 5 and 6 are diagrams illustrating the method for calculating the contact position Rp according to this embodiment. Figure 5 shows an example in which the ground surface in contact with the ground contact surface of the vehicle 3 and the work target OB are substantially located in the same plane. Figure 6 shows an example in which the work target OB is inclined with respect to the ground surface in contact with the ground contact surface of the vehicle 3. In both Figures 5 and 6, the 3D sensor 40 can detect 3D data of the work target OB in the local coordinate system defined for the bulldozer 10.

[0063] When performing a ripper operation in which the ripper point 13 is driven into the work target OB, the operator often operates the remote control device 140 so that the lift cylinder 16 retracts while the tilt cylinder 15 does not extend or retract, after the ripper implement 11 has been tilted back to its maximum position. In other words, the operation of the ripper implement 11 in a ripper operation is generally standardized.

[0064] Based on the operation signal from the remote control device 140, the ripper work implement control unit 83 retracts the lift cylinder 16 from the ripper work implement 11's maximum tilt-back position without driving the tilt cylinder 15. As the ripper work implement 11 retracts the lift cylinder 16 from its maximum tilt-back position, the ripper point 13 moves downward and approaches the work target OB.

[0065] The movement path Tr of the tip 13T of the ripper point 13 from the maximum tilt-back position of the ripper work implement 11 until the ripper point 13 penetrates the work target OB can be pre-determined based on the structure of the ripper work implement 11. The ripper point 13 makes contact with the work target by following the movement path Tr, which is either a semicircular or elliptical trajectory. As described above, the operation of the ripper work implement 11 in ripper work is generally standardized. In ripper work where the ripper point 13 penetrates the work target OB, the lift cylinder 16 is driven from the maximum tilt-back position of the ripper work implement 11, so the contact position calculation unit 106 can pre-determine the movement path Tr based on the structure of the ripper work implement 11. The contact position calculation unit 106 can calculate the contact position Rp of the tip 13T of the ripper point 13 on the surface of the work target OB based on the three-dimensional data of the surface of the work target OB, the initial position of the tip 13T, and the previously derived movement path Tr of the tip 13T.

[0066] The contact position image generation unit 107 generates a contact position image showing the contact position Rp calculated by the contact position calculation unit 106.

[0067] The map data generation unit 108 generates 3D map data of the work site where the bulldozer 10 is working. The 3D map data includes 3D data of the work target OB. The map data generation unit 108 can generate 3D map data of the work site based on the position of the bulldozer 10 acquired by the position data acquisition unit 102 and the 3D data of the work site acquired by the 3D data acquisition unit 103. The 3D map data generated by the map data generation unit 108 is stored in the map data storage unit 111.

[0068] In this embodiment, the contact position calculation unit 106 calculates the contact position Rp based on the 3D map data stored in the map data storage unit 111. Alternatively, the contact position calculation unit 106 may calculate the contact position Rp based on the 3D data of the work target OB acquired by the 3D data acquisition unit 103.

[0069] The display control unit 109 combines the work target image and the contact position image and displays them on the display device 120.

[0070] <Example of display on a display device> Figure 7 shows an example of display data shown on the display device 120 according to this embodiment.

[0071] The display control unit 109 sets multiple split display screens on the display device 120. As shown in Figure 7, the display device 120 is set to have multiple split display screens: a front display screen 121, a rear display screen 122, a first ripper display screen 123, and a second ripper display screen 124. The front display screen 121 displays an image captured by camera 51. The rear display screen 122 displays an image captured by camera 52. The first ripper display screen 123 displays an image captured by camera 53. The second ripper display screen 124 displays an image captured by camera 54.

[0072] Furthermore, the display control unit 109 causes the display device 120 to display icons 131, an inclinometer 132, and a tilt indicator 133 on a portion of the display device 120. The inclinometer 132 and the tilt indicator 133 display the pitch angle and roll angle of the bulldozer 10 with respect to the horizontal plane.

[0073] <3D map data> Figure 8 is a flowchart showing the method for generating 3D map data according to this embodiment. In this embodiment, 3D map data of the work site is generated before the ripper work of the ripper machine 11 is performed.

[0074] The bulldozer 10 travels around the work site. The position of the bulldozer 10 as it travels around the work site is detected by the position sensor 30. The 3D sensor 40 detects 3D data of the work site around the bulldozer 10 as it travels around the work site. The 3D data acquisition unit 103 acquires 3D data of the work site from the 3D sensor 40. The position data acquisition unit 102 acquires the position of the bulldozer 10 when the 3D data was detected from the position sensor 30 (step SA1).

[0075] The map data generation unit 108 generates 3D map data of the work site based on the position of the bulldozer 10 acquired by the position data acquisition unit 102 and the 3D data of the work site acquired by the 3D data acquisition unit 103 (step SA2).

[0076] The 3D data detected by the 3D sensor 40 includes the relative distance and relative position between the 3D sensor 40 and each of the multiple detection points defined in the detection target. The 3D data is defined in the local coordinate system of the bulldozer 10. The position of the bulldozer 10 is defined in the global coordinate system. Therefore, the map data generation unit 108 can generate 3D map data defined in the global coordinate system based on the position of the bulldozer 10 and the 3D data of the work site acquired by the 3D data acquisition unit 103. The map data generation unit 108 can also generate 3D map data defined in the local coordinate system by performing coordinate transformations.

[0077] The map data storage unit 111 stores the 3D map data generated in step SA2 (step SA3).

[0078] The 3D map data stored in the map data storage unit 111 may be the 3D map data for the entire work site, or it may be the 3D map data for only a part of the work site.

[0079] <Display method> Figure 9 is a flowchart showing the display method during operation of the ripper work machine 11 according to this embodiment.

[0080] When the ripper machine 11 starts ripper work, the 3D data acquisition unit 103 acquires 3D data of the target OB of the ripper machine 11 from the 3D map data stored in the map data storage unit 111. Based on the detection data of the position sensor 30 when the ripper work is started, the 3D data acquisition unit 103 can extract 3D data of the target OB of the ripper machine 11 from the 3D map data of the work site stored in the map data storage unit 111. Also, when the ripper work is started, the image data acquisition unit 104 acquires image data from at least the camera 53. The image captured by the camera 53 includes at least an image of the ripper point 13 and an image of the target OB of the ripper machine 11 (step SB1).

[0081] The initial position determination unit 105 determines the initial position of the tip 13T of the ripper point 13 based on a predetermined maximum tilt-back posture (step SB2).

[0082] The contact position calculation unit 106 calculates the contact position Rp at which the tip 13T of the ripper point 13 makes contact with the work target OB, based on the 3D data of the work target OB acquired in step SB1, the initial position of the tip 13T of the ripper point 13 determined in step SB2, and the movement path Tr of the tip 13T of the ripper point 13 which is derived in advance based on the structure of the ripper work machine 11 (step SB4).

[0083] The contact position image generation unit 107 generates a contact position image showing the contact position Rp (step SB4).

[0084] The display control unit 109 combines the work target image and the contact position image and displays them on the display device 120 (step SB5).

[0085] Figure 10 shows the work target image and contact position image 61 displayed on the display device 120 according to this embodiment. As shown in Figure 10, the display control unit 109 combines the work machine image showing the ripper work machine 11, the work target image showing the work target OB, and the contact position image 61 showing the contact position Rp, and displays them on the display device 120.

[0086] The display control unit 109 combines the work machine image, the work object image, and the contact position image 61 before the ripper point 13 contacts the work object OB, and displays them on the display device 120. Combining the work object image and the contact position image 61 includes superimposing the contact position image 61 onto the work object image. The contact position image 61 is displayed so as to follow the surface of the work object OB.

[0087] In this embodiment, the contact position image 61 is displayed as a line extending in the left-right direction. The dimensions of the contact position image 61 in the left-right direction may be equal to, for example, the dimensions of the ripper point 13.

[0088] In this embodiment, the display control unit 109 causes the display device 120 to display a grid image 65 along the surface of the work target OB. The grid image 65 includes a plurality of first line images 66 and a plurality of second line images 67. The plurality of first line images 66 are displayed with spacing in the left-right direction. The plurality of second line images 67 are displayed with spacing in the front-back direction. The first line images 66 and the second line images 67 intersect.

[0089] <Effects> As described above, in this embodiment, the display system 300 of the bulldozer 10 includes a 3D data acquisition unit 103 that acquires 3D data of the work target OB of the ripper work implement 11 connected to the body 1 of the bulldozer 10, an image data acquisition unit 104 that acquires a work target image showing an image of the work target OB, a contact position calculation unit 106 that calculates the contact position Rp at the work target OB where the ripper point 13 makes contact based on the 3D data, the initial position of the ripper point 13 provided on the shank 12 of the ripper work implement 11, and the movement path Tr of the ripper point 13, a contact position image generation unit 107 that generates a contact position image 61 showing the contact position Rp, and a display control unit 109 that combines the work target image and the contact position image 61 and displays them on the display device 120.

[0090] According to this embodiment, before the ripper point 13 contacts the work target OB, the contact position Rp at which the ripper point 13 will contact the work target OB is calculated, and a contact position image 61 indicating the contact position Rp is displayed on the display device 120 superimposed on the work target image. Therefore, the operator remotely controlling the bulldozer 10 can recognize the contact position of the ripper point 13 by looking at the contact position of the contact position image 61 relative to the work target OB before the ripper point 13 contacts the work target OB. Furthermore, by looking at the distance between the contact position of the ripper point 13 and the contact position image 61, the operator can recognize the relative position between the ripper point 13 and the work target OB. This suppresses a decrease in work efficiency.

[0091] In this embodiment, the display system 300 includes an initial position determination unit 105 that determines the initial position of the ripper point 13. The initial position determination unit 105 determines the initial position of the ripper point 13 based on a predetermined initial posture of the ripper work machine 11.

[0092] In ripper operations, where the ripper point 13 is driven into the work target OB, the ripper implement 11 is often operated to its maximum tilt-back position as its initial posture. The initial position determination unit 105 can determine the initial position of the ripper point 13 based on a predetermined maximum tilt-back posture of the ripper implement 11. For example, the initial position of the ripper point 13 can be determined without installing a posture sensor on the bulldozer 10 to detect the initial posture of the ripper implement 11.

[0093] In this embodiment, the contact position calculation unit 106 calculates the contact position Rp based on a pre-determined movement path Tr.

[0094] In ripper operations, after the ripper implement 11 is operated to its maximum tilt-back position, the tilt cylinder 15 is often not driven, and the lift cylinder 16 is driven instead. Therefore, the movement path Tr can be derived in advance based on the structure of the ripper implement 11. The contact position calculation unit 106 can calculate the contact position Rp based on the pre-determined movement path Tr.

[0095] In this embodiment, the map data generation unit 108 generates 3D map data of the work site. The contact position calculation unit 106 extracts 3D data of the work target OB from the 3D map data stored in the map data storage unit 111 and calculates the contact position Rp. This allows the contact position calculation unit 106 to calculate the contact position Rp appropriately. In ripper work, if the contact position Rp is calculated based on 3D data transmitted in real time from the 3D sensor 40, a delay in the communication system 400 may make it difficult for the contact position calculation unit 106 to calculate the contact position Rp appropriately. In this embodiment, the 3D map data of the work site is generated in advance before the start of ripper work, and during ripper work, the 3D data of the work target OB is extracted from the 3D map data stored in the map data storage unit 111, allowing the contact position calculation unit 106 to calculate the contact position Rp appropriately.

[0096] Examples of causes of delay in the communication system 400 include sudden changes in the vehicle's direction, inability to acquire terrain information directly beneath the vehicle using sensors, and the use of data acquired by the vehicle's front camera and front sensors as data for the ripper when the vehicle is moving forward.

[0097] <Variation> Figure 11 shows a contact position image 61B according to a modified example of this embodiment. As shown in Figure 11, the contact position image 61B may be displayed as a dot to indicate the contact position Rp of the tip portion 13T. The display form is not limited to dots and may be any shape such as a circle, cross, or ellipse.

[0098] Figure 12 shows a contact position image 61C according to a modified example of this embodiment. In the example shown in Figure 12, the ripper work machine 11B has a plurality of ripper points 13 arranged in the left-right direction. In the example shown in Figure 12, three ripper points 13 are arranged with intervals between them in the left-right direction. The contact position image 61C may be displayed as a line connecting each of the contact positions Rp of the plurality of tip portions 13T.

[0099] Figure 13 shows a contact position image 61D according to a modified example of this embodiment. In the example shown in Figure 13, the ripper work machine 11B has a plurality of ripper points 13 arranged in the left-right direction. In the example shown in Figure 13, three ripper points 13 are arranged with intervals between them in the left-right direction. The contact position image 61D may be displayed as dots to indicate each of the contact positions Rp of the plurality of tip portions 13T. The display form is not limited to dots and may be any shape such as a circle, cross, or ellipse.

[0100] [Second Embodiment] A second embodiment will now be described. In the following description, components that are the same as or equivalent to those in the above-described embodiment will be denoted by the same reference numerals, and the descriptions of those components will be simplified or omitted.

[0101] Figure 14 is a block diagram showing a remote control system 100 including a display system 300 according to this embodiment. In this embodiment, a posture sensor 31 is provided on the bulldozer 10. The posture sensor 31 detects the posture of the ripper work machine 11. The posture sensor 31 can detect the initial posture of the ripper work machine 11. Examples of posture sensors 31 include stroke sensors provided on the tilt cylinder 15 and the lift cylinder 16, respectively. The stroke sensor provided on the tilt cylinder 15 detects a first stroke length indicating the amount of extension or retraction of the tilt cylinder 15. The stroke sensor provided on the lift cylinder 16 detects a second stroke length indicating the amount of extension or retraction of the lift cylinder 16. Note that the posture sensor 31 may also be, for example, an inertial measurement unit (IMU) provided on the shank 12.

[0102] The controller 80 has a posture data transmission unit 87 that transmits detection data from the posture sensor 31 to the controller 110. The posture data transmission unit 87 transmits initial posture data to the controller 110 that indicates the initial posture of the ripper work machine 11 detected by the posture sensor 31. The initial posture may be the maximum tilt-back posture or the ripper point 13 during ripping work.

[0103] In this embodiment, a sensor capable of measuring the ripper posture is installed, and the position of the ripper point 13 can be confirmed from the sensor. Therefore, the initial posture may be the ripper point 13 during the ripping operation, regardless of the maximum tilt-back posture described in paragraph 0058.

[0104] In this embodiment, the initial position determination unit 105 determines the initial position of the ripper point 13 based on the detection data from the attitude sensor 31 that detects the initial posture of the ripper work machine 11. The contact position calculation unit 106 calculates the contact position Rp at which the ripper point 13 makes contact with the work target OB based on the three-dimensional data of the work target OB, the initial position of the ripper point 13 determined based on the detection data from the attitude sensor 31, and the movement path Tr of the ripper point 13 derived from the structure of the ripper work machine 11 and the extension and retraction operation of the lift cylinder 16.

[0105] In this embodiment, the controller 110 has a guide image generation unit 112 that generates a guide image 62 to be displayed between at least a portion of the shank 12 and the work object OB based on the detection data of the attitude sensor 31. The guide image 62 is a line image connecting the work object and the ripper point 13. The line segments of the guide image 62 may be lines extending vertically from the ripper point to the work object, or lines tangential to the ripper point and directed towards the ripper point 13 and the work object surface. The display control unit 109 combines the work object image, the guide image, and the contact position image and displays them on the display device 120.

[0106] Figure 15 shows the work target image and contact position image 61 displayed on the display device 120 according to this embodiment. As shown in Figure 15, the guide image generation unit 112 generates a first guide image 62 and a second guide image 63. Each of the guide image 62 and guide image 63 is linear. At least a portion of the guide image 62 is displayed between the right end of the ripper point 13 and the position where the ripper cutting edge on the surface of the work target OB makes contact. At least a portion of the guide image 63 is displayed between the left end of the ripper point 13 and the position where the ripper cutting edge on the surface of the work target OB makes contact.

[0107] In this embodiment, the contact position image 61 is displayed as a line extending in the left-right direction. The dimensions of the contact position image 61 in the left-right direction are equal to the dimensions of the ripper point 13. The guide image 62 is displayed connecting the right end of the ripper point 13 and the right end of the contact position image 61. The guide image 63 is displayed connecting the left end of the ripper point 13 and the left end of the contact position image 61.

[0108] Based on the position of the ripper point 13, the guide images 62 and 63 on the display device 120 change accordingly. When the ripper point 13 approaches the work target OB, the guide images 62 and 63 become shorter. When the ripper point 13 moves away from the work target OB, the guide images 62 and 63 become longer. In this embodiment, the posture of the ripper work machine 11 is detected by the posture sensor 31. The guide image generation unit 112 can calculate the relative position between the ripper point 13 and the work target OB based on the detection data from the posture sensor 31 and the 3D data of the work target OB. Based on the position of the ripper point 13 relative to the work target OB, the guide image generation unit 112 can change the length of the guide image 62 so that the guide image 62 continues to connect the right end of the ripper point 13 and the right end of the contact position image 61. Similarly, the guide image generation unit 112 can change the length of the guide image 63 based on the position of the ripper point 13 relative to the work target OB, so that the guide image 63 continues to connect the left end of the ripper point 13 with the right end of the contact position image 61.

[0109] <Effects> As described above, in this embodiment, a posture sensor 31 is provided to detect the initial posture of the ripper workpiece 11. The initial position determination unit 105 determines the initial position of the ripper point 13 based on the detection data from the posture sensor 31. As a result, even if the ripper workpiece 11 is not operated to the maximum tilt-back position, the initial position determination unit 105 can determine the initial position of the ripper point 13 based on the initial posture of the ripper workpiece 11 detected by the posture sensor 31.

[0110] In this embodiment, the guidance image generation unit 112 generates guidance images 62 and 63 based on the detection data of the attitude sensor 31. The guidance image generation unit 112 can calculate the relative position between the ripper point 13 and the work target OB based on the detection data of the attitude sensor 31 and the 3D data of the work target OB. The guidance image generation unit 112 can change the guidance images 62 and 63 based on the position of the ripper point 13 relative to the work target OB.

[0111] <Variation> Figure 16 shows a contact position image 61C according to a modified example of this embodiment. Similar to the example shown in Figure 12, the ripper work machine 11B has a plurality of ripper points 13 arranged in the left-right direction. The contact position image 61C is displayed linearly, connecting each of the contact positions Rp of the plurality of tip portions 13T. The guide image 62C is displayed connecting the right end of the rightmost ripper point 13 among the plurality of ripper points 13 with the right end of the contact position image 61C. The guide image 63C is displayed connecting the left end of the leftmost ripper point 13 among the plurality of ripper points 13 with the left end of the contact position image 61C.

[0112] [Computer System] Figure 17 is a block diagram showing a computer system 1000 according to an embodiment. Each of the controllers 110 and 80 described above includes the computer system 1000. The computer system 1000 includes a processor 1001 such as a CPU (Central Processing Unit), a main memory 1002 including non-volatile memory such as ROM (Read Only Memory) and volatile memory such as RAM (Random Access Memory), a storage 1003, and an interface 1004 including input / output circuits. The respective functions of the controllers 110 and 80 described above are stored in the storage 1003 as computer programs. The processor 1001 reads the computer program from the storage 1003, loads it into the main memory 1002, and executes the above-described processing according to the program. The computer program may be distributed to the computer system 1000 via a network.

[0113] The computer program or computer system 1000 can perform the following actions in accordance with the above-described embodiment: acquire three-dimensional data of the work target OB of the ripper work implement 11 attached to the body 1 of the bulldozer 10; acquire a work target image showing an image of the work target OB; calculate the contact position Rp at which the ripper point 13 contacts the work target OB based on the three-dimensional data, the initial position of the ripper point 13 provided on the shank 12 of the ripper work implement 11, and the movement path Tr of the ripper point 13; generate a contact position image 61 showing the contact position Rp; and combine the work target image and the contact position image 61 and display them on the display device 120.

[0114] [Other embodiments] In the above embodiment, 3D map data of the work site is generated in advance before the start of the ripper operation, 3D data of the work target OB is extracted from the 3D map data during the ripper operation, and the contact position Rp is calculated based on the extracted 3D data of the work target OB. The 3D map data does not necessarily have to be used in the calculation of the contact position Rp. The contact position calculation unit 106 may calculate the contact position Rp based on the 3D data of the work target OB detected in real time by the 3D sensor 40 during the ripper operation.

[0115] In the above-described embodiment, the display system 300 is applied to the remote control system 100. The display system 300 does not have to be applied to the remote control system 100, and may be mounted on the bulldozer 10. For example, the display device 120 may be located in the cab 9 of the bulldozer 10, and the functions of the controller 110 described in the above-described embodiment may be provided on the controller 80 of the bulldozer 10. The bulldozer 10 may be operated by an operator riding in the cab 9. [Explanation of symbols]

[0116] 1...Body, 2...Drilling blade, 2C...Cutting edge, 3...Running body, 3C...Tracks, 4...Frame, 5...Tilt cylinder, 6...Lift cylinder, 7...Support member, 8...Engine room, 9...Cab, 10...Bulldozer (working machine), 11...Ripper work machine, 11B...Ripper work machine, 12...Shank, 13...Ripper point, 13T...Tip section, 14...Ripper arm, 15...Tilt cylinder, 16...Lift cylinder, 17...Beam, 20...Drilling work machine, 21...Engine, 22...Engine hood, 23...Work platform, 24...Guard rail, 26...Fuel tank 30...Position sensor, 31...Attitude sensor, 40...3D sensor, 41...3D sensor, 42...3D sensor, 43...3D sensor, 44...3D sensor, 50...Camera, 51...Camera, 52...Camera, 53...Camera, 54...Camera, 61...Contact position image, 61B...Contact position image, 61C...Contact position image, 62...Guidance image, 62C...Guidance image, 63...Guidance image, 63C...Guidance image, 65...Grid image, 66...First line image, 67...Second line image, 80...Controller, 81...Traveling body control unit, 82...Excavation work machine control unit, 83...Rit 84...Work machine control unit, 85...Position data transmission unit, 86...3D data transmission unit, 87...Position data transmission unit, 100...Remote control system, 101...Operation signal transmission unit, 102...Position data acquisition unit, 103...3D data acquisition unit, 104...Image data acquisition unit, 105...Initial position determination unit, 106...Contact position calculation unit, 107...Contact position image generation unit, 108...Map data generation unit, 109...Display control unit, 110...Controller, 111...Map data storage unit, 112...Guidance image generation unit, 120...Display device, 121... Front display screen, 122... Rear display screen, 123... First ripper display screen, 124... Second ripper display screen, 131... Icon, 132... Inclinometer, 133... Inclination indicator, 140... Remote control device, 150... Pilot's seat, 200... Remote control room, 300... Display system, 400... Communication system, 401... Communication device, 402... Communication device, 1000... Computer system, 1001... Processor, 1002... Main memory, 1003... Storage, 1004... Interface, OB... Work target, Rp... Contact position, Tr... Movement path.

Claims

1. A 3D data acquisition unit that acquires 3D data of the work target of a ripper work machine attached to the body of the work machine, An image data acquisition unit that acquires an image of the work target showing the image of the work target, A contact position calculation unit calculates the contact position where the ripper point makes contact with the workpiece, based on the three-dimensional data, the initial position of the ripper point provided on the shank of the ripper workpiece, and a predetermined movement path of the ripper point. A contact position image generation unit generates a contact position image indicating the aforementioned contact position, The system includes a display control unit that combines the work target image and the contact position image and displays them on a display device. A display system for industrial machinery.

2. The display control unit combines the work object image and the contact position image before the ripper point contacts the work object and displays them on the display device. A display system for a work machine according to claim 1.

3. Combining the work target image and the contact position image includes superimposing the contact position image onto the work target image. A display system for a work machine according to claim 2.

4. The system includes an initial position determination unit that determines the initial position, The initial position determination unit determines the initial position based on a predetermined initial posture of the ripper work machine. A display system for a work machine according to claim 1.

5. The system includes an initial position determination unit that determines the initial position, The initial position determination unit determines the initial position based on the detection data from the attitude sensor that detects the initial posture of the ripper work machine. A display system for a work machine according to claim 1.

6. The system includes a guidance image generation unit that generates a guidance image to be displayed between at least a portion of the shank and the work object based on the detection data of the attitude sensor, The display control unit combines the work target image and the guidance image and displays them on the display device. A display system for a work machine according to claim 5.

7. The ripper work machine includes a lift cylinder connected to the vehicle body for moving the shank including the ripper point in the vertical direction, a tilt cylinder connected to the vehicle body for moving the ripper point in the front-rear direction, and a ripper arm connecting the vehicle body and the shank. The aforementioned travel path is derived in advance based on the structure of the ripper work machine. A display system for a work machine according to claim 1.

8. A map data generation unit generates 3D map data of the work site based on the 3D data of the work site where the work machine operates, which is acquired by the 3D data acquisition unit. The system comprises a map data storage unit that stores the three-dimensional map data generated by the map data generation unit, The aforementioned 3D map data includes the 3D data of the work target, The contact position calculation unit calculates the contact position based on the three-dimensional map data stored in the map data storage unit. A display system for a work machine according to claim 1.

9. A display system for a work machine according to any one of claims 1 to 8, The display device is located in a remote location away from the work site where the work machine is operating. A remote control system for industrial machinery.

10. A display system for a work machine according to any one of claims 1 to 8, Agricultural machinery.

11. To acquire 3D data of the workpiece of a ripper work machine attached to the body of a work machine, To obtain a work target image that shows the image of the work target, Based on the three-dimensional data, the initial position of the ripper point provided on the shank of the ripper work machine, and the predetermined movement path of the ripper point, the contact position where the ripper point contacts the work object is calculated. To generate a contact position image showing the aforementioned contact position, This includes combining the work target image and the contact position image and displaying them on a display device. How to display the names of working machines.

Images