Remote control system for work machine, remote controller for work machine, and remote control method for work machine

Abstract

To provide a remote control system capable of suppressing reduction in work efficiency of a work machine remotely controlled.SOLUTION: A remote control system is provided with a display device and a remote controller. The remote controller comprises: an image data acquisition part receiving image data captured through a camera mounted on the work machine; a display data generation part sampling the image data in a sampling area smaller than an outer shape of the image data from the image data; a movement signal receiving part receiving movement signal in order to move in the sampling area: and a display control part making the display device display the display data in the sampling area by moving in the sampling area on a basis of the movement signal.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present disclosure relates to a work machine remote control system, a work machine remote controller, and a work machine remote control method. [Background technology]

[0002] BACKGROUND ART In the technical field related to work machines, a work machine such as that disclosed in Patent Document 1 is known. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-025223 Summary of the Invention [Problem to be solved by the invention]

[0004] A remote control system for a work machine includes a display device located outside the work machine. Image data captured by a camera mounted on the work machine is displayed on the display device. An operator remotely controls the work machine while checking the image data displayed on the display device. In order to prevent a decline in work efficiency, it is necessary to provide the operator with appropriate information about the situation at the work site.

[0005] An object of the present disclosure is to suppress a decrease in the work efficiency of a remotely operated work machine. [Means for solving the problem]

[0006] According to the present disclosure, there is provided a remote operation system for a work machine including a display device and a remote controller. The remote controller has an image data acquisition unit that receives image data captured by a camera mounted on the work machine, a display data generation unit that extracts display data of an extraction range that is smaller than the outer shape of the image data from the image data, a movement signal receiving unit that receives a movement signal for moving the extraction range, and a display control unit that moves the extraction range based on the movement signal and displays the display data of the extraction range on the display device. [Effects of the Invention]

[0007] According to the present disclosure, a decrease in the work efficiency of a remotely operated work machine is suppressed. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram that schematically shows a remote control system for a work machine according to a first embodiment. [Figure 2] FIG. 2 is a side view that schematically shows the work machine according to the first embodiment. [Figure 3] FIG. 3 is a hardware configuration diagram showing the remote controller according to the first embodiment. [Figure 4] FIG. 4 is a functional block diagram showing the work machine and remote controller according to the first embodiment. [Figure 5] FIG. 5 is a flowchart showing the method for remotely controlling a work machine according to the first embodiment. [Figure 6] FIG. 6 is a diagram showing the relationship between image data and display data according to the first embodiment. [Figure 7] FIG. 7 is a diagram illustrating the normal display and the preceding moving display according to the first embodiment. [Figure 8] FIG. 8 is a diagram showing the relationship between image data and display data according to the second embodiment. [Figure 9] FIG. 9 is a diagram schematically showing a remote control room according to the third embodiment. [Figure 10]FIG. 10 is a diagram illustrating a normal display and a follow-up movement display according to the fourth embodiment. [Figure 11] FIG. 11 is a diagram illustrating the relationship between the respective operations of the revolving body and the work machine and the display data according to the fourth embodiment. [Figure 12] FIG. 12 is a diagram illustrating a normal display and a preceding moving display according to the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings, but the present disclosure is not limited to the embodiments. The components of the embodiments described below can be combined as appropriate. In addition, some components may not be used.

[0010] [First embodiment] A first embodiment will be described.

[0011] <Remote control system> FIG. 1 is a diagram schematically showing a remote operation system 1 for a work machine 2 according to an embodiment. The remote operation system 1 remotely operates a work machine 2 located at a work site 101. At least a part of the remote operation system 1 is placed in a remote operation room 102 provided outside the work machine 2. The remote operation system 1 comprises an operation device 7, a display device 8, and a remote controller 9.

[0012] The operating device 7 is arranged in a remote control room 102 outside the work machine 2. The operating device 7 is a remote operating device that includes a remote control lever. The operating device 7 is operated by an operator in the remote control room 102. The operator can operate the operating device 7 while seated in the operator's seat 10.

[0013] The display device 8 is placed in a remote control room 102 outside the work machine 2. Examples of the display device 8 include a flat panel display such as a liquid crystal display or an organic EL display. The display device 8 displays an image of the work site 101. The image of the work site 101 includes a peripheral image, which is an image of the area around the work machine 2. The peripheral image of the work machine 2 includes at least an image of the work target of the work machine 2. The display device 8 may also output audio of the work site 101.

[0014] The operator operates the operation device 7 while checking an image of the work site 101 displayed on the display device 8. The work machine 2 is remotely controlled by the operation device 7.

[0015] The remote controller 9 is placed in a remote operation room 102 outside the work machine 2. The remote controller 9 includes a computer system. The remote controller 9 and the work machine 2 communicate via a communication system 11. Examples of the communication system 11 include the internet, a local area network (LAN), a mobile phone communication network, and a satellite communication network.

[0016] <Work machinery> In this embodiment, the work machine 2 is a hydraulic excavator. The work machine 2 has a traveling body 3, a rotating body 4 supported by the traveling body 3, a work implement 5 supported by the rotating body 4, and a work implement cylinder 6 that operates the work implement 5.

[0017] The running body 3 runs on the work site 101 while supporting the rotating body 4. The running body 3 has a pair of tracks 3A. The work machine 2 runs as the tracks 3A rotate. The rotating body 4 is capable of rotating while being supported by the running body 3.

[0018] The revolving unit 4 is the body of the work machine 2. The revolving unit 4 is disposed above the running unit 3. The revolving unit 4 is supported by the running unit 3 so as to be able to revolve.

[0019] The work implement 5 is supported by the revolving unit 4. The work implement 5 includes a boom 5A, an arm 5B, and a bucket 5C. The boom 5A is rotatably connected to the front of the revolving unit 4. The arm 5B is rotatably connected to the tip of the boom 5A. The bucket 5C is rotatably connected to the tip of the arm 5B.

[0020] The work implement cylinder 6 includes a boom cylinder 6A, an arm cylinder 6B, and a bucket cylinder 6C. The boom cylinder 6A, the arm cylinder 6B, and the bucket cylinder 6C are each a hydraulic cylinder driven by hydraulic oil.

[0021] The boom cylinder 6A operates the boom 5A. The operation of the boom 5A includes a raising operation and a lowering operation. When the boom cylinder 6A extends, the boom 5A performs a raising operation. When the boom cylinder 6A retracts, the boom 5A performs a lowering operation.

[0022] The arm cylinder 6B operates the arm 5B. The operations of the arm 5B include an excavation operation and a dumping operation. When the arm cylinder 6B extends, the arm 5B performs an excavation operation. When the arm cylinder 6B retracts, the arm 5B performs a dumping operation.

[0023] The bucket cylinder 6C operates the bucket 5C. The operations of the bucket 5C include an excavation operation and a dumping operation. When the bucket cylinder 6C extends, the bucket 5C performs an excavation operation. When the bucket cylinder 6C retracts, the bucket 5C performs a dumping operation.

[0024] <Motion sensors and cameras> 2 is a side view that schematically shows a work machine 2 according to an embodiment. As shown in FIG. 2, the work machine 2 has a swing angle sensor 12, an attitude sensor 13, and a camera 14.

[0025] The rotating bed 4 and the work implement 5 are each an example of a movable part of the work machine 2. The rotating bed 4 rotates about a rotation axis RX. The work implement 5 performs a work operation. The work operation of the work implement 5 includes moving operations such as a dumping operation and an excavation operation. The boom 5A is supported by the rotating bed 4 so as to be rotatable about a rotation axis AX1. The arm 5B is supported by the boom 5A so as to be rotatable about a rotation axis AX2. The bucket 5C is supported by the arm 5B so as to be rotatable about a rotation axis AX3. The rotation axes AX1, AX2, and AX3 are parallel to each other. The rotation axes AX1, AX2, and AX3 are perpendicular to the rotation axis RX.

[0026] The swing angle sensor 12 and the attitude sensor 13 are each an example of a motion sensor that detects the motion of a movable part of the work machine 2. The swing angle sensor 12 detects the swing motion of the revolving unit 4. The swing angle sensor 12 can detect the swing speed and swing direction of the revolving unit 4. The attitude sensor 13 detects the movement motion of the work implement 5. The attitude sensor 13 detects the attitude of the work implement 5. The attitude of the work implement 5 includes the angle of the work implement 5. The attitude sensor 13 includes a boom attitude sensor 13A that detects the angle θ11 of the boom 5A about the rotation axis AX1, an arm attitude sensor 13B that detects the angle θ12 of the arm 5B about the rotation axis AX2, and a bucket attitude sensor 13C that detects the angle θ13 of the bucket 5C about the rotation axis AX3. In the embodiment, the angle θ11 of the boom 5A is the angle formed by the swing axis RX of the revolving unit 4 and the boom 5A. Angle θ12 of arm 5B is the angle formed between boom 5A and arm 5B. An angle θ13 of bucket 5C is the angle formed between arm 5B and bucket 5C.

[0027] In this embodiment, the posture sensor 13 is a stroke sensor disposed in the work implement cylinder 6. The work implement cylinder 6 has a cylinder tube, a piston that moves inside the cylinder tube, and a rod connected to the piston. The stroke sensor detects the stroke length of the work implement cylinder 6, which indicates the travel distance of the rod. The stroke length is the travel distance from the stroke end of the work implement cylinder 6. The stroke end refers to the end position of the rod's movable range. In other words, the stroke end is the position of the rod when the work implement cylinder 6 is most retracted or the position of the rod when the work implement cylinder 6 is most extended. The posture sensor 13 may include an angle sensor such as a potentiometer that can detect the angle of the work implement 5. The posture sensor 13 may also be an inertial sensor (IMU: Inertial Measurement Unit) attached to the work implement 5.

[0028] The camera 14 captures an image of the work site 101. The camera 14 captures an image of at least the work site 101 in front of the revolving unit 4. The camera 14 is fixed to the revolving unit 4.

[0029] <Remote Controller> FIG. 3 is a hardware configuration diagram showing a remote controller 9 according to an embodiment. The remote controller 9 includes a computer system. The remote controller 9 has a processor 91 such as a CPU (Central Processing Unit), a main memory 92 including a non-volatile memory such as a ROM (Read Only Memory) and a volatile memory such as a RAM (Random Access Memory), a storage 93, and an interface 94 including an input / output circuit. The functions of the remote controller 9 are stored in the storage 93 as a computer program. The processor 91 reads the computer program from the storage 93, loads it into the main memory 92, and executes processing in accordance with the computer program. The computer program may be distributed to the remote controller 9 via a network.

[0030] FIG. 4 is a functional block diagram showing a work machine 2 and a remote controller 9 according to an embodiment. As shown in FIG. 4, the work machine 2 has an on-vehicle controller 15. The on-vehicle controller 15 includes a computer system. Like the remote controller 9, the on-vehicle controller 15 has a processor, main memory, storage, and an interface. The detection signal of the turning angle sensor 12 and the detection signal of the attitude sensor 13 are input to the on-vehicle controller 15. Image data 31 captured by the camera 14 is input to the on-vehicle controller 15.

[0031] An operation device 7 and a display device 8 are connected to a remote controller 9. The remote controller 9 communicates with an in-vehicle controller 15 via a communication system 11. The remote controller 9 has an operation signal transmission unit 21, an image data acquisition unit 22, a display data generation unit 23, a movement signal reception unit 24, and a display control unit 25.

[0032] The operation signal transmitter 21 transmits an operation signal generated by operating the operation device 7 to the work machine 2. The operation signal is transmitted to the on-board controller 15 via the communication system 11. The on-board controller 15 outputs control commands for performing at least some of the traveling operation of the traveling body 3, the rotating operation of the rotating body 4, and the movement operation of the work machine 5, based on the operation signal.

[0033] The image data acquisition unit 22 receives image data 31 captured by the camera 14 mounted on the work machine 2. The on-board controller 15 transmits the image data 31 captured by the camera 14 to the remote controller 9. The image data 31 captured by the camera 14 is transmitted to the remote controller 9 via the communication system 11. The image data acquisition unit 22 receives the image data 31 transmitted from the on-board controller 15.

[0034] As described above, camera 14 is fixed to the revolving unit 4. When the revolving unit 4 revolves, the orientation of camera 14 changes along with the revolving unit 4, and the imaging area of ​​camera 14 moves. When the imaging area of ​​camera 14 moves, the content of image data 31 captured by camera 14 changes. When the revolving unit 4 revolves left, the orientation of camera 14 changes to the left, and the imaging area of ​​camera 14 moves to the left. When the imaging area of ​​camera 14 moves to the left, the content of image data 31 changes so that image data 31 moves to the left. When the revolving unit 4 revolves right, the orientation of camera 14 changes to the right, and the imaging area of ​​camera 14 moves to the right. When the imaging area of ​​camera 14 moves to the right, the content of image data 31 changes so that image data 31 moves to the right.

[0035] In the following description, a change in the content of image data 31 due to movement of the imaging area of ​​camera 14 will be expressed as image data 31 moving, as appropriate. When the imaging area of ​​camera 14 moves leftward, image data 31 moves leftward. When the imaging area of ​​camera 14 moves rightward, image data 31 moves rightward. When the imaging area of ​​camera 14 moves upward, image data 31 moves upward. When the imaging area of ​​camera 14 moves downward, image data 31 moves downward. The direction of movement of the imaging area of ​​camera 14 and the direction of movement of image data 31 are the same.

[0036] The display data generating unit 23 extracts display data 32 of an extraction range smaller than the outer shape of the image data 31 from the image data 31 acquired by the image data acquiring unit 22 .

[0037] The movement signal receiving unit 24 receives a movement signal for moving within an extraction range that defines the display data 32. The movement signal includes detection signals from operation sensors that detect the operation of movable parts of the work machine 2. In this embodiment, the operation signal includes detection signals from the swing angle sensor 12 and the attitude sensor 13. The on-vehicle controller 15 transmits the detection signals from the swing angle sensor 12 and the attitude sensor 13 to the remote controller 9. The detection signals from the swing angle sensor 12 and the attitude sensor 13 are transmitted to the remote controller 9 via the communication system 11. The movement signal receiving unit 24 receives the detection signals from the swing angle sensor 12 and the attitude sensor 13 transmitted from the on-vehicle controller 15.

[0038] The display control unit 25 moves the extraction range based on the movement signal received by the movement signal receiving unit 24, and displays display data 32 of the extraction range on the display device 8. The display control unit 25 moves the extraction range in the movement direction of the movable part of the work machine 2.

[0039] <Remote control method> 5 is a flowchart showing a method for remotely controlling a work machine 2 according to an embodiment. Image data 31 of a work site 101 is captured by the camera 14. The image data 31 of the work site 101 captured by the camera 14 is transmitted from the on-vehicle controller 15 to the remote controller 9 via the communication system 11. The image data acquisition unit 22 receives the image data 31 transmitted from the on-vehicle controller 15 (step S1).

[0040] The display data generating unit 23 extracts display data 32 of an extraction range smaller than the outer shape of the image data 31 from the image data 31 acquired by the image data acquiring unit 22 (step S2).

[0041] FIG. 6 is a diagram showing the relationship between image data 31 and display data 32 according to the embodiment. The outer shape of the image data 31 is rectangular. The outer shape of the display data 32 is rectangular and smaller than the outer shape of the image data 31. The size of the outer shape of the image data 31 is uniquely determined based on, for example, the imaging area of ​​the camera 14 (the field of view area of ​​the optical system of the camera 14). The display data 32 is defined by an extraction range that is smaller than the outer shape of the image data 31. The display data 32 is a part of the image data 31. The display data 32 is image data 31 inside the extraction range. The display data 32 is defined inside the outer shape of the image data 31.

[0042] 6, when the moving parts of the work machine 2 are stationary, the extraction range is positioned at the center of the image data 31, and the display data 32 is defined at the center of the image data 31. In this embodiment, when the moving parts of the work machine 2 are stationary, the center of the image data 31 and the center of the display data 32 coincide. Note that when the moving parts of the work machine 2 are stationary, the center of the image data 31 and the center of the display data 32 do not have to coincide.

[0043] The movement signal receiving unit 24 determines whether or not a movement signal has been received (step S3). In the embodiment, the movement signal receiving unit 24 determines whether or not at least one of the detection signal of the turning angle sensor 12 and the detection signal of the attitude sensor 13 has been received.

[0044] If it is determined in step S3 that a movement signal has been received (step S3: Yes), the display control unit 25 displays the display data 32 as a preceding movement (step S4). If it is determined in step S3 that a movement signal has not been received (step S3: No), the display control unit 25 displays the display data 32 normally (step S6). After the processing of step S4 or step S6, the remote controller 9 determines whether or not to end the remote operation (step S5). If it is determined in step S5 that the remote operation will not be ended (step S5: No), the remote operation of the work machine 2 continues. If it is determined in step S5 that the remote operation will be ended (step S5: Yes), the remote operation of the work machine 2 ends.

[0045] FIG. 7 is a diagram illustrating normal display and preceding movement display according to the embodiment. The upper left diagram of FIG. 7 is a diagram illustrating the relationship between the display data 32 that is normally displayed and the image data 31. The lower left diagram of FIG. 7 is a diagram illustrating a schematic diagram of the state of the work machine 2 when the display data 32 is normally displayed. The upper right diagram of FIG. 7 is a diagram illustrating the relationship between the display data 32 that is displayed in a preceding movement mode and the image data 31. The lower right diagram of FIG. 7 is a diagram illustrating a schematic diagram of the state of the work machine 2 when the display data 32 is displayed in a preceding movement mode. Note that the angles shown in FIG. 7 are just an example.

[0046] Normal display refers to a display mode in which display data 32 whose relative position with respect to image data 31 is constant is displayed on the display device 8. Leading moving display refers to displaying display data 32 that moves ahead of image data 31 on the display device 8.

[0047] The display control unit 25 normally displays the display data 32 when the movable part of the work machine 2 is stationary. As shown in the left diagram of FIG. 7 , when the revolving unit 4 is stationary, the extraction range is positioned at the center of the image data 31, and the display data 32 is defined at the center of the image data 31. In the embodiment, when the revolving unit 4 is stationary, the center of the image data 31 and the center of the display data 32 coincide. Note that when the revolving unit 4 is stationary, the center of the image data 31 and the center of the display data 32 do not have to coincide. The display device 8 displays the display data 32. The image data 31 around the display data 32 is not displayed on the display device 8.

[0048] As shown in the right diagram of FIG. 7 , when the revolving unit 4 revolves based on an operation signal from the operation device 7, the display control unit 25 moves the extraction range in the image data 31 in the revolving direction of the revolving unit 4. The display control unit 25 starts moving the extraction range when the revolving operation starts. The display control unit 25 can start moving the extraction range when the revolving operation starts based on the detection signal from the revolving angle sensor 12. In the example shown in FIG. 7 , the revolving unit 4 revolves left, so the display control unit 25 moves the extraction range to the left in the image data 31. The camera 14 is fixed to the revolving unit 4. As the revolving unit 4 revolves, the imaging area of ​​the camera 14 moves to the left. As the imaging area of ​​the camera 14 moves to the left, the image data 31 moves to the left. The image data 31 moves at a revolving speed V1 of the revolving unit 4. The display control unit 25 moves the extraction range at a moving speed V2 that is higher than the revolving speed V1 of the revolving unit 4. The rotation speed V1 is detected by the rotation angle sensor 12. The display control unit 25 moves the extraction range at a movement speed V2 that is higher than the rotation speed V1 detected by the rotation angle sensor 12. In the advanced movement display, the display data 32 moves relative to the image data 31. The center of the display data 32 shifts to the left, which is the rotation direction, from the center of the image data 31. The display device 8 displays the display data 32 that shows a portion of the image data 31 of the work site 101 that is located in the rotation direction of the rotating unit 4. The display device 8 displays the display data 32 that is ahead of the center of the image data 31 in the rotation direction. Note that the rotation speed V1 of the rotating unit 4 and the movement speed V2 of the extraction range may be the same.

[0049] The display control unit 25 ends the movement of the extraction range when the rotation operation of the rotating unit 4 ends. When the rotation operation of the rotating unit 4 ends, the display control unit 25 ends the movement of the extraction range with the center of the image data 31 and the center of the display data 32 aligned. Note that when ending the movement of the extraction range, the display control unit 25 does not have to ensure that the center of the image data 31 and the center of the display data 32 are aligned.

[0050] <Effects> As described above, the remote controller 9 of the remote operation system 1 according to the embodiment includes an image data acquisition unit 22 that receives image data 31 captured by the camera 14 mounted on the work machine 2, a display data generation unit 23 that extracts display data 32 of an extraction range that is smaller than the outer shape of the image data 31 from the image data 31, a movement signal receiving unit 24 that receives a movement signal for moving the extraction range, and a display control unit 25 that moves the extraction range based on the movement signal and displays the display data 32 of the extraction range on the display device 8.

[0051] According to the embodiment, display data 32, which is an extracted portion of image data 31, is enlarged and displayed on display device 8. When image data 31 is displayed on display device 8, objects present at work site 101 appear small on display device 8, which may make it difficult for the operator to recognize the situation at work site 101. According to the embodiment, display data 32, which is an extracted portion of image data 31, is enlarged and displayed on display device 8, which makes it easier for the operator to recognize the situation at work site 101.

[0052] When the display data 32 is enlarged and displayed on the display device 8, the range of the work site 101 that the operator can recognize becomes narrower. In this embodiment, when the revolving unit 4 rotates, the center of the display data 32 shifts in the rotation direction relative to the center of the image data 31. The display device 8 displays the display data 32 that is ahead of the center of the image data 31 in the rotation direction. This allows the operator to quickly recognize the status of the work site 101 that is located in the rotation direction of the revolving unit 4. The operator can properly recognize the status of the work site 101. Because the operator can properly recognize the status of the work site 101, the operator can smoothly perform remote operation of the work machine 2. This prevents a decrease in the work efficiency of the remotely operated work machine 2.

[0053] [Second embodiment] A second embodiment will be described below. In the following description, the same or equivalent components as those in the above-described embodiment will be denoted by the same reference numerals, and the description of those components will be simplified or omitted.

[0054] FIG. 8 is a diagram showing the relationship between image data 31 and display data 32 according to the embodiment. In the first embodiment described above, an example was described in which the extraction range defining the display data 32 moves in the rotation direction of the revolving unit 4. As shown in FIG. 8, the display control unit 25 may move the extraction range in the movement direction of the work implement 5. The movement signal receiving unit 24 can calculate the position of the bucket 5C based on the detection signal of the attitude sensor 13 and dimensional data of the work implement 5. The dimensional data of the work implement 5 includes the lengths of the boom 5A, the arm 5B, and the bucket 5C. The dimensional data of the work implement 5 is known data that can be determined from the specification data of the work implement 5. As described with reference to FIG. 2, the attitude sensor 13 can detect angles θ11, θ12, and θ13. The movement signal receiving unit 24 can calculate the position of the bucket 5C relative to the reference position of the revolving unit 4, for example, the position of the tip of the bucket 5C, based on the angles θ11, θ12, and θ13, and the lengths of the boom 5A, the arm 5B, and the bucket 5C. The display control unit 25 moves the extraction range based on the position of the tip of the bucket 5C.

[0055] In addition, if a position sensor that detects the position of the bucket 5C, for example the position of the tip of the bucket 5C, is attached to at least a part of the bucket 5C, the movement signal receiving unit 24 may calculate the position of the tip of the bucket 5C based on the detection signal of the position sensor.

[0056] The movement signal receiving unit 24 may calculate the position of the bucket 5C based on image data 31 of the work machine 50 captured by the camera 14. The movement signal receiving unit 24 may calculate the position of the bucket 5C by performing image processing on the image data 31 of the work machine 50.

[0057] As shown in FIG. 8, when the bucket 5C moves based on an operation signal from the operation device 7, the display control unit 25 moves the extraction range in the movement direction of the bucket 5C. In the example shown in FIG. 8, the bucket 5C rises, so the display control unit 25 moves the extraction range upward. The display control unit 25 moves the extraction range at a movement speed V3 that is higher than the movement speed of the bucket 5C. The display control unit 25 can calculate the movement speed of the bucket 5C based on the detection signal from the attitude sensor 13. The center of the display data 32 is shifted upward, which is the movement direction of the bucket 5C, relative to the center of the image data 31. The display device 8 displays display data 32 that shows a portion of the image data 31 of the work site 101 that exists in the movement direction of the bucket 5C. The display device 8 displays display data 32 that is ahead of the center of the image data 31 in the movement direction of the bucket 5C. Note that the movement speed of the bucket 5C and the movement speed V3 of the extraction range may be the same.

[0058] For example, if a speed sensor that detects the movement speed of the bucket 5C is attached to at least a part of the bucket 5C, the movement signal receiving unit 24 may calculate the movement speed of the bucket 5C based on the detection signal of the speed sensor.

[0059] The movement signal receiving unit 24 may calculate the movement speed of the bucket 5C based on image data 31 of the work machine 50 captured by the camera 14. The movement signal receiving unit 24 may calculate the movement speed of the bucket 5C by performing image processing on the image data 31 of the work machine 50.

[0060] [Third embodiment] A third embodiment will be described below. In the following description, the same or equivalent components as those in the above-described embodiment are denoted by the same reference numerals, and the description of these components will be simplified or omitted.

[0061] FIG. 9 is a diagram schematically illustrating a remote control room 102 according to an embodiment. In the embodiment, the remote control system 1 includes a gaze sensor 16 that detects the gaze of an operator operating the operation device 7. The movement signal receiving unit 24 receives a detection signal from the gaze sensor 16 as a movement signal. The display control unit 25 moves the extraction range in the direction of the gaze. The display control unit 25 displays the display data 32 by moving it ahead of the gaze based on the direction of the gaze. For example, if the operator's gaze moves leftward, the display control unit 25 moves the extraction range leftward at a speed faster than the gaze movement speed. The center of the display data 32 shifts leftward relative to the center of the image data 31. The display device 8 displays the display data 32 that is ahead of the center of the image data 31 in the direction of the gaze movement. Note that the gaze movement speed and the extraction range movement speed may be the same.

[0062] [Fourth embodiment] A fourth embodiment will be described below. In the following description, the same or equivalent components as those in the above-described embodiments are denoted by the same reference numerals, and the description of these components will be simplified or omitted.

[0063] 10 is a diagram illustrating a normal display and a following movement display according to the embodiment. In the embodiment, the display control unit 25 moves the extraction range so as to follow the bucket 5C at the tip of the work machine 5. The following movement display means that display data 32 that follows the moving parts of the work machine 2 is displayed on the display device 8.

[0064] In this embodiment, the work machine 2 is equipped with a work implement 50 having a swing boom such as that disclosed in Japanese Patent Application Laid-Open No. 2002-348899. The base end of the swing boom of the work implement 50 is connected to a revolving unit 4 so as to be rotatable about a rotation axis CX. The rotation axis CX and the rotation axis RX are parallel to each other. In the example shown in FIG. 10, the revolving unit 4 does not rotate. Note that the angles shown in FIG. 10 are merely examples.

[0065] The left diagram of Fig. 10 is a diagram that schematically shows the state of the work machine 2 when the display data 32 is normally displayed. The right diagram of Fig. 10 is a diagram that schematically shows the state of the work machine 2 when the display data 32 is displayed in a following movement manner.

[0066] When the work machine 50 is stationary, the display control unit 25 normally displays the display data 32. When the work machine 50 is stationary, the center of the image data 31 and the center of the display data 32 coincide. The display device 8 displays the display data 32. Note that when the work machine 50 is stationary, the center of the image data 31 and the center of the display data 32 do not have to coincide.

[0067] As shown in the right diagram of FIG. 10 , when the work implement 50 rotates around the rotation axis CX based on the operation signal of the operating device 7, the display control unit 25 moves the extraction range in the rotation direction of the work implement 50. A rotation sensor that detects the rotation angle of the work implement 50 around the rotation axis CX is provided in the work implement 50. The display control unit 25 can start moving the extraction range based on the detection signal of the rotation sensor. In the example shown in FIG. 10 , the work implement 50 rotates to the left, so the display control unit 25 moves the extraction range to the left. In the example shown in FIG. 10 , the revolving unit 4 does not rotate, so the image data 31 does not move. The display control unit 25 moves the extraction range to follow the work implement 50. The display control unit 25 may move the extraction range at the same movement speed as the rotation speed of the work implement 50. The rotation speed of the work implement 50 is detected by the rotation sensor.

[0068] When the work implement 50 rotates around the rotation axis CX, the display control unit 25 may display the display data 32 in a manner that moves ahead. The display control unit 25 may move the extraction range at a speed that is faster than the rotation speed of the work implement 50. When the display data 32 is displayed in a manner that moves ahead, the center of the display data 32 shifts in the rotation direction of the work implement 50 relative to the center of the image data 31. The display device 8 displays the display data 32 that moves ahead of the center of the image data 31 in the rotation direction of the work implement 50.

[0069] FIG. 11 is a diagram illustrating the relationship between the respective operations of the revolving unit 4 and the work implement 50 according to the embodiment and the display data 32. In the example shown in FIG. 11, the revolving unit 4 revolves to the right, and the work implement 50 rotates to the left about the rotation axis CX. As shown in FIG. 11, the display control unit 25 may move the extraction range so as to follow the bucket 5C at the tip of the work implement 5. The movement signal receiving unit 24 can calculate the position of the bucket 5C based on the detection signal of the rotation sensor that detects the rotation angle of the work implement 50, the detection signal of the attitude sensor 13, and dimensional data of the work implement 50.

[0070] For example, if a position sensor that detects the position of the bucket 5C is attached to at least a part of the bucket 5C, the movement signal receiving unit 24 may calculate the position of the bucket 5C based on the detection signal of the position sensor.

[0071] The movement signal receiving unit 24 may calculate the position of the bucket 5C based on image data 31 of the work machine 50 captured by the camera 14. The movement signal receiving unit 24 may calculate the position of the bucket 5C by performing image processing on the image data 31 of the work machine 50.

[0072] [Fifth embodiment] A fifth embodiment will be described below. In the following description, the same or equivalent components as those in the above-described embodiments are denoted by the same reference numerals, and the description of these components will be simplified or omitted.

[0073] Figure 12 is a diagram illustrating a normal display and a preceding movement display according to an embodiment. In the example shown in Figure 12, the work machine 200 is a wheel loader. The work machine 200 has a front frame 201, a rear frame 202 positioned rearward of the front frame 201, an articulating mechanism 203 connecting the front frame 201 and the rear frame 202, a work implement 204 connected to the front frame 201, and a camera mounting compartment 205 provided in the rear frame 202. The camera 140 is fixed to the camera mounting compartment 205 of the rear frame 202.

[0074] The left diagram in Fig. 12 is a diagram that schematically shows the state of the work machine 200 when the display data 32 is displayed normally. The right diagram in Fig. 12 is a diagram that schematically shows the state of the work machine 200 when the display data 32 is displayed in a forward movement mode.

[0075] When the work machine 200 is moving straight, the display control unit 25 displays the display data 32 normally. When the work machine 200 is moving straight, the center of the image data 31 and the center of the display data 32 coincide. The display device 8 displays the display data 32. Note that when the work machine 200 is moving straight, the center of the image data 31 and the center of the display data 32 do not have to coincide.

[0076] As shown in the right diagram of FIG. 12 , when the front frame 201 bends relative to the rear frame 202 so that the work machine 200 turns right, based on an operation signal from the operation device 7, the display control unit 25 moves the extraction range in the bending direction of the front frame 201. In the example shown in FIG. 12 , the bending direction of the front frame 201 is right, so the display control unit 25 moves the extraction range to the right. In the example shown in FIG. 12 , the orientation of the rear frame 202 does not change, so the image data 31 does not move. The display control unit 25 moves the extraction range at a moving speed higher than the bending speed of the front frame 201. The center of the display data 32 shifts in the bending direction of the front frame 201 relative to the center of the image data 31. The display device 8 displays display data 32 that is ahead of the center of the image data 31 in the bending direction of the front frame 201. Note that the bending speed of the front frame 201 and the moving speed of the extraction range may be the same. [Explanation of symbols]

[0077] 1... remote operation system, 2... work machine, 3... running body, 3A... track, 4... rotating body, 5... work implement, 5A... boom, 5B... arm, 5C... bucket, 6... work implement cylinder, 6A... boom cylinder, 6B... arm cylinder, 6C... bucket cylinder, 7... operation device, 8... display device, 9... remote controller, 10... operation seat, 11... communication system, 12... swing angle sensor, 13... attitude sensor, 13A... boom attitude sensor, 13B... arm attitude sensor, 13C... bucket attitude sensor, 14... camera, 15... on-board controller, 16... line-of-sight sensor, 21... operation signal transmission unit, 22... image data acquisition unit, 23... display device data generation unit, 24...movement signal receiving unit, 25...display control unit, 31...image data, 32...display data, 50...work machine, 91...processor, 92...main memory, 93...storage, 94...interface, 101...work site, 102...remote control room, 140...camera, 200...work machine, 201...front frame, 202...rear frame, 203...articulate mechanism, 204...work machine, 205...camera mounting room, AX1...rotation axis, AX2...rotation axis, AX3...rotation axis, CX...rotation axis, RX...swivel axis, V1...swivel speed, V2...movement speed, V3...movement speed, θ11...angle, θ12...angle, θ13...angle.

Claims

1. A display device; a remote controller; The remote controller an image data acquisition unit that receives image data captured by a camera mounted on the work machine; a display data generation unit that extracts display data from the image data within an extraction range that is smaller than the outer shape of the image data; a movement signal receiving unit that receives a movement signal for moving the extraction range; a display control unit that moves the extraction range based on the movement signal and causes the display device to display display data of the extraction range. Remote control system for work machines.

2. the work machine has a movable part and a motion sensor that detects motion of the movable part, the movement signal includes a detection signal of the motion sensor; the display control unit moves the extraction range in a moving direction of the movable unit. The remote control system for a work machine according to claim 1.

3. the work machine has a rotating body, the camera is fixed to the rotating body, the motion sensor detects a rotation motion of the rotating body; the display control unit moves the extraction range in a rotation direction of the rotating body. The remote control system for a work machine according to claim 2.

4. the display control unit moves within the extraction range at a moving speed higher than a rotation speed of the rotating body. The remote control system for a work machine according to claim 3.

5. the display control unit starts moving the extraction range when the turning operation is started. The remote control system for a work machine according to claim 4.

6. the display control unit ends the movement of the extraction range when the turning operation ends. The remote control system for a work machine according to claim 5.

7. The work machine has a work implement, The motion sensor detects a movement motion of the work machine, The display control unit moves the extraction range in a moving direction of the work machine. The remote control system for a work machine according to claim 2.

8. the display control unit moves within the extraction range at a moving speed higher than a moving speed of the work machine. The remote control system for a work machine according to claim 7.

9. the display control unit moves the extraction range so as to follow the work machine. The remote control system for a work machine according to claim 7.

10. An operating device; a line-of-sight sensor for detecting the line of sight of an operator operating the operation device, the movement signal includes a detection signal from the line of sight sensor, the display control unit moves the extraction range in the direction of movement of the line of sight. The remote control system for a work machine according to claim 1.

11. The display device is disposed outside the work machine. The remote control system for a work machine according to claim 1.

12. the remote controller is located external to the work machine; The remote control system for a work machine according to claim 1.

13. a display data generation unit that extracts display data from image data captured by a camera mounted on the work machine, the display data having an extraction range that is smaller than the outer shape of the image data; a movement signal receiving unit that receives a movement signal for moving the extraction range; a display control unit that moves the extraction range based on the movement signal and displays display data of the extraction range on a display device that is arranged outside the work machine, Remote controller for work machine.

14. extracting display data from image data captured by a camera mounted on a work machine, the display data having an extraction range smaller than the outer shape of the image data; receiving a movement signal for moving the extraction range; moving the extraction range based on the movement signal and displaying display data of the extraction range on a display device. A method for remotely controlling a work machine.

Images