Telemanipulator

Abstract

The present application provides a remote control device capable of efficiently and safely remotely operating a working machine operated from a remote site even in the case of having a communication delay. It includes a communication control unit (222) that receives a camera image of a work site taken by a vehicle-mounted camera (91) and vehicle body information of a working machine (hydraulic excavator 1), a predicted trajectory calculation unit (220) that calculates a predicted trajectory of the working machine (hydraulic excavator 1) based on the vehicle body information and outputs predicted trajectory data for display in the form of an image on a display device (202), and a display control unit (221) that causes the display device (202) to display the camera image and the image of the predicted trajectory on the same screen (at the same time).

Description

Technical Field

[0001] The present invention relates to a remote control device. Background Art

[0002] There is a technology for operating a work machine from a remote location (remote control). To provide visual information to the remote operator during remote control of the work machine, the work machine is equipped with an image acquisition device (camera). The image of the work site captured by the image acquisition device (camera) is displayed on a display device on the remote operator's remote control device. The operator controls the work machine through the remote control device while checking the image displayed on the display device.

[0003] For example, Patent Document 1 discloses the following technology: a work site is photographed by an image acquisition device of a work machine, a virtual viewpoint image of the work machine observed from a virtual viewpoint is generated, and the actual image and the virtual viewpoint image captured by the image acquisition device of the work machine are respectively displayed at different positions on the same screen of a display device.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2019-54464 Summary of the Invention

[0007] Problems to be solved by the invention

[0008] Patent Document 1 provides the operator with a work site image (actual image) and an image from a virtual viewpoint (virtual viewpoint image) from an image acquisition device, thereby expecting to suppress a decrease in work efficiency in remote operation.

[0009] Here, the following issues arise.

[0010] Image transmission involves large amounts of data, and therefore, can be subject to delays depending on the conditions of the communication lines. If this delay occurs, the image captured by the image acquisition device and the image generated from the virtual viewpoint, even though displayed on the same screen, will exhibit different behaviors. If these images displayed on the same screen do not match, it is difficult to prevent a decrease in operator efficiency and the risk of accidents.

[0011] The present invention has been made to solve the above-mentioned problems, and an object of the present invention is to provide a remote control device that can remotely control a working machine operated from a remote location efficiently and safely even when there is a communication delay.

[0012] Means for solving problems

[0013] In order to solve the above-mentioned problems, the remote control device of the present invention is provided with a control device for sending operation signals to a working machine and controlling the working machine from a remote location, wherein the control device includes: a communication control unit, which receives a camera image of the working site captured by a camera and body information of the working machine; a predicted trajectory calculation unit, which calculates the predicted trajectory of the working machine based on the body information and outputs predicted trajectory data for display on a display device in the form of an image; and a display control unit, which causes the display device to display the camera image and the image of the predicted trajectory on the same screen.

[0014] Effects of the Invention

[0015] According to the present invention, even when image transmission is delayed due to communication line conditions, the operator can still identify the current status of the vehicle body by displaying the image of the predicted trajectory. Furthermore, by offsetting the simulated camera image (a (simulated) image of the working machine as viewed from a camera, generated based on the predicted trajectory), the difference from the actual camera image is displayed when communication line delays occur, visually notifying the operator of image delays. Consequently, remote operation of a working machine from a remote location is possible, even in the presence of communication delays.

[0016] Other aspects and effects than those described above can be understood from the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a diagram showing the configuration of a remote control system for a hydraulic excavator (working machine).

[0018] Figure 2 This diagram shows a controller and a hydraulic drive device of a hydraulic excavator (working machine).

[0019] Figure 3 This is a detailed diagram of the solenoid valve unit.

[0020] Figure 4 yes Figure 1 Hardware structure diagram of the remote control device of the remote control device.

[0021] Figure 5 yes Figure 1 A functional block diagram of a remote manipulation control device of a remote manipulation device according to a first embodiment of the present invention.

[0022] Figure 6 This is a flowchart of the calculation process of the predicted trajectory calculation unit in the first embodiment.

[0023] Figure 7 This is a control process flowchart of the display control unit in the first embodiment.

[0024] Figure 8 This is a control process flowchart of the communication control unit in the first embodiment.

[0025] Figure 9 This is a functional block diagram of a remote control device of a remote control device according to a second embodiment.

[0026] Figure 10 This is a flowchart of the calculation process of the communication status determination unit in the second embodiment.

[0027] Figure 11 This is a flowchart of the control process of the communication status transmitting unit in the second embodiment.

[0028] Figure 12 This is a control process flowchart of the communication control unit in the second embodiment.

[0029] Figure 13 It is a diagram showing a display example (overlapping image) of a display screen of a display device.

[0030] Figure 14 It is a diagram showing a display example of a display screen of a display device (images displayed at different positions without overlapping). DETAILED DESCRIPTION

[0031] Hereinafter, the embodiments of the present invention will be described using the accompanying drawings. In each drawing, parts having the same function are marked with the same reference numerals and repeated descriptions are omitted. In the following description of this specification, the working machine as the remote control object, such as Figure 1 As shown, a hydraulic excavator is shown as an example, in which a bucket 8 is provided as a work tool at the front end of the work device. It should be noted that if the structure of the work device is a multi-jointed type formed by connecting multiple link members (attachments, bucket arms, booms, etc.), it can also be applied to work machines other than hydraulic excavators.

[0032] In the following description of this specification, when there are multiple identical components, a letter may be added to the end of the reference numeral (number). However, there are also cases where the letter is omitted and the multiple components are collectively described. For example, when there are two travel hydraulic motors 4a and 4b on the left and right, they may be collectively referred to as the travel hydraulic motor 4.

[0033] [First embodiment]

[0034] Basic Structure

[0035] Figure 11 is a diagram showing an example of a remote control system 100 for a working machine 1 according to a first embodiment of the present invention. The remote control system 100 remotely controls the working machine 1. In this embodiment, the working machine 1 is a hydraulic excavator. Figure 2 This figure shows a controller and a hydraulic drive device of a hydraulic excavator (working machine) 1 according to a first embodiment of the present invention. Figure 3 yes Figure 2 Detailed view of the solenoid valve unit 160 in FIG.

[0036] exist Figure 1 In the figure, the hydraulic excavator 1 consists of a traveling body 3 and a working front section 2, which is a multi-articulated working device. The traveling body 3 is composed of a lower traveling section 3a and an upper slewing section 3b. The lower traveling section 3a is driven by left and right traveling hydraulic motors 4a and 4b (right traveling hydraulic motor 4a and left traveling hydraulic motor 4b). The upper slewing section 3b is mounted on the lower traveling section 3a and slewed by a slewing hydraulic motor 5. The working front section 2 is composed of a plurality of driven components (a boom 6, an arm 7, and a bucket 8) that are connected and rotate in the vertical direction.

[0037] The base end of the boom 6 is rotatably supported at the front of the upper rotating part 3b by means of a boom pin. The boom 7 is rotatably connected to the front end of the boom 6 by means of an boom pin, and the bucket 8 is rotatably connected to the front end of the boom 7 by means of a bucket pin. The boom 6 is driven by a boom cylinder 61, the boom 7 is driven by an arm cylinder 71, and the bucket 8 is driven by a bucket cylinder 81. In order to be able to measure the rotation of the boom 6, the boom 7, and the bucket 8, a boom angle sensor 62 is mounted on the boom pin, an arm angle sensor 72 is mounted on the arm pin, a bucket angle sensor 82 is mounted on the bucket link 9, and a vehicle body inclination angle sensor 31 is mounted on the upper rotating part 3b for detecting the inclination angle of the upper rotating part 3b relative to a reference plane (e.g., a horizontal plane). It should be noted that the angle sensors 62, 72, and 82 can be replaced by angle sensors relative to a reference plane (e.g., a horizontal plane). A rotation angle sensor 32 is mounted on the rotation center axis to measure the relative angle between the upper rotation unit 3b and the lower traveling unit 3a. An operating device 10 for operating the hydraulic excavator 1 is provided in a cab 12 provided on the upper rotation unit 3b.

[0038] The operating device 10 is composed of a right operating lever 10a for operating the boom cylinder 61 (boom 6) and the bucket cylinder 81 (bucket 8); a left operating lever 10b for operating the arm cylinder 71 (arm 7) and the swing hydraulic motor 5 (upper swing unit 3b); a right traveling lever 10c for operating the right traveling hydraulic motor 4a (lower traveling unit 3a); and a left traveling lever 10d for operating the left traveling hydraulic motor 4b (lower traveling unit 3a). Hereinafter, the right operating lever 10a, the left operating lever 10b, the right traveling lever 10c, and the left traveling lever 10d may be collectively referred to as the operating device 10.

[0039] like Figure 2 As shown, the engine 11 as the prime mover mounted on the upper rotating part 3b drives the hydraulic pumps 20a, 20b and the pilot pump 30. The hydraulic pumps 20a, 20b are variable capacity pumps that can be controlled by regulators 20aa, 20ba, and the pilot pump 30 is a fixed capacity pump. The hydraulic pump 20 and the pilot pump 30 draw working oil from the container 170. In this embodiment, the control signal output from the controller 40 as a control device is input to the regulators 20aa, 20ba. The detailed structure of the regulators 20aa, 20ba is omitted, and the discharge flow rate of the hydraulic pumps 20a, 20b is controlled corresponding to the control signal. The pump pipe 143a as the discharge piping of the pilot pump 30 is connected to each electromagnetic proportional valve in the electromagnetic valve unit 160 after passing through the locking valve 39. In this example, the locking valve 39 is an electromagnetic switching valve, and its electromagnetic drive part is connected to the electromagnetic proportional valve 160 disposed in the cab 12 ( Figure 1 ) is electrically connected to the position detector of the door locking rod in the door locking rod. The position of the door locking rod is detected by the position detector, and a signal corresponding to the position of the door locking rod is input from the position detector to the locking valve 39. If the position of the door locking rod is in the locking position, the locking valve 39 is closed and the pump pipe 143a is cut off. If it is in the unlocking position, the locking valve 39 is opened and the pump pipe 143a is opened. In other words, when the pump pipe 143a is cut off, the operation of the operating device 10 is invalidated, and actions such as walking, turning, and digging are prohibited. The operating device 10 (10a, 10b, 10c, 10d) is an electrical control lever type, which generates an electrical signal corresponding to the operator's operation amount and operation direction. The electrical signal generated in this way is input to the controller 40 in order to drive the electromagnetic proportional valves 54 to 59 (see Figure 3 , 57 to 59 are not shown), the controller 40 outputs an electrical signal to the solenoid valve unit 160. The electrical signal is input to the hydraulic drive unit 150a to 155b via the pilot pipes 144a to 149b. The hydraulic oil discharged from the hydraulic pump 20 passes through the flow control valves 15a, 15b, 15c, 15d, 15e, and 15f (see FIG. 1 ) which are actuated by the electrical signal input to the hydraulic drive unit 150a to 155b. Figure 2 or Figure 3,exist Figure 3 15d, 15e, and 15f (not shown) are supplied to the boom cylinder 61, arm cylinder 71, bucket cylinder 81, rotary hydraulic motor 5, right travel hydraulic motor 4a, and left travel hydraulic motor 4b, which serve as actuators. The boom cylinder 61, arm cylinder 71, and bucket cylinder 81 extend and retract under the action of the supplied hydraulic oil, thereby rotating the boom 6, arm 7, and bucket 8, respectively, and changing the position and posture of the bucket 8. In addition, the rotary hydraulic motor 5 rotates under the action of the supplied hydraulic oil, thereby rotating the upper rotary section 3b relative to the lower travel section 3a. Furthermore, the right travel hydraulic motor 4a and left travel hydraulic motor 4b rotate under the action of the supplied hydraulic oil, thereby moving the lower travel section 3a.

[0040] The boom cylinder 61, arm cylinder 71, and bucket cylinder 81 are equipped with load detection devices 16a to 16f to detect their cylinder pressures. In this embodiment, the load detection device 16 is a pressure sensor that detects the pressure on the bottom side and the rod side of each of the boom cylinder 61, arm cylinder 71, and bucket cylinder 81 and outputs it to the controller 40 in the form of an electrical signal.

[0041] like Figure 1 As shown, the upper swing section 3b is equipped with an onboard camera 91 serving as an image acquisition device. When the hydraulic excavator 1 is operated from a remote location (teleoperation), the onboard camera 91 acquires a camera image (worksite image) of the work site in order to provide visual information to the remote operator (as described later). It should be noted that in this example, the camera serving as the image acquisition device for capturing the work site is mounted on (the upper swing section 3b of) the hydraulic excavator 1. Alternatively, for example, a camera installed at the work site may be used to capture a camera image (worksite image) of the work site.

[0042] Furthermore, a communication device 90 is mounted on the upper swing section 3b. The communication device 90 connects the hydraulic excavator 1 to a remote control device 200 via a network so that communication is possible. The remote control device 200 is installed at a remote location away from the work site to remotely control (remotely operate) the hydraulic excavator 1. The communication device 90 transmits camera images from an onboard camera 91 and vehicle body information from various sensors (angle sensors 62, 72, 82, 32, and load detection devices 16a to 16f) to the remote control device 200. It also receives control signals (including operation signals from the operating device 10 of the hydraulic excavator 1) from the remote control device 200 and transmits them to the operating device 10.

[0043] The remote control device 200 includes: a remote control device (control device) 201, which is composed of an electronic device having communication and processing functions with the hydraulic excavator 1; a display device 202, which displays images; a right operating lever 203a for operating the boom cylinder 61 (boom 6) and the bucket cylinder 81 (bucket 8); a left operating lever 203b for operating the arm cylinder 71 (arm 7) and the swing hydraulic motor 5 (upper swing unit 3b); a right traveling lever 203c for operating the right traveling hydraulic motor 4a (lower traveling unit 3a); a left traveling lever 203d for operating the left traveling hydraulic motor 4b (lower traveling unit 3a); and a driver's seat 204. Hereinafter, the right operating lever 203a, the left operating lever 203b, the right traveling lever 203c, and the left traveling lever 203d may be collectively referred to as the remote control device 203. The remote operating device 203 inputs operation signals (e.g., electrical signals corresponding to the operator's operation amount and direction) to the remote control device 201. The remote control device 201 then transmits control signals (including operation signals from the operating device 10 of the hydraulic excavator 1) to the communication device 90 mounted on the hydraulic excavator 1 via the network, thereby remotely controlling the hydraulic excavator 1. The display device 202 provides visual information to the remote operator by displaying a superimposed image from the remote control device 201 (as described later).

[0044] <Solenoid valve unit (front control hydraulic unit) 160>

[0045] like Figure 3 As shown, the solenoid valve unit (in Figure 3 Only the front control hydraulic unit is shown) 160 comprises: electromagnetic proportional valves 54a~56b, whose primary port side is connected to the pilot pump 30 via the pump pipe 143a, reducing the pilot pressure from the pilot pump 30 and outputting it to the pilot pipes 144a~146b; and electromagnetic proportional valves 57a~59b, which are connected to the pilot pump 30 via the pump pipe 143a. Figure 3 Omitted due to paper reasons (see also Figure 2 ), but similarly, the pilot pressure from the pilot pump 30 is reduced in pressure and output to the pilot lines 147a to 149b.

[0046] The electromagnetic proportional valves 54a to 59b are opened to the minimum when not energized, and their opening increases with increasing current as a control signal from the controller 40. Thus, the openings of the electromagnetic proportional valves 54a to 59b also correspond to the control signals from the controller 40.

[0047] In the solenoid valve unit 160 constructed in the above manner, if a control signal is output from the controller 40 to drive the solenoid proportional valves 54a~59b, a pilot pressure can be generated even if the operator does not operate the corresponding operating device 10, thereby forcing the action of each actuator (4, 5, 61, 71, 81).

[0048] <Remote control device 201>

[0049] Figure 4 FIG. 2 is a diagram showing the hardware configuration of the remote control device 201 of the remote control device 200 according to this embodiment. Figure 4 In FIG, the remote control device 201 includes a communication interface 211 , a central processing unit (CPU) 212 as a processor, a read-only memory (ROM) 213 and a random access memory (RAM) 214 as storage devices, an input interface 215 , and an output interface 216 .

[0050] The communication interface 211 sends and receives signals to and from the communication device 90 mounted on the hydraulic excavator 1. Images from the hydraulic excavator 1's onboard camera and vehicle body information are received from the communication device 90 and input to the CPU 212. Furthermore, the communication device 211 transmits control signals from the remote control device 203 to the communication device 90. Based on the control program stored in the ROM 213, the CPU 212 performs predetermined computations on the signals received from the communication interface 211, the ROM 213, and the RAM 214. The ROM 213 is a recording medium that stores control programs for executing control content, including the processing in the flowcharts described below, and various information necessary for executing the flowcharts.

[0051] The input interface 215 receives a control signal from the remote operation device 203 as an input signal, and outputs the control signal to the communication interface 211 .

[0052] The output interface 216 generates an output signal corresponding to the calculation result of the CPU 212 , and displays the signal on the screen of the display device 202 .

[0053] Figure 5 This is a functional block diagram of the remote manipulation control device 201 of the remote manipulation device 200 according to the present embodiment.

[0054] exist Figure 5 In FIG, the remote control device 201 includes a predicted trajectory calculation unit 220 , a display control unit 221 , and a communication control unit 222 .

[0055] The predicted trajectory calculation unit 220 calculates the predicted trajectory based on the data from the communication interface 211 ( Figure 4) to calculate the predicted trajectory of the hydraulic excavator 1 and output the predicted trajectory data. For example, based on the sensor values ​​of various parts transmitted from the communication device 90 mounted on the hydraulic excavator 1, an image signal (corresponding to the predicted trajectory data) depicting the posture of the hydraulic excavator 1 in one second is output to the display control unit 221.

[0056] The display control unit 221 communicates the calculation result of the predicted trajectory calculation unit 220 of the remote control device 201 with the communication interface 211 ( Figure 4 ) is overlapped with the vehicle-mounted camera image of the hydraulic excavator 1. The overlapped image signal is output to the display device 202.

[0057] The communication control unit 222 packages the transmitted signal and transmits it to the communication device 90 of the hydraulic excavator 1. The communication control unit 222 also decodes the packaged signal from the communication device 90 so that the predicted trajectory calculation unit 220 and the display control unit 221 can read it.

[0058] <Calculation Flow of the Predicted Trajectory Calculation Unit 220>

[0059] exist Figure 6 The calculation process performed by the predicted trajectory calculation unit 220 of the remote control device 201 of this embodiment is shown in FIG. Figure 5 Implementation is performed using the signals of the communication device 90 shown in FIG.

[0060] In S600 , an input signal from the communication control unit 222 is detected and the process proceeds to S610 .

[0061] In S610, the predicted trajectories of the boom 6, arm 7, bucket 8, and upper swing unit 3b are calculated based on the input signals detected in S600. In this embodiment, the predicted trajectories of the boom 6, arm 7, bucket 8, and upper swing unit 3b are calculated based on the positions, velocities, and accelerations of the boom 6, arm 7, bucket 8, and upper swing unit 3b stored in the vehicle body information. The vector direction and magnitude are calculated every predetermined number of seconds from the current time. After the calculations are completed, the process proceeds to S620.

[0062] In S620, the vector direction and magnitude, resulting from the calculation, are converted into an interface compatible with the processing of the display control unit 221. It should be noted that conversion is not performed if it is not necessary. In this embodiment, the vector direction and magnitude are converted into point cloud data. The calculation result with the adjusted interface is output to the display control unit 221, and the processing ends.

[0063] <Control Flow of Display Control Unit 221>

[0064] Figure 7The control flow implemented by the display control unit 221 of the remote control device 201 of this embodiment is shown in FIG. Figure 5 This is implemented by inputting a signal from the communication device 90 shown in FIG. 1 and a calculation result from the predicted trajectory calculation unit 220 .

[0065] In S700 , input signals from the communication control unit 222 and the predicted trajectory calculation unit 220 are detected, and the process proceeds to S710 .

[0066] In S710, the predicted trajectory detected in S700 is plotted as an image signal based on the calculation results of the predicted trajectory calculation unit 220. In this embodiment, the predicted trajectory is plotted using a predicted image of the vehicle body (predicted image data of the hydraulic excavator 1 as viewed from a camera (camera viewpoint), generated based on the calculation results (predicted trajectory) of the predicted trajectory calculation unit 220, also referred to as a simulated camera image in this specification) and symbols indicating the direction and magnitude of the vector that gradually change with each passing second. After the plotting is completed, the process proceeds to S720.

[0067] In S720 , the predicted trajectory (image signal: simulated camera image) drawn in S710 is superimposed on the image signal from the communication control unit 222 detected in S700 (that is, the actual image of the onboard camera of the hydraulic excavator 1 ) and the process proceeds to S730 .

[0068] In S730, the image signal superimposed in S720 is output to the display device 202 and the process is terminated. In the display device 202, based on the image signal sent from the display control unit 221, the predicted trajectory (simulated camera image) and the image of the vehicle-mounted camera of the hydraulic excavator 1 (the actual image obtained by photographing the work site) are superimposed and displayed on the same screen (simultaneously) (see Figure 13 ).

[0069] <Control Flow of Communication Control Unit 222>

[0070] exist Figure 8 The control flow implemented by the communication control unit 222 of the remote control device 201 of this embodiment is shown in FIG. Figure 5 This is implemented by inputting a signal from the remote operation device 203 or the communication device 90 shown in FIG.

[0071] In S800 , an input signal (received signal) from the remote operation device 203 or the communication device 90 is detected and the process proceeds to S810 .

[0072] In S810 , if the input signal detected in S800 is an input signal from the remote operation device 203 , the process proceeds to S820 . Otherwise, the process proceeds to S830 .

[0073] In S820, the input signal is packaged and sent to the communication device 90, and the processing ends.

[0074] In S830, if the input signal detected in S800 is an input signal from the communication device 90, the process proceeds to S840. Otherwise, the process proceeds to S850.

[0075] In S840 , the data is decoded and outputted so as to be readable by the predicted trajectory calculation unit 220 and the display control unit 221 , and the process ends.

[0076] In S850, since all processing conditions are not met, the input signal is discarded and the processing ends.

[0077] <Action / Effect>

[0078] As described above, the control device (remote control device) 201 of the remote control device 200 of this embodiment includes: a communication control unit 222, which receives the camera image of the work site captured by the vehicle-mounted camera 91 and the body information of the work machine (hydraulic excavator) 1; a predicted trajectory calculation unit 220, which calculates the predicted trajectory of the work machine (hydraulic excavator) 1 based on the body information, and outputs predicted trajectory data 202 for displaying on the display device in the form of an image; and a display control unit 221, which enables the display device 202 to display the camera image and the image of the predicted trajectory on the same screen (simultaneously).

[0079] In addition, the display control unit 221 causes the display device 202 to display the camera image and the image of the predicted trajectory in an overlapping manner on the same screen.

[0080] In addition, the display control unit 221 generates a simulated camera image (simulated) as a predicted image of the working machine (hydraulic excavator) 1 when observed from the vehicle-mounted camera 91 based on the predicted trajectory, and causes the display device 202 to display the camera image and the simulated camera image overlapping on the same screen.

[0081] That is, the remote control device 200 of this embodiment superimposes the vehicle camera image and the predicted trajectory image and displays them to the operator at the remote location, thereby transmitting the vehicle body status to the operator independently of image data and visually notifying the operator of image delay.

[0082] In the remote control system 100 including the remote control device 200 configured as described above, even when image transmission is delayed due to communication line conditions, the operator can still recognize the current status of the vehicle body by displaying the image of the predicted trajectory. Furthermore, by superimposing a simulated camera image (a predicted image of the hydraulic excavator 1 as viewed from a camera, generated based on the predicted trajectory), the difference from the actual camera image is displayed when communication line delays occur, visually notifying the operator of image delays. Consequently, remote control of the work machine (hydraulic excavator) 1 from a remote location is possible, even in the presence of communication delays.

[0083] [Second embodiment]

[0084] use Figures 9-12 A second embodiment of the present invention will be described. Figure 9 This is a functional block diagram of the remote manipulation control device 201a of the remote manipulation device 200a according to the present embodiment.

[0085] Based on the first embodiment, Figure 9 As shown, the remote control device 201a of the remote control device 200a of the second embodiment is configured by adding a communication status determination unit 223 and a communication status transmission unit 224 , and the image signal superimposed in the display control unit 221 is input to the communication status determination unit 223 .

[0086] <Remote control device 201a>

[0087] The communication status determination unit 223 determines the transmission status of the communication line between the hydraulic excavator 1 and the remote control device 200a based on the difference between the predicted trajectory (simulated camera image) of the image signal input from the display control unit 221 and the vehicle-mounted camera image, and outputs the determination result to the communication status transmission unit 224 and the display device 202.

[0088] The communication status transmitting unit 224 transmits the determination result of the communication status determining unit 223 (the determination result of the transmission status of the communication line) to the communication device 90 of the hydraulic excavator 1 via the communication control unit 222 a .

[0089] <Calculation Flow of Communication Status Determination Unit 223>

[0090] Figure 10 The control flow chart of the communication status determination unit 223 of the remote control device 201a according to the second embodiment is shown in FIG. Figure 9 This is implemented by the output of the display control unit 221 shown in FIG.

[0091] In S1000 , an input signal from the display control unit 221 is detected and the process proceeds to S1010 .

[0092] In S1010, the communication status is determined based on the input signal from the display control unit 221 detected in S1000. In this embodiment, the delay time is calculated based on the difference distribution of each feature point between the actual camera image and the simulated camera image within the superimposed image using a delay calculation table. After the calculation of the communication line delay time is completed, the process proceeds to S1020.

[0093] In S1020, the delay time calculated in S1010 is output to the communication status transmission unit 224 and the display device 202, and the processing is terminated. The display device 202 displays the delay time as the determination result sent from the communication status determination unit 223 together with the predicted trajectory (simulated camera image) and the vehicle-mounted camera image of the hydraulic excavator 1 (the actual image obtained by photographing the work site) on the same screen (see Figure 13 ).

[0094] <Control Flow of Communication Status Transmitter 224>

[0095] Figure 11 The control flow chart of the communication status transmitting unit 224 of the remote control device 201a according to the second embodiment is shown in FIG. Figure 9 This is implemented based on the output of the communication status determination unit 223 shown in FIG.

[0096] In S1100 , an input signal from the communication status determination unit 223 is detected and the process proceeds to S1110 .

[0097] In S1110, the input signal from the communication status determination unit 223 detected in S1100 is output to the communication control unit 222a. In this embodiment, the input signal is converted into one of the operation signals of the remote operation device 203 and output to the communication control unit 222a. After the input signal is output to the communication control unit 222a, the process ends.

[0098] <Control Flow of Communication Control Unit 222a>

[0099] Figure 12 The control flow implemented by the communication control unit 222a of the remote control device 201a of the second embodiment is shown in FIG. Figure 9 This is implemented by a signal from the remote operation device 203, the communication device 90, or the communication status transmitting unit 224 shown in FIG.

[0100] In S1200 , an input signal (received signal) from the remote operation device 203 , the communication device 90 , or the communication status transmitting unit 224 is detected, and the process proceeds to S1210 .

[0101] In S1210, if the input signal detected in S1200 is an input signal from the remote operation device 203 or the communication status transmitting unit 224, the process proceeds to S1220. Otherwise, the process proceeds to S1230.

[0102] In S1220, the input signal is packaged and sent to the communication device 90, and the processing ends.

[0103] In S1230, if the input signal detected in S1200 is an input signal from the communication device 90, the process proceeds to S1240. Otherwise, the process proceeds to S1250.

[0104] In S1240 , the data is decoded and outputted so as to be readable by the predicted trajectory calculation unit 220 and the display control unit 221 , and the process ends.

[0105] In S1250, the input signal is discarded because it does not meet all the processing conditions, and the processing ends.

[0106] <Operation of Hydraulic Excavator (Working Machine) 1>

[0107] The hydraulic excavator (working machine) 1 of this embodiment suppresses vehicle body movement based on the determination result of the communication status determination unit 223 received from the communication status transmission unit 224. For example, the hydraulic excavator 1 includes a table of upper limits for movement speeds corresponding to the delay time corresponding to the determination result of the communication status determination unit 223. As the delay time of the determination result of the communication status determination unit 223 increases, the upper limit for movement speed is further suppressed, resulting in slower movement. Alternatively, if the communication status determination unit 223 determines that communication is interrupted, the hydraulic excavator 1 stops all drive units.

[0108] It should be noted that, in this example, the communication status sending unit 224 transmits the determination result (delay time) of the communication status determination unit 223 directly to the communication device 90 of the hydraulic excavator 1. However, in the case where, for example, only the delay time calculated by the communication status determination unit 223 is greater than a prescribed threshold value and a communication failure is determined, the determination result (delay time) of the communication status determination unit 223 may also be transmitted to the communication device 90 of the hydraulic excavator 1.

[0109] <Action / Effect>

[0110] As described above, the remote control device 200a of this embodiment further includes a communication status determination unit 223 that determines the transmission status of the communication line between the working machine (hydraulic excavator) 1 and the remote control device 200 based on the difference between the camera image and the simulated camera image.

[0111] In addition, the communication status determination unit 223 causes the display device 202 to display the determination result of the transmission status of the communication line.

[0112] Furthermore, the system further includes a communication status transmitting unit 224 for transmitting a determination result of the transmission status of the communication line to the working machine (hydraulic excavator) 1 (for example, when the communication status determining unit 223 determines that there is a communication failure).

[0113] In the remote control system 100 and the hydraulic excavator 1, including the remote control device 200a configured as described above, by overlaying the two images, not only can the delay status be visually communicated to the operator, as in the first embodiment, but the delay time can also be determined as a result of the delay determination. Furthermore, since the delay determination result can be transmitted to the hydraulic excavator 1, the hydraulic excavator 1 can be controlled accordingly.

[0114] It should be noted that, in the above embodiment, the predicted trajectory (simulated camera image) and the image of the vehicle-mounted camera of the hydraulic excavator 1 (the actual image obtained by shooting the work site) are (simultaneously) overlapped and displayed on the same screen in the display device 202 (see Figure 13 ), for example, Figure 14 As shown, the predicted trajectory (simulated camera image) and the vehicle-mounted camera image of the hydraulic excavator 1 (the actual image obtained by shooting the work site) can also be displayed at different positions on the same screen (see Figure 14 ).

[0115] It should be noted that the present invention includes various variations and is not limited to the above-described embodiments. For example, the above-described embodiments are detailed descriptions for the purpose of clearly and easily explaining the present invention, but are not limited to all of the described configurations. In addition, a portion of the configuration of a certain embodiment can be converted into the configuration of another embodiment, and a configuration of another embodiment can be added to the configuration of a certain embodiment. In addition, other configurations can be added, deleted, or replaced with a portion of the configuration of each embodiment.

[0116] In addition, the various functions of the controller of the above embodiment can also be implemented by hardware by partially or entirely designing them as integrated circuits. Alternatively, they can be implemented by software by interpreting and executing programs that implement the various functions on a processor. In addition to being stored in a storage device within the controller, information such as programs, tables, and files that implement the various functions can also be stored in a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0117] Description of Reference Signs

[0118] 1…Hydraulic excavator (working machinery)

[0119] 2…Front working part (working device)

[0120] 3…walking body

[0121] 4…Travel hydraulic motor

[0122] 5…Swing hydraulic motor

[0123] 6… boom

[0124] 7…arm

[0125] 8…Bucket

[0126] 9…Bucket connecting rod

[0127] 10…Operating device

[0128] 11…Engine

[0129] 12…Cab

[0130] 15…Flow control valve

[0131] 16…Load detection device

[0132] 20…Hydraulic pump

[0133] 30…Pilot pump

[0134] 31…Body tilt angle sensor

[0135] 32…Rotary angle sensor

[0136] 39…Lock valve

[0137] 40…Controller

[0138] 54~59…Solenoid proportional valve

[0139] 61…Boom cylinder

[0140] 62…Boom angle sensor

[0141] 71…Arm cylinder

[0142] 72…Arm angle sensor

[0143] 81…Bucket cylinder

[0144] 82…Bucket angle sensor

[0145] 90…communication device

[0146] 91…Car camera

[0147] 100…Remote control system

[0148] 143a…Pump tube

[0149] 144a~149b…pilot tube

[0150] 150a~155b…Hydraulic drive unit

[0151] 160...Solenoid valve unit

[0152] 170…Container

[0153] 200…Remote control device

[0154] 201…Remote control device (control device)

[0155] 202…Display device

[0156] 203…Remote operating device

[0157] 204…Control seat

[0158] 211…Communication interface

[0159] 212…Central Processing Unit (CPU)

[0160] 213…Read-only memory (ROM)

[0161] 214…Random Access Memory (RAM)

[0162] 215…Input interface

[0163] 216…output interface

[0164] 220…Prediction trajectory calculation unit

[0165] 221…Display control unit

[0166] 222…Communication Control Unit

[0167] 223 ...Communication Status Determination Unit (Second Embodiment)

[0168] 224 ...Communication status transmitting unit (Second embodiment)

Claims

1. A remote control device comprising a control device that transmits an operation signal to a working machine and controls the working machine from a remote location, wherein: The control device comprises: a communication control unit that receives a camera image of the work site captured by a camera and body information of the work machine; a predicted trajectory calculation unit that calculates a predicted trajectory of the work machine based on the vehicle body information and outputs predicted trajectory data for display on a display device in the form of an image; and a display control unit that causes the display device to display the camera image and the image of the predicted trajectory on the same screen; The display control unit generates a simulated camera image based on the predicted trajectory, and causes the display device to display the camera image and the simulated camera image superimposed on the same screen, wherein the simulated camera image is a predicted image of the working machine when viewed from the camera.

2. The remote control device according to claim 1, wherein: The remote control device further includes a communication status determination unit configured to determine a transmission status of a communication line between the working machine and the remote control device based on a difference between the camera image and the simulated camera image.

3. The remote control device according to claim 2, wherein: The communication status determination unit causes the display device to display a determination result of the transmission status of the communication line.

4. The remote control device according to claim 2, wherein: The device further includes a communication status transmitting unit configured to transmit a determination result of the transmission status of the communication line to the working machine.

5. The remote control device according to claim 2, wherein: The working machine further includes a communication status transmitting unit configured to transmit a determination result of the transmission status of the communication line to the working machine when the communication status determining unit determines that there is a communication failure.

Images