Remote control system and method for operating machinery from a remote location.

Abstract

To appropriately operate an on-board information terminal located inside the driver's cab of a ride-on work machine, in accordance with the content displayed on the terminal, from a remote location. [Solution] The remote control system 100 for a work machine is a remote control system for operating the work machine 1 from a remote location. It comprises a remote display device 200 located away from the work machine 1, a remote input device 202 that receives input operations for the remote display device 200, and one or more processors. The processor displays the captured image 310 received from the work machine 1 on the remote display device 200, generates an operation signal for operating an in-vehicle information terminal located in the driver's cab of the work machine 1 based on the input operation for the captured image 310 displayed on the remote display device 200, and outputs the generated operation signal.

Description

Technical Field

[0001] The present disclosure relates to a remote operation system and a remote operation method for operating a work machine from a remote location.

Background Art

[0002] Conventionally, remote operation technologies are known that operate the traveling device and work device of an existing ride-on work machine with operation signals from a remote location. Also known is a technology that allows an operator at a remote location to recognize a cluster gauge by capturing an image of the cluster gauge disposed in the cab of the work machine with a camera and displaying the captured image on a remote operation device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problem to be Solved by the Invention

[0004] Incidentally, an in-vehicle information terminal provided in the cab of a ride-on work machine can be directly operated by an occupant. When a work machine equipped with such an operable in-vehicle information terminal is remotely operated from a remote location, although the conventional technology can visually confirm the content displayed on the in-vehicle information terminal, it is difficult to appropriately operate the in-vehicle information terminal from the remote location in accordance with the displayed content.

[0005] The present disclosure aims to appropriately operate the in-vehicle information terminal from a remote location in accordance with the content displayed on the in-vehicle information terminal provided in the cab of a ride-on work machine.

Means for Solving the Problem

[0006] According to this disclosure, a remote control system for a work machine is a remote control system for operating a work machine from a remote location. It comprises a remote display device located away from the work machine, a remote input device that receives input operations to the remote display device, and one or more processors. The processor displays an image captured from the work machine on the remote display device, generates an operation signal for operating an in-vehicle information terminal located in the operator's cab of the work machine based on the input operation to the image captured on the remote display device, and outputs the generated operation signal.

[0007] According to this disclosure, the remote control system for a work machine is a remote control system for operating a work machine from a remote location. It comprises an onboard information terminal located in the operator's cabin of the work machine and displaying information including operating information of the work machine; a camera for capturing images of the display unit of the onboard information terminal; a communication device for transmitting the images captured by the camera to a remote location; a remote display device located at the remote location; a remote input device for receiving input operations to the remote display device; and one or more processors. The onboard information terminal displays a screen related to application software executed to assist the work machine in performing tasks. The processor displays the captured images on the remote display device, generates operation signals for operating the onboard information terminal based on operations on the captured images displayed on the remote display device, and transmits the generated operation signals to the onboard information terminal.

[0008] According to this disclosure, a remote operation method for a work machine is a remote operation method for operating a work machine from a remote location. It includes: displaying an image captured from the work machine on a remote display device located away from the work machine; generating an operation signal for operating an in-vehicle information terminal located in the driver's cab of the work machine based on an input operation on the image captured on the remote display device; and outputting the generated operation signal. [Effects of the Invention]

[0009] According to this disclosure, an in-vehicle information terminal installed in the driver's cab of a ride-on work machine can be appropriately operated remotely in accordance with the content displayed on the terminal. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 is a schematic diagram showing a remote control system according to the first embodiment. [Figure 2] Figure 2 is a side view showing a work machine according to the first embodiment. [Figure 3] Figure 3 shows the operator's cab of the work machine according to the first embodiment. [Figure 4] Figure 4 is a block diagram showing the control system of a work machine according to the first embodiment. [Figure 5] Figure 5 is a block diagram showing the remote controller of the remote control room according to the first embodiment. [Figure 6] Figure 6 is a block diagram showing a computer system according to the first embodiment. [Figure 7] Figure 7 shows an example of a screen for setting the extraction range of an image captured on a remote display device. [Figure 8] Figure 8 shows an example of an image captured and displayed on a remote display device. [Figure 9] Figure 9 is a schematic diagram illustrating the area where the positions corresponding to the four corners can be specified. [Figure 10] Figure 10 is a flowchart showing the operation of the remote controller according to the first embodiment. [Figure 11] Figure 11 shows an example of an image displayed on an information display device. [Figure 12] Figure 12 shows an example of an operation icon image displayed on a remote display device. [Figure 13] Figure 13 is a block diagram showing the remote controller of the remote control room according to the second embodiment. [Figure 14] Figure 14 is a flowchart showing the operation of the remote controller according to the second embodiment. [Mode for Carrying Out the Invention]

[0011] <First Embodiment> [Remote Control System] Figure 1 is a schematic diagram illustrating a remote control system 100 according to the first embodiment. The remote control system 100 for a work machine 1 remotely controls the work machine 1 present at a work site from a remote display device 200 in a remote control room separated from the work site. The work machine 1 and the remote display device 200 are connected by a communication facility 30 so that data can be transmitted and received therebetween.

[0012] [Work Machine] Figure 2 is a side view illustrating the work machine 1 according to the first embodiment. The work machine 1 performs work at a work site. In the embodiment, the work machine 1 is a hydraulic excavator. Examples of work that the work machine 1 can perform include excavation work, earth discharging work, and loading work. The work machine 1 includes a main body 2, a traveling body 3, a work implement 4, work implement cylinders 5, and a power source 6.

[0013] The main body 2 is supported by the traveling body 3. The traveling body 3 has a pair of crawler tracks 3A. The work machine 1 travels as the crawler tracks 3A rotate. The main body 2 is disposed above the traveling body 3. The main body 2 is rotatably supported by the traveling body 3. The main body 2 has a cab 7. An operator of the work machine 1 rides in the cab 7.

[0014] The work implement 4 is rotatably attached to the main body 2. The work implement 4 includes a boom 4A, an arm 4B, and a bucket 4C. The boom 4A is rotatably connected to the front portion of the main body 2. The arm 4B is rotatably connected to the distal end portion of the boom 4A. The bucket 4C is rotatably connected to the distal end portion of the arm 4B.

[0015] The work implement cylinder 5 operates the work implement 4 attached to the main body 2. The work implement cylinder 5 is a hydraulic cylinder driven by hydraulic fluid. The work implement cylinder 5 includes a boom cylinder 5A, an arm cylinder 5B, and a bucket cylinder 5C.

[0016] The boom cylinder 5A operates the boom 4A. The operation of the boom 4A includes raising and lowering movements. When the boom cylinder 5A extends, the boom 4A moves upward. When the boom cylinder 5A retracts, the boom 4A moves downward.

[0017] The arm cylinder 5B operates the arm 4B. The operation of the arm 4B includes digging and dumping operations. When the arm cylinder 5B extends, the arm 4B performs the digging operation. When the arm cylinder 5B retracts, the arm 4B performs the dumping operation.

[0018] Bucket cylinder 5C operates bucket 4C. The operation of bucket 4C includes digging and dumping. Bucket 4C digs when bucket cylinder 5C extends. Bucket 4C dumps when bucket cylinder 5C retracts.

[0019] The power source 6 generates power to drive the work machine 1. The power source 6 is, for example, an internal combustion engine or an electric motor driven by an energy storage device. The power generated by the power source 6 operates the traveling body 3, the main body 2, and the work machine 4. The power generated by the power source 6 drives a hydraulic pump (not shown). When the hydraulic pump is driven, hydraulic fluid is supplied from the hydraulic pump to the work machine cylinder 5, and the work machine 4 operates.

[0020] Figure 3 shows the operator's cab 7 of the work machine 1 according to the first embodiment. The operator sits in a seat 13 located in the operator's cab 7. As shown in Figure 3, the operator's cab 7 is equipped with an operating device 9 and an information display device 11.

[0021] The operating device 9 is operated by an operator. The operating device 9 is operated to operate at least one of the traveling body 3, the main body 2, and the work equipment 4. The operating device 9 includes a left operating lever 9A and a right operating lever 9B, a left traveling lever 9C and a right traveling lever 9D, and a left foot pedal 9E and a right foot pedal 9F.

[0022] The left control lever 9A is operated to rotate the main body 2. The left control lever 9A is also operated to operate the work implement 4. The right control lever 9B is operated to operate the work implement 4. When the left control lever 9A is operated in the left-right direction, the main body 2 rotates to the left or right. When the left control lever 9A is operated in the front-back direction, the arm 4B performs a dumping or digging operation. When the right control lever 9B is operated in the left-right direction, the bucket 4C performs an digging or dumping operation. When the right control lever 9B is operated in the front-back direction, the boom 4A lowers or raises. Alternatively, when the left control lever 9A is operated in the front-back direction, the main body 2 may rotate to the right or left, and when the left control lever 9A is operated in the left-right direction, the arm 4B may perform a dumping or digging operation. When the right control lever 9B is operated in the left-right direction or the front-back direction, the main body 2 may rotate to the left or right.

[0023] The left travel lever 9C and the right travel lever 9D are operated to move the vehicle 3. When the left travel lever 9C is operated in the forward / backward direction, the left track 3A of the vehicle 3 moves forward or backward. When the right travel lever 9D is operated in the forward / backward direction, the right track 3A of the vehicle 3 moves forward or backward.

[0024] The left foot pedal 9E is linked to the left travel lever 9C. The right foot pedal 9F is linked to the right travel lever 9D. By operating the left foot pedal 9E and the right foot pedal 9F, the vehicle 3 may move forward or backward.

[0025] The information display device 11 is positioned to the right front of the seat 13. The information display device 11 can display operating information of the work machine 1. As operating information of the work machine 1, the information display device 11 can display, for example, the temperature of the hydraulic oil, the temperature of the cooling water, the remaining amount of energy required to operate the power source 6, and the total operating time of the work machine 1. The information display device 11 may be, for example, a tablet terminal or a slate PC. In this embodiment, the information display device 11 is an example of an in-vehicle information terminal. The information display device 11 has an input device 111 and a display unit 112. More details about the information display device 11 will be described later.

[0026] Figure 4 is a block diagram showing the control system of the work machine 1 according to the first embodiment. As shown in Figure 4, the work machine 1 includes a position sensor 20, an attitude sensor 21, an angle sensor 22, and a first camera 23.

[0027] The position sensor 20 detects the position of the work machine 1. The position sensor 20 detects the position of the work machine 1 using the Global Navigation Satellite System (GNSS). The Global Navigation Satellite System detects the position in a global coordinate system defined by latitude, longitude, and altitude coordinate data. A global coordinate system is a coordinate system fixed to the Earth. The position sensor 20 includes a GNSS receiver and detects the position of the work machine 1 in the global coordinate system. In this embodiment, at least two position sensors 20 are provided on the main body 2. The bearing of the main body 2 is calculated based on the detection data from each of the two position sensors 20.

[0028] The attitude sensor 21 detects the attitude of the main body 2. The attitude sensor 21 detects the tilt angle of the main body 2 with respect to the horizontal plane. The attitude sensor 21 includes an inertial measurement unit (IMU). The attitude sensor 21 is positioned on the main body 2.

[0029] The angle sensor 22 detects the angle of the work implement 4. The angle sensor 22 includes a boom angle sensor 22A that detects the angle of the boom 4A relative to the main body 2, an arm angle sensor 22B that detects the angle of the arm 4B relative to the boom 4A, and a bucket angle sensor 22C that detects the angle of the bucket 4C relative to the arm 4B. The angle sensor 22 may also be a stroke sensor that detects the stroke of the work implement cylinder 5, a potentiometer that detects the angle of the work implement 4, or a work implement IMU that detects the posture of the work implement 4.

[0030] The first camera 23 captures images of the area around the work machine 1. Multiple first cameras 23 are provided on the main body 2. The first cameras 23 include, for example, a front camera for capturing images in front of the main body 2, a rear camera for capturing images behind the main body 2, a left camera for capturing images to the left of the main body 2, and a right camera for capturing images to the right of the main body 2. The number and imaging range of the first cameras 23 are arbitrary. An overhead view image of the work machine 1 is generated based on the images captured by each of the multiple first cameras 23.

[0031] The second camera (camera) 24 captures images including the display unit 112 of the information display device 11, which is an on-board information terminal of the work machine 1 and is located in the driver's cab 7. Multiple second cameras 24 are provided in the driver's cab 7 of the main unit 2.

[0032] The communication device 25 transmits image data of images captured from the first camera 23 and the second camera 24, as well as vehicle data, etc., to the remote controller 220 via the communication system NW. The communication device 25 receives operation signals from the remote controller 220 via the communication system NW.

[0033] Examples of communication systems (NW) include the internet, local area networks (LANs), mobile phone networks, and satellite communication networks.

[0034] [Machine Controller] As shown in Figure 4, the work machine 1 includes a machine controller 14. The machine controller 14 includes a computer. The machine controller 14 comprises a machine control unit 141, a peripheral monitoring control unit 142, a guidance control unit 143, and a measurement unit 144.

[0035] The machine controller 14 controls at least the main body 2, the traveling body 3, the work equipment 4, and the power source 6. The machine controller 14 outputs operation data to display operation information of the work machine 1 on the information display device 11.

[0036] In this embodiment, the operation signal from the operating device 9, the detection data from the position sensor 20, the detection data from the attitude sensor 21, the detection data from the angle sensor 22, and the captured image from the first camera 23 are input to the machine controller 14. The peripheral monitoring control unit 142, the guidance control unit 143, the machine control unit 141, and the measurement unit 144 of the machine controller 14 can perform calculation processing and image processing based on the operation signal, the detection data, and the captured image, respectively. By performing calculation processing and image processing, the machine controller 14 can perform functions to support the execution of tasks by the work machine 1.

[0037] The machine control unit 141 is configured to perform an automatic control function. The automatic control function refers to a function that automatically controls at least the work machine 4.

[0038] The surrounding monitoring control unit 142 is configured to perform a surrounding monitoring function. The surrounding monitoring function refers to the function of detecting obstacles in the vicinity of the main unit 2. The surrounding monitoring control unit 142 will be described later.

[0039] The guidance control unit 143 is configured to perform a guidance function. The guidance function refers to the function of providing position information of the work machine 4. The guidance control unit 143 will be described later.

[0040] The measurement unit 144 is configured to perform a payload function. The payload function refers to the function of detecting the weight of the excavated material held in the bucket 4C of the work machine 4. The measurement unit 144 will be described later.

[0041] The machine controller 14 receives input data generated by the input device 111 of the information display device 11. Based on the input data generated by the input device 111 of the information display device 11, the machine controller 14 can perform calculations and image processing.

[0042] [Information display device] The information display device 11 comprises a controller 110, an input device 111, and a display unit 112.

[0043] The input device 111 generates input data when operated by an operator. In this embodiment, the display unit 112 is a touchscreen. The input device 111 includes a touch sensor provided on the display unit 112. The input device 111 may also include physical buttons such as button switches, a mouse, or a keyboard.

[0044] The display unit 112 includes a touch panel that functions as an input device 111. The display unit 112 can provide the operator with various information, not just operating information of the work machine 1. In one embodiment, the display unit 112 can display screens related to application software, which will be described later, in addition to operating information of the work machine 1. A panel display such as a liquid crystal display or an organic EL display is an example of the display unit 112.

[0045] The controller 110 controls the operation of the information display device 11. The controller 110 includes a display control unit 120. The display control unit 120 controls the display of the display unit 112.

[0046] The controller 110 transmits the input data generated by the input device 111 to the machine controller 14. For example, the controller 110 detects a touch operation on an icon displayed on the screen by the input device 111 and transmits an execution command for the application belonging to the touched icon to the machine controller 14.

[0047] [Application Software] Multiple software programs are stored in the controller 110 of the information display device 11. These programs are computer programs. The software includes an operating system and application software. The operating system is the basic software. Application software is software applied to various purposes. The storage of the controller 110 stores the operating system and multiple application software programs. In the following description, application software will be referred to as "application" as appropriate. In this embodiment, an application may include multiple applications.

[0048] The controller 110 can execute applications. The processor of the controller 110 executes the processing corresponding to the application by reading the operating system and applications from storage, loading them into main memory, and executing them. The processor of the controller 110 stores the processing data in storage. The controller 110 has one or more processors.

[0049] The application is software for performing functions related to tasks using machine 1. The application is executed to perform tasks using machine 1. The application is executed to assist in the execution of tasks using machine 1. The application is executed to assist the operator.

[0050] The controller 110 causes the screen to be displayed on the display unit 112 when an application is executed. In this embodiment, the screen includes a plurality of screens related to each application. The controller 110 causes each screen to be displayed on the display unit 112 when each application is executed.

[0051] The controller 110 displays, for example, operating information of the work machine on the display unit 112 of the information display device 11. The controller 110 also displays, for example, screens related to applications that support the execution of tasks by the work machine, such as the screen of the surrounding monitoring system, machine guidance, payload meter, and machine control, on the display unit 112 of the information display device 11.

[0052] The application is, for example, an application for monitoring the surroundings of the work machine 1. When this application is executed, a screen showing the surroundings is displayed on the display unit 112.

[0053] The screen may include, for example, an overhead view of the area surrounding the work machine 1.

[0054] The peripheral monitoring control unit 142 of the machine controller 14 can detect obstacles around the work machine 1 based on the image captured by the first camera 23. When an obstacle is detected around the work machine 1, the screen includes an image of the detected obstacle. An obstacle is an object that hinders the work of the work machine 1. Examples of obstacles include people, vehicles, and rocks. In this embodiment, the peripheral monitoring control unit 142 extracts features from the image captured by the first camera 23. Based on the extracted features and dictionary data of obstacles, the peripheral monitoring control unit 142 detects obstacles from the image captured by the first camera 23. Examples of obstacle detection methods include pattern matching and object detection processing based on machine learning. If the work machine 1 is equipped with a non-contact sensor such as LiDAR (Light Detection and Ranging), the peripheral monitoring control unit 142 may detect obstacles around the work machine 1 based on detection data from the non-contact sensor. When an obstacle is detected, the surrounding monitoring control unit 142 generates a marker indicating the location of the obstacle and creates a screen in which the overhead view image, guide lines, and marker are superimposed on the model image. Because the location of the obstacle is emphasized by the marker superimposed on the overhead view image, the operator can easily recognize obstacles around the work machine 1.

[0055] The application is, for example, an application that presents the operator with the relative position between the cutting edge of bucket 4C and a design surface set at the work site. The design surface refers to the target construction surface that indicates the target shape of the work object. When this application is executed, a guidance screen showing the relative position between the cutting edge of bucket 4C and the design surface is displayed on the display unit 112.

[0056] The guidance control unit 143 of the machine controller 14 can calculate the relative position between the reference position of the main body 2 and the cutting edge of the bucket 4C based on the detection data of the angle sensor 22 and the known dimensional data of the work implement 4. The reference position may be set to a position that coincides with the slewing axis RX of the main body 2, or it may be set to the connection point between the boom 4A and the main body 2. Based on the relative position between the reference position and the cutting edge of the bucket 4C, and the position data of the position sensor 20 and the detection data of the attitude sensor 21, the guidance control unit 143 can calculate the position of the cutting edge of the bucket 4C in the global coordinate system.

[0057] When the control device 9 is operated and the work implement 4 is activated, the work implement in the model image moves in sync with the actual work implement 4. By operating the control device 9 while checking the screen, the operator can move the cutting edge of the bucket 4C to follow the design surface.

[0058] The application is, for example, an application for displaying the status of the automatic control of the work machine 1 to the operator. When the work machine 1 is to be controlled automatically, this application is executed. When this application is executed, the automatic control screen is displayed on the display unit 112.

[0059] In the automatic control of the main unit 2 and the work implement 4, an automatic control command is output from the machine control unit 141 of the machine controller 14. As described above, the position of the cutting edge of the bucket 4C in the global coordinate system is calculated based on the detection data of the position sensor 20, the detection data of the attitude sensor 21, the detection data of the angle sensor 22, and the known data of the dimensions of the work implement 4. In the rotation operation, the machine controller 14 outputs an automatic control command so that the cutting edge of the bucket 4C moves according to a predetermined target trajectory.

[0060] The application is, for example, an application for displaying to the operator the weight of the excavated material held in bucket 4C and the total weight of the excavated material discharged to the excavation target 80. When this application is executed, a screen showing the measurement results is displayed on the display unit 112.

[0061] The weight of the excavated material held in the bucket 4C is detected by a weight sensor installed on the work machine 1. An example of a weight sensor is a pressure sensor that detects the pressure of the hydraulic fluid in the boom cylinder 5A. The measurement unit 144 can calculate the weight of the excavated material held in the bucket 4C based on the data detected by the weight sensor. The measurement unit 144 can calculate the total weight of the excavated material discharged to the soil removal target 80 based on the weight of the excavated material held in the bucket 4C and the number of automatically controlled hoist rotation movements.

[0062] [Remote Control Room] Figure 5 is a block diagram showing the remote controller 220 of the remote control room according to the first embodiment. The remote control room is located in a remote location, such as an administrative office, away from the work machine 1. The remote control room is configured to allow the work machine 1 to be operated from a distance. The remote control room is equipped with devices for remotely operating the work machine 1. For example, the remote control room may contain a remote display device 200, a remote control device 201, a remote input device 202, a communication device 210, and a remote controller 220.

[0063] The remote display device 200 is located away from the work machine 1. The remote display device 200 is positioned independently of the work machine 1. The remote display device 200 displays an image 300 (see Figure 8) captured by the second camera 24 on the display unit 112 of the information display device 11 of the work machine 1. The remote display device 200 includes a flat panel display such as a liquid crystal display or an organic EL display. The remote display device 200 includes a touch panel that functions as a remote input device 202.

[0064] The remote control device 201 consists of various control levers for remotely operating the work machine 1. The remote control device 201 is operated by an operator seated in a control seat located in the remote control room. The operator operates the remote control device 201 while viewing images displayed on the remote display device 200.

[0065] Operation signals indicating operations performed on each control lever of the remote control device 201 are transmitted via the communication system to the machine controller 14 through the remote controller 220, the communication device 210, and the communication device 25 of the work machine 1.

[0066] The remote control device 201 includes, for example, a left work lever and a right work lever operated for the operation of the slewing body 102 and the work implement 101, and a left travel pedal and a right travel pedal operated for the operation of the traveling body 103.

[0067] The remote input device 202 is an operating member that receives input operations for the remote display device 200. The remote input device 202 generates input data when operated by an operator in a remote control room. In one embodiment, the remote input device 202 is a mouse. The remote input device 202 may also be a touchscreen, in which case the remote input device 202 includes a touch sensor provided on the remote display device 200. Alternatively, the remote input device 202 may be a button switch or other physical operating member such as a keyboard.

[0068] The communication device 210 receives captured image data and vehicle data from the work machine 1 via the communication system NW. The communication device 210 transmits operation signals to the work machine 1 via the communication system NW.

[0069] [Remote controller] The remote controller 220 is connected to the remote display device 200 and the remote operation device 201. The remote controller 220 includes a numerical processing unit (processor), such as a CPU (Central Processing Unit). The remote controller 220 has one or more processors.

[0070] The remote controller 220 receives input data generated by the remote input device 202. Based on the input data generated by the remote input device 202, the remote controller 220 can perform calculations and image processing.

[0071] The remote display device 220 receives various data, including captured image data and vehicle data, from the work machine 1 via the communication system NW. The remote controller 220 transmits operation signals indicating operations performed on the remote control device 201 to the machine controller 14 of the work machine 1 via the communication system NW. The remote controller 220 transmits various data, including captured image data, to the remote display device 200.

[0072] The remote controller 220 and the machine controller 14 can communicate with each other via the communication system NW.

[0073] The remote controller 220 includes a storage unit 221, a display control unit 222, an image extraction unit 223, an operation signal generation unit 224, a coordinate transformation unit 225, and an operation type transformation unit 226.

[0074] The memory unit 221 sets the correspondence between the operation type performed by the remote input device 202 to the remote display device 200 and the operation type performed to the in-vehicle information terminal, and stores this as operation type information in the memory unit.

[0075] The types of operations performed by the remote input device 202 include, for example, pressing the mouse, moving the mouse, and releasing the mouse. Other types of operations performed by the remote input device 202 include, for example, clicking the mouse and double-clicking the mouse. Clicking the mouse consists of pressing the mouse and releasing it. Double-clicking the mouse is represented by repeatedly pressing and releasing the mouse in a short period of time. Dragging the mouse is represented by moving the mouse at intervals of, for example, 30 milliseconds while continuing to press the mouse. The types of operations performed by the remote input device 202 are not limited to these.

[0076] Examples of operations performed on the in-vehicle information terminal include tapping, double-tapping, flicking, swiping, sliding, long-pressing, dragging, and pinching in / out on the display unit 112 of the information display device 11. However, the types of operations performed on the in-vehicle information terminal are not limited to these.

[0077] The correspondence between the operation type performed by the remote input device 202 and the operation type performed on the in-vehicle information terminal can be arbitrarily set in advance by the service technician or operator of the work machine 1. For example, the operation of pressing the mouse, moving the mouse, and releasing the mouse performed by the remote input device 202 may be set to correspond to long press, swipe, and tap on the display unit 112 of the information display device 11, which is the in-vehicle information terminal, respectively.

[0078] The display control unit 222 displays an image on the remote display device 200 that corresponds to the screen displayed on the display unit 112 of the information display device 11, which is located in the operator's cab 7 of the work machine 1. In this embodiment, the display control unit 222 displays an captured image on the remote display device 200 as an image corresponding to the screen displayed on the display unit 112 of the information display device 11. For example, the display control unit 222 displays an captured image 300 (see Figure 8) captured by the second camera 24 on the remote display device 200. For example, the display control unit 222 displays an captured image 310 (see Figure 8) extracted by the image extraction unit 223 on the remote display device 200.

[0079] The image extraction unit 223 extracts the captured image 310 after extraction, which is an image of the extraction range corresponding to the display unit 112 of the information display device 11, from the captured image 300 before extraction displayed on the remote display device 200.

[0080] The image extraction unit 223 sets the extraction range (trimming range) of the captured image 300 by specifying positions corresponding to the four corners of the display unit 112 of the information display device 11 on the captured image 300 displayed on the remote display device 200. The image extraction unit 223 extracts (trimmes) the captured image 310 within the set extraction range from the captured image 300.

[0081] The display unit 112 of the information display device 11 in the captured image 300 may be displayed on the remote display device 200 in a shape different from its actual shape, depending on the mounting position and orientation of the second camera 24 relative to the display unit 112 of the information display device 11. Therefore, the image extraction unit 223 performs image correction processing on the extracted captured image 310 so that the display unit 112 of the information display device 11 is displayed on the remote display device 200 in the correct shape. For example, the image extraction unit 223 performs image correction processing so that the extracted captured image 310 becomes rectangular.

[0082] The display unit 112 of the information display device 11 shown on the captured image 300 may be tilted due to rotation, for example, if the second camera 24 is tilted. Also, if the distance between the second camera 24 and the information display device 11 is large, the display unit 112 of the information display device 11 shown on the captured image 300 may appear small. Therefore, image correction processing is performed to resolve these issues.

[0083] Figure 7 shows an example of a screen for setting the extraction range of the captured image 300 displayed on the remote display device 200. The remote display device 200 displays the captured image 300 captured by the second camera 24. The display unit 112 of the information display device 11 in the captured image 300 is displayed on the remote display device 200 in a shape different from its actual shape. In the example shown in Figure 7, positions P1 corresponding to the upper left, P2 corresponding to the upper right, P3 corresponding to the lower right, and P4 corresponding to the lower left are specified among the four corners of the display unit 112 of the information display device 11 in the captured image 300. In the example shown in Figure 7, positions P1, P2, P3, and P4 are indicated by circles centered on the specified positions.

[0084] The designation of the positions corresponding to the four corners may be done manually by the operator via the remote control device 201, or it may be done automatically by the image extraction unit 223.

[0085] This section describes an example of how an operator can manually specify the positions corresponding to the four corners. The operator selects a menu item for specifying the positions corresponding to the four corners from a menu screen displayed on the remote display device 200. The remote display device 200 displays the captured image 300 captured by the second camera 24, and the default positions of the four corners are displayed as circular icon images. The default positions of the four corners are any four positions if the previous positions of the four corners are not stored in the memory unit, such as during initial setup. If the previous positions of the four corners are stored, the default positions of the four corners are the positions of the four corners from that time. If the icon images representing the positions P1, P2, P3, and P4 of the four corners displayed on the captured image 300 on the remote display device 200 are misaligned with the positions of the four corners on the display unit 112 of the information display device 11 in the captured image 300, the operator moves the icon images to the desired positions via the remote control device 201. Once the operator has finished specifying the positions corresponding to the four corners, they perform an operation to indicate completion (setting) from a menu screen displayed on the remote display device 200.

[0086] Figure 8 shows an example of the extracted captured image 310 displayed on the remote display device. In Figure 8, the captured image 310, which has been extracted from the captured image 300 within an extraction range defined by the positions corresponding to the four specified corners and has undergone image correction processing, is displayed on the remote display device 200.

[0087] When an operator manually specifies the positions corresponding to the four corners, the area in which the positions corresponding to the four corners can be specified may be restricted so that the quadrilateral formed by the specified positions does not become concave. The image extraction unit 223 restricts the area in which the positions corresponding to the four corners can be specified when the operator specifies the positions corresponding to the four corners.

[0088] Figure 9 is a schematic diagram illustrating the area in which positions corresponding to the four corners can be specified. As an example, the area in which position P1, corresponding to the upper left corner of the display unit 112 of the information display device 11, can be specified will be described. Position P1 can be specified within the area that satisfies the following conditions (1) to (4). (1) Located to the right of the straight line L1 that runs from the upper right corner P2 to the lower right corner P3 (2) Located to the right of the straight line L2 that runs from the upper right corner P2 to the lower left corner P4 (3) Located to the right of the straight line L3 that runs from the lower right corner P3 to the lower left corner P4 (4) Located at a certain distance from straight lines L1, L2 and L3

[0089] This section describes how to automatically specify the positions corresponding to the four corners. In this case, markers such as AR (Augmented Reality) markers are pre-placed at the four corners of the display unit 112 of the information display device 11. The image extraction unit 223 recognizes the markers from the captured image 300 and uses them as the positions of the four corners.

[0090] The operation signal generation unit 224 generates operation signals for operating the information display device 11 located in the driver's cab 7 of the work machine 1, based on input operations on the image displayed on the remote display device 200. In this embodiment, the operation signal generation unit 224 generates operation signals for operating the in-vehicle information terminal from a remote location, based on operations on the extracted captured image 310 displayed on the remote display device 200 via the remote control device 201, as the image displayed on the remote display device 200. Specifically, when the extracted captured image 310 is displayed on the remote display device 200, the operation signal generation unit 224 detects, for example, an operation to press the mouse, an operation to move the mouse, or an operation to release the mouse using the remote input device 202, and acquires the input position where the operation was performed and the type of operation. The operation signal generation unit 224 uses the coordinate transformation unit 225 to convert the input position of the operation to the remote display device 200 into coordinates on the display unit 112 of the information display device 11. The operation signal generation unit 224 converts the operation type performed on the remote input device 202 to the operation type for the in-vehicle information terminal based on the operation type information stored in the storage unit 221, using the operation type conversion unit 226. The operation signal generation unit 224 generates an operation signal that includes information indicating the input position of the operation to the remote display device 200 converted by the coordinate transformation unit 225 and the operation type for the in-vehicle information terminal converted by the operation type conversion unit 226. The operation signal generation unit 224 transmits the generated operation signal to the in-vehicle information terminal.

[0091] The operation signal includes information indicating the input position of the operation to the remote display device 200, which has been converted by the coordinate transformation unit 225, and the type of operation to the in-vehicle information terminal, which has been converted by the operation type transformation unit 226.

[0092] The operation signal generation unit 224 may generate operation signals with a modified transmission mode depending on the processing load of the remote controller 220 or the communication environment between the work machine 1 and the remote controller 220. If the communication environment is good, the operation signal generation unit 224 generates operation signals corresponding to the normal transmission mode. If the processing load of the work machine 1 is high, or if the communication environment is poor, such as in a mine, the operation signal generation unit 224 generates operation signals corresponding to the batch transmission mode.

[0093] The control signals corresponding to the normal transmission mode are, for example, the control signals corresponding to the operation of pressing the mouse, the operation of moving the mouse, and the operation of releasing the mouse.

[0094] An operation signal corresponding to the batch transmission mode is, for example, a mouse click operation signal that combines the operation of pressing the mouse and the operation of releasing the mouse. An operation signal corresponding to pressing the left click button, a so-called short press operation, is, for example, an operation signal where the period from pressing the mouse to releasing the mouse is 50 milliseconds. An operation signal corresponding to pressing the right click button, a so-called long press operation, is, for example, an operation signal where the period from pressing the mouse to releasing the mouse is 500 milliseconds.

[0095] A time lag occurs between the transmission of the generated operation signal to the in-vehicle information terminal and the execution of the operation based on that signal in the work machine 1. To reduce the discomfort felt by the operator due to this time lag, the operation signal generation unit 224 may output operation effects in response to the operation input to the remote display device 200. For example, the operation signal generation unit 224 may display a ripple image centered on the operation input position to the remote display device 200, display a busy cursor or loading icon while the operation based on the operation signal is being executed, or output sound effects corresponding to the operation from speakers placed around the remote display device 200.

[0096] The coordinate transformation unit 225 converts the input position of the operation to the remote display device 200 into coordinates on the display unit 112 of the information display device 11. The coordinate transformation unit 225 converts the input position of the operation to the captured image 310 displayed on the remote display device 200 into coordinates on the display unit 112 of the information display device 11.

[0097] As described above, the display unit 112 of the information display device 11 in the captured image 300 may be displayed on the remote display device 200 in a shape different from the actual shape, depending on the mounting position and orientation of the second camera 24 relative to the display unit 112 of the information display device 11. The coordinate transformation unit 225 converts the input position of the operation performed on the captured image 310 after image correction processing has been performed by the image extraction unit 223 into coordinates on the display unit 112 of the information display device 11.

[0098] The operation type conversion unit 226 converts the operation type of the remote input device 202 to the remote display device 200 to the operation type of the in-vehicle information terminal based on the operation type information stored in the storage unit 221.

[0099] Figure 6 is a block diagram showing a computer system 1000 according to the first embodiment. The machine controller 14, controller 110, and remote controller 220 each include the computer system 1000. The computer system 1000 has a processor 1001 such as a CPU (Central Processing Unit), a main memory 1002 including non-volatile memory such as ROM (Read Only Memory) and volatile memory such as RAM (Random Access Memory), a storage 1003, and an interface 1004 including input / output circuits. The functions of the machine controller 14, controller 110, and remote controller 220 are stored in the storage 1003 as computer programs. The processor 1001 reads the computer program from the storage 1003, loads it into the main memory 1002, and executes processing according to the computer program. The computer program may be distributed to the computer system 1000 via a network.

[0100] [Display method] Figure 10 is a flowchart showing the operation of the remote controller 220 according to the first embodiment.

[0101] The remote controller 220 sets the four corners of the display unit 112 of the information display device 11 on the captured image 300 (step S101). The remote controller 220 displays the captured image 300 captured by the second camera 24 on the remote display device 200 using the display control unit 222. The remote controller 220 displays the default positions of the four corners superimposed on the captured image 300 using circular icon images or the like using the display control unit 222. The operator drags the respective circular icon images displayed at the four corner positions P1, P2, P3, and P4 to the desired positions and selects a menu indicating completion (setting). Proceed to step S102.

[0102] The remote controller 220 crops (extracts) the captured image 300 and performs image correction processing on the extracted captured image 310 (step S102). The remote controller 220 uses the image extraction unit 223 to set an image extraction range from positions corresponding to the specified four corners and extracts the captured image 310 from the captured image 300 within the extraction range. The remote controller 220 also uses the image extraction unit 223 to perform image correction processing on the extracted captured image 310. Proceed to step S103.

[0103] The remote controller 220 displays the extracted captured image 310, after image correction processing has been performed, on the remote display device 200 (step S103). The remote controller 220 displays the extracted captured image 310, after image correction processing has been performed, on the remote display device 200 via the display control unit 222. Proceed to step S104.

[0104] The remote controller 220 generates an operation signal (step S104). The remote controller 220, using the operation signal generation unit 224, generates an operation signal for remotely operating the in-vehicle information terminal based on the operation on the captured image 310 displayed on the remote display device 200. Proceed to step S105.

[0105] The remote controller 220 transforms the coordinates (step S105). The remote controller 220 uses the coordinate transformation unit 225 to convert the input position of the operation to the remote display device 200 into coordinates on the display unit 112 of the information display device 11. Proceed to step S106.

[0106] The remote controller 220 converts the operation type (step S106). The remote controller 220, using the operation type conversion unit 226, converts the operation type from the remote input device 202 to the remote display device 200 to the operation type for the in-vehicle information terminal, based on the operation type information stored in the storage unit 221. Proceed to step S107.

[0107] The remote controller 220 transmits an operation signal (step S107). The remote controller 220 transmits the operation signal generated by the operation signal generation unit 224 to the in-vehicle information terminal.

[0108] The process from step S101 to step S102 may be displayed, for example, when the work machine 1 is started, or when a menu for specifying the positions corresponding to the four corners is selected.

[0109] [effect] As described above, in this embodiment, the display unit 112 of the information display device 11 located in the driver's cab 7 is imaged by the second camera 24, and the imaged data is transmitted to the remote display device 200 in the remote control room. In this embodiment, an imaged image 310 corresponding to the display unit 112 of the information display device 11 is extracted from the imaged image 300 captured by the second camera 24, and image correction processing is performed to display it on the remote display device 200. In this embodiment, an operation signal for operating the in-vehicle information terminal from a remote location is generated based on the operation performed on the imaged image 310 displayed on the remote display device 200, and the generated operation signal can be transmitted to the in-vehicle information terminal. In this embodiment, the in-vehicle information terminal located in the driver's cab 7 of the ride-on work machine 1 can be appropriately operated from a remote location according to the content displayed on the in-vehicle information terminal.

[0110] In this embodiment, an image corresponding to the display unit 112 of the information display device 11 is extracted from the captured image 300 displayed on the remote display device 200. In this embodiment, an operation signal for remotely operating the in-vehicle information terminal can be generated based on the operation on the extracted image.

[0111] In this embodiment, the extraction range of the captured image 300 can be set by specifying positions corresponding to the four corners of the display unit 112 of the information display device 11 on the captured image 300 displayed on the remote display device 200.

[0112] In this embodiment, the input position for operations on the remote display device 200 can be converted into coordinates on the display unit 112 of the information display device 11.

[0113] In this embodiment, the operation type of the remote input device 202 to the remote display device 200 can be converted to the operation type to the in-vehicle information terminal.

[0114] In this embodiment, the correspondence between the operation type performed by the remote input device 202 to the remote display device 200 and the operation type performed to the in-vehicle information terminal can be set and stored.

[0115] In this embodiment, the transmission mode of the operation signal can be changed depending on the processing load of the remote controller 220 or the communication environment between the work machine 1 and the remote controller 220.

[0116] In this embodiment, when the operator specifies the positions corresponding to the four corners, the area that can be specified as the positions corresponding to the four corners can be restricted so that the quadrilateral formed by the specified positions does not become concave. According to this embodiment, it is easier to specify appropriate positions when specifying the extraction range.

[0117] <Second Embodiment> The second embodiment will be described using Figures 11 to 14. Figure 11 is a diagram showing an example of an image displayed on an information display device. Figure 12 is a diagram showing an example of an operation icon image displayed on a remote display device. Figure 13 is a block diagram of the remote controller in the remote control room according to the second embodiment. Figure 14 is a flowchart showing the operation of the remote controller according to the second embodiment. The basic configuration of the remote control system 100 is the same as in the first embodiment.

[0118] In this embodiment, the remote display device 200 displays an operation icon image 360 ​​(see Figure 12) that corresponds to the operation icon image 350 (see Figure 11) displayed on the display unit 112 of the information display device 11 of the work machine 1. The operator in the remote control room operates the work machine 1 by performing operations on the operation icon image 360 ​​displayed on the remote display device 200.

[0119] The remote controller 220 includes a storage unit 221, a display control unit 222, an operation signal generation unit 224, a coordinate transformation unit 225, and an operation type transformation unit 226.

[0120] The memory unit 221 stores an operation icon image 360 ​​that corresponds to the operation icon image 350 displayed on the display unit 112 of the information display device 11 of the work machine 1. The operation icon image 350 is an image that simulates the part of the image displayed on the display unit 112 of the information display device 11 that can be touch-operated by an operator sitting in the driver's seat 7 of the work machine 1, such as a toolbar with multiple operation icons.

[0121] The display control unit 222 displays an image on the remote display device 200 that corresponds to the screen displayed on the display unit 112 of the information display device 11 located in the operator's cab 7 of the work machine 1. In this embodiment, the display control unit 222 displays an operation icon image 360 ​​(see Figure 12) stored in the storage unit 221 on the remote display device 200 as an image corresponding to the screen displayed on the display unit 112 of the information display device 11.

[0122] The operation signal generation unit 224 generates operation signals for operating the information display device 11 located in the driver's cab 7 of the work machine 1, based on input operations on the image displayed on the remote display device 200. In this embodiment, the operation signal generation unit 224 generates operation signals for operating the in-vehicle information terminal from a remote location, based on operations on the operation icon image 360 ​​displayed on the remote display device 200 via the remote control device 201, as the image displayed on the remote display device 200. Specifically, when the operation icon image 360 ​​is displayed on the remote display device 200, the operation signal generation unit 224 detects, for example, an operation to press the mouse, an operation to move the mouse, or an operation to release the mouse using the remote input device 202, and acquires the input position where the operation was performed and the type of operation.

[0123] [Display method] The operation of the remote controller 220 according to the second embodiment will be explained using Figure 14.

[0124] The remote controller 220 displays the operation icon image 360 ​​on the remote display device 200 (step S201). The remote controller 220 displays the operation icon image 360 ​​stored in the storage unit 221 on the remote display device 200 via the display control unit 222. Proceed to step S202.

[0125] The remote controller 220 generates an operation signal (step S202). The remote controller 220 generates an operation signal for remotely operating the in-vehicle information terminal based on the operation of the operation icon image 360 ​​displayed on the remote display device 200, using the operation signal generation unit 224. Proceed to step S203.

[0126] The processing in steps S203 to S205 is the same as the processing in steps S105 to S107 of the flowchart shown in Figure 10.

[0127] [effect] As described above, in this embodiment, an operation icon image 360, which simulates a part of the image displayed on the display unit 112 of the information display device 11 that can be touch-operated by an operator seated in the driver's seat 7 of the work machine 1, such as a toolbar with multiple operation icons, is displayed on the remote display device 200. In this embodiment, an operation signal for remotely operating an in-vehicle information terminal can be generated based on an operation performed on the operation icon image 360 ​​displayed on the remote display device 200, and the generated operation signal can be transmitted to the in-vehicle information terminal. In this embodiment, the in-vehicle information terminal, which is installed in the driver's seat 7 of the ride-on work machine 1, can be appropriately operated remotely according to the content displayed on the in-vehicle information terminal.

[0128] In the above embodiment, the work machine 1 is assumed to be a hydraulic excavator. However, the work machine 1 can be any machine that has a work implement. For example, the work machine 1 could be a wheel loader or a bulldozer. Alternatively, the work machine 1 could be a work vehicle that does not have a work implement, such as a dump truck. [Explanation of Symbols]

[0129] 1...Working machine, 2...Main body, 3...Traction unit, 3A...Tracks, 4...Working machine, 4A...Boom, 4B...Arm, 4C...Bucket, 5...Working machine cylinder, 5A...Boom cylinder, 5B...Arm cylinder, 5C...Bucket cylinder, 6...Power source, 7...Driver's cab, 9...Control device, 9A...Left control lever, 9B...Right control lever, 9C...Left travel lever, 9D...Right travel lever, 9E...Left foot pedal, 9F...Right foot pedal, 11...Information display device (on-board information terminal), 13...Seat, 14...Machine controller, 20...Position sensor, 21...Attitude sensor, 22...Angle sensor, 22A...Boom angle sensor, 22B...Arm angle sensor, 22C...Bucket angle sensor, 23...First camera Ra, 24...Second camera (camera), 25...Communication device, 100...Remote control system, 110...Controller, 111...Input device, 112...Display unit, 120...Display control unit, 141...Machine control unit, 142...Peripheral monitoring control unit, 143...Guidance control unit, 144...Measurement unit, 200...Remote display device, 201...Remote control device, 202...Remote input device, 220...Remote controller, 221...Storage unit, 222...Display control unit, 223...Image extraction unit, 224...Operation signal generation unit, 225...Coordinate transformation unit, 226...Operation type transformation unit, 1000...Computer system, 1001...Processor, 1002...Main memory, 1003...Storage, 1004...Interface.

Claims

1. A remote control system for operating machinery from a remote location, A remote display device is placed at a location away from the aforementioned work machine, A remote input device that receives input operations for the remote display device, One or more processors, Equipped with, The aforementioned processor, The remote display device displays an image corresponding to the screen displayed on the display unit of the in-vehicle information terminal located in the driver's cabin of the aforementioned work machine. Based on the input operation on the image displayed on the remote display device, an operation signal for operating the in-vehicle information terminal is generated. The generated operation signal is output. Remote control system.

2. The aforementioned image is an image displayed on the display unit of the in-vehicle information terminal. The remote control system according to claim 1.

3. The aforementioned image is an image captured by a camera, including the display unit of the in-vehicle information terminal. The aforementioned processor, The captured image received from the work machine is displayed on the remote display device. From the captured image displayed on the remote display device, an image corresponding to the extraction range of the display unit of the in-vehicle information terminal is extracted, Based on the operations performed on the extracted image, an operation signal for operating the in-vehicle information terminal is generated. The remote control system according to claim 1 or 2.

4. The aforementioned processor, The extraction range of the captured image is set by specifying the positions corresponding to the four corners of the display unit of the in-vehicle information terminal on the captured image displayed on the remote display device. The remote control system according to claim 3.

5. The aforementioned processor, The input position for the operation on the remote display device is converted into coordinates on the display unit of the in-vehicle information terminal. The remote control system according to claim 1.

6. The aforementioned processor, The operation type performed by the remote input device on the remote display device is converted to the operation type on the in-vehicle information terminal. The remote control system according to claim 1.

7. The aforementioned processor, The system sets and stores the correspondence between the type of operation performed by the remote input device on the remote display device and the type of operation performed on the in-vehicle information terminal. The remote control system according to claim 1.

8. The aforementioned processor, To prevent the quadrilateral formed by the four specified corners from being concave, the area that can be specified as the position corresponding to the four corners is restricted. The remote control system according to claim 4.

9. The aforementioned processor, The transmission mode of the operation signal is changed depending on the processing load of the processor or the communication environment between the work machine and the processor. The remote control system according to claim 1.

10. A remote control system for operating machinery from a remote location, An in-vehicle information terminal is installed in the operator's cabin of the work machine and displays information including the operation information of the work machine, A camera for capturing images of the display unit of the in-vehicle information terminal, A communication device that transmits the captured image taken by the aforementioned camera to a remote location, The remote display device located at the remote location, A remote input device that receives input operations for the remote display device, One or more processors, Equipped with, The in-vehicle information terminal displays a screen related to application software executed to assist the work machine in performing tasks, The aforementioned processor, The captured image is displayed on the remote display device, Based on the operation on the captured image displayed on the remote display device, an operation signal for operating the in-vehicle information terminal is generated. The generated operation signal is transmitted to the in-vehicle information terminal. Remote control system.

11. A remote control method for operating a work machine from a remote location, The remote display device, located away from the aforementioned work machine, displays an image corresponding to the screen displayed on the display unit of the in-vehicle information terminal located in the driver's cabin of the aforementioned work machine. Based on input operations on the image displayed on the remote display device, the system generates an operation signal for operating the in-vehicle information terminal located in the driver's cabin of the work machine. Outputting the generated operation signal, A remote control method, including

12. The aforementioned image is an image displayed on the display unit of the in-vehicle information terminal. The remote control method according to claim 11.

13. The aforementioned image is an image captured by a camera, including the display unit of the in-vehicle information terminal. The captured image received from the aforementioned work machine is displayed on the remote display device, The remote control method according to claim 12, which includes the following:

Images