Operation device and operation system

Abstract

To provide an operation device and an operation system that enable a skill acquired in a virtual space to be utilized on a real work machine.SOLUTION: An operation device is capable of operating a work machine 120 in a virtual space where the work of the work machine 20 is simulated, using at least some of input devices.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an operating device capable of remotely operating a work machine, and an operating system including the operating device. [Background technology]

[0002] Patent Document 1 discloses a remote control device capable of remotely controlling a work machine. When the work machine to be remotely controlled is connected to the remote control device, for example, the wipers are operated. This makes it possible to check the connection status with the work machine to be remotely controlled. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-152338 A Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, there is an operation device that operates a work machine in a virtual space where work by the work machine is simulated. However, if the operation specifications of the operation device that operates the work machine in the virtual space and the operation device that remotely operates the real work machine are different, the user cannot remotely operate the real work machine with the same operations as those used to operate the work machine in the virtual space. In this case, the user cannot utilize the skills acquired in the virtual space on the real work machine.

[0005] An object of the present invention is to provide an operating device and an operating system that enable skills acquired in a virtual space to be utilized on a real work machine. [Means for solving the problem]

[0006] The operating device of the present invention is an operating device capable of remotely operating a work machine in response to operation by a user via an input device, and is characterized in that in a virtual space in which work by the work machine is simulated, it is possible to operate the work machine in the virtual space using at least a part of the input device.

[0007] In addition, the operation system of the present invention has a work machine, an operation device capable of remotely operating the work machine, and an operation verification device that simulates work of the work machine in a virtual space, and is characterized in that the operation device is capable of operating the work machine in the virtual space. Effect of the Invention

[0008] According to the present invention, in a virtual space in which work by the work machine is simulated, it is possible to operate a work machine in the virtual space using at least a part of the input device. In other words, the operation device can not only remotely operate a real work machine, but also operate a work machine in the virtual space. Thus, a user can operate a work machine in the virtual space with the same operation or substantially the same operation as that for remotely operating a real work machine. Thus, for example, a user can operate a work machine in the virtual space to learn the procedure for remotely operating a real work machine and the method for remotely operating a real work machine. Then, after operating the work machine in the virtual space to practice a predetermined work, the user can actually perform the predetermined work by remotely operating the real work machine. This allows the user to utilize the skills acquired in the virtual space in the real work machine. [Brief description of the drawings]

[0009] [Figure 1] 1 is an overall schematic diagram of an operation system. [Diagram 2] FIG. 2 is a perspective view of an input device of the operating device. [Diagram 3] FIG. 2 is a front view of an output device of the operation device. [Figure 4] FIG. 2 is a functional configuration diagram of the operation system. [Diagram 5] 1 is a flowchart of a sequence. [Figure 6] 13 is a flowchart of a pre-operation process. [Figure 7] 4 is a flowchart of a start preparation process. [Figure 8] 13 is a flowchart of an authentication result confirmation process. [Figure 9] 4 is a flowchart of a starting process. [Figure 10] 13 is a flowchart of an end process. [Figure 11] 13 is a flowchart of an action selection process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[0011] (Operation system configuration) The operation system according to this embodiment is equipped with an operation device according to this embodiment. As shown in Fig. 1, which is an overall schematic diagram of the operation system 100, the operation device 1 is capable of remotely operating an actual work machine 20 that exists at a work site or the like. Note that, although Fig. 1 shows only one actual work machine 20, the operation device 1 is capable of switching between and remotely operating a plurality of work machines 20.

[0012] Furthermore, the operation device 1 is capable of operating a work machine 120 in the virtual space. The work of the work machine 120 in the virtual space is simulated by a simulator (operation verification device) 90. Multiple types of work machines 120 are placed in the virtual space, and these may be selectively operated.

[0013] (Work machine configuration) The work machine 20 is a machine that performs work using an attachment 30, such as a hydraulic excavator. The work machine 120 in the virtual space is a simulation of the actual work machine 20, and therefore has the same structure as the actual work machine 20. The work machine 20 has a machine body 24 equipped with a lower traveling body 21 and an upper rotating body 22, the attachment 30, and a cylinder 40.

[0014] The lower traveling structure 21 is a portion that causes the work machine 20 to travel, and is equipped with, for example, crawlers. The upper rotating structure 22 is attached to the upper part of the lower traveling structure 21 so as to be able to rotate via a rotating device 25. A cab (operator's compartment) 23 is provided at the front part of the upper rotating structure 22.

[0015] The cab 23 may be a vertical elevator cab or a link-type elevator cab that is moved up and down by a vertical movement mechanism.

[0016] A driver's seat and various machine-side input devices 83 (see FIG. 4) are provided in the cab 23. Since the input device 2 (see FIG. 4) having the same configuration as the machine-side input device 83 is provided in the operation device 1, a detailed description of the machine-side input device 83 will be omitted.

[0017] A wiper 26 (see FIG. 4) is provided inside the cab 23. The wiper 26 wipes off water droplets on the front windshield.

[0018] The attachment 30 is attached to the upper rotating body 22 so as to be rotatable in the vertical direction. The attachment 30 includes a boom 31, an arm 32, and a bucket 33. The boom 31 is attached to the upper rotating body 22 so as to be rotatable (raised and lowered) in the vertical direction. The arm 32 is attached to the boom 31 so as to be rotatable in the vertical direction. The bucket 33 is a tip attachment that is the tip of the attachment 30, and is attached to the arm 32 so as to be rotatable in the front-to-rear direction of the upper rotating body 22. The bucket 33 is a part that performs tasks such as excavating, leveling, and scooping soil and sand.

[0019] The end attachment is not limited to the bucket 33, but may be a lifting magnet for holding scrap iron or a nibbler for crushing objects.

[0020] The cylinder 40 is capable of hydraulically rotating the attachment 30. The cylinder 40 is a hydraulic telescopic cylinder. Note that the cylinder 40 may be an electric cylinder. The cylinder 40 includes a boom cylinder 41, an arm cylinder 42, and a bucket cylinder 43.

[0021] The boom cylinder 41 rotates the boom 31 relative to the upper rotating body 22. A base end of the boom cylinder 41 is rotatably attached to the upper rotating body 22. A tip end of the boom cylinder 41 is rotatably attached to the boom 31.

[0022] The arm cylinder 42 rotates the arm 32 relative to the boom 31. A base end of the arm cylinder 42 is rotatably attached to the boom 31. A tip end of the arm cylinder 42 is rotatably attached to the arm 32.

[0023] The bucket cylinder 43 rotates the bucket 33 relative to the arm 32. A base end of the bucket cylinder 43 is rotatably attached to the arm 32. A tip end of the bucket cylinder 43 is rotatably attached to a link member 34 that is rotatably attached to the bucket 33.

[0024] The work machine 20 also has a sensor 50. The sensor 50 has a turning angle sensor 52, a tilt angle sensor 60, and a tilt angle sensor 55.

[0025] The slewing angle sensor 52 detects the slewing angle of the upper rotating body 22 relative to the lower traveling body 21. The slewing angle sensor 52 is, for example, an encoder, a resolver, or a gyro sensor. In this embodiment, the slewing angle of the upper rotating body 22 when the front of the upper rotating body 22 coincides with the front of the lower traveling body 21 is set to 0°.

[0026] The inclination angle sensor 60 detects the attitude of the attachment 30. The inclination angle sensor 60 includes a boom inclination angle sensor 61, an arm inclination angle sensor 62, and a bucket inclination angle sensor 63.

[0027] The boom inclination angle sensor 61 is attached to the boom 31 and detects the attitude of the boom 31. The boom inclination angle sensor 61 is a sensor that acquires the inclination angle of the boom 31 with respect to the horizontal line, and is, for example, an inclination (acceleration) sensor. Note that the boom inclination angle sensor 61 may be a rotation angle sensor that detects the rotation angle of the boom foot pin (boom base end) or a stroke sensor that detects the stroke amount of the boom cylinder 41.

[0028] The arm inclination angle sensor 62 is attached to the arm 32 and detects the posture of the arm 32. The arm inclination angle sensor 62 is a sensor that obtains the inclination angle of the arm 32 with respect to the horizontal line, and is, for example, an inclination (acceleration) sensor. Note that the arm inclination angle sensor 62 may be a rotation angle sensor that detects the rotation angle of the arm connecting pin (base end of the arm) or a stroke sensor that detects the stroke amount of the arm cylinder 42.

[0029] The bucket inclination angle sensor 63 is attached to the link member 34 and detects the attitude of the bucket 33. The bucket inclination angle sensor 63 is a sensor that obtains the inclination angle of the bucket 33 with respect to the horizontal line, and is, for example, an inclination (acceleration) sensor. Note that the bucket inclination angle sensor 63 may be a rotation angle sensor that detects the rotation angle of the bucket connecting pin (bucket base end) or a stroke sensor that detects the stroke amount of the bucket cylinder 43.

[0030] The inclination angle sensor 55 is provided on the upper rotating body 22 and detects the inclination angle of the upper rotating body 22 with respect to the horizontal plane. The inclination angle sensor 55 is, for example, an inclination (acceleration) sensor or an IMU (Inertial Measurement Unit).

[0031] The work machine 20 also has an imaging device 27 (see FIG. 4). The imaging device 27 includes a camera installed in the cab 23, a camera mounted on the attachment 30, a rear camera mounted on the upper rotating body 22, and a surrounding camera installed on the machine body 24. Note that in the virtual space, the imaging device 27 can be freely placed in any position. Also, multiple imaging devices 27 can be installed in the virtual space.

[0032] (Configuration of the operation device) As shown in FIG. 2, which is a perspective view of the input device 2 of the operating device 1, the operating device 1 includes the input device 2 and a driver's seat 4.

[0033] The input device 2 is operated by a user of the operating device 1. The input device 2 has an operation mechanism 6. The operation mechanism 6 has a pair of left and right travel levers 11 and a pair of left and right travel pedals 12. The travel levers 11 and the travel pedals 12 are disposed in front of the driver's seat 4. The travel levers 11 and the travel pedals 12 operate the travel direction and travel speed of the crawlers of the lower traveling body 21.

[0034] The operation mechanism 6 also has a pair of left and right operation levers 13. The operation levers 13 are disposed on the left and right sides of the driver's seat 4. The operation levers 13 are used to operate the attachment 30.

[0035] The operation mechanism 6 may have an optional operation lever. The optional operation lever is used when operating a special attachment 30 such as a nibbler.

[0036] The input device 2 also has a key switch 14. A key inserted into the key switch 14 is rotated between a key-on position, a key-start position, and a key-off position. When the key is in the key-on position, the electrical system is connected. When the key is in the key-start position, the engine is started. When the key is in the key-off position, the engine is stopped.

[0037] The input device 2 also has a camera 15 (see FIG. 3). The camera 15 captures an image of the face of a user sitting in the driver's seat 4, etc.

[0038] The input device 2 also has an information terminal 16. The information terminal 16 is disposed in front of the driver's seat 4. The information terminal 16 is a tablet, a smartphone, or the like. The information terminal 16 has a display and a touch panel. The information terminal 16 also serves as the output device 3. The output device 3 will be described later.

[0039] The input device 2 also has an emergency stop switch 18 (see FIG. 4) and various other switches. The emergency stop switch 18 is for emergency stopping the work machine 20. The various switches include a switch attached to the operation lever 13, a horn sounding switch, a switch for changing the camera direction, a lever lock switch, a work mode changeover switch, a wiper operation switch, a switch for turning the lights on and off, and the like.

[0040] As shown in FIG. 3, which is a front view of the output device 3 of the operation device 1, the operation device 1 has the output device 3. The output device 3 has seven monitors 17 (17a to 17g). The main monitor 17a displays an image from a camera installed in the cab 23. The upper right monitor 17b and the upper left monitor 17c display an image from an overhead camera set up at the work site. The upper center monitor 17d displays an image from a camera mounted on the attachment 30 and an image from a rear camera mounted on the upper rotating body 22. The left monitor 17e displays an analysis report of the work situation and the situation at the work site. The right monitor 17f displays information on the work machine 20 side. The lower center monitor 17g displays an image from a surrounding camera installed on the machine body 24. The number and display contents of the monitors 17a to 17g are not limited to those described above.

[0041] Images captured by the imaging devices 27 arranged in the virtual space can be displayed on any of the monitors 17, regardless of which imaging device 27 is used. The monitor 17 that outputs the images of the imaging devices 27 can be specified, and one or more monitors 17 that output the images of the imaging devices 27 can be specified. In the simulator 90, one or more imaging devices 27 can be freely arranged in the virtual space. Therefore, by having the imaging device 27 capture images from a viewpoint or field of view that is difficult to confirm when operating a work machine and using this as a reference for the user, it is possible to promote mastery of operating the work machine.

[0042] The output device 3 also has a speaker 19 (see FIG. 4). The speaker 19 outputs sound picked up by a microphone mounted on the work machine 20.

[0043] The output device 3 also has a seat tilt device 7 (see FIG. 4). The seat tilt device 7 is a device that changes the tilt angle of the driver's seat 4 with respect to the horizontal plane. The seat tilt device 7 reproduces the tilt of the upper rotating body 22 detected by the tilt angle sensor 55 in the driver's seat 4.

[0044] The output device 3 also has various display devices (not shown). The contents displayed by these display devices include the illumination status of the PATLITE (registered trademark), the lever lock status, the fuel level, the fuel consumption amount, the engine water temperature, the urea tank level, the soot accumulation information of the DPF, the operation time, etc.

[0045] (Simulator configuration) Returning to FIG. 1, the simulator 90 is composed of a personal computer or the like. The simulator 90 simulates work of the work machine 120 in a virtual space by executing simulation software. In this embodiment, the simulator 90 is separate from the operation device 1. The simulator 90 and the operation device 1 may be integrated together. In this case, the operation device-side control device 71 and the operation device-side storage device 72 (see FIG. 4) of the operation device 1 execute the simulation software to simulate work of the work machine 120 in the virtual space.

[0046] Note that the server 110 (see FIG. 4), which will be described later, may execute simulation software to simulate the work of the work machine 120 in a virtual space.

[0047] (Functional configuration of the operating system) As shown in FIG. 4, which is a functional configuration diagram of the operation system 100, the operation device 1 has an operation device side control device 71, an operation device side storage device 72, and an operation device side communication device 73.

[0048] The operating device side control device 71 is composed of a drive control element 71a and a machine state detection unit 71b. The drive control element 71a controls the operation of the connected real work machine 20 or the connected work machine 120 in the virtual space. The machine state detection unit 71b detects the state of the connected real work machine 20 or the connected work machine 120 in the virtual space.

[0049] Various information is input to the operation device-side control device 71 from the input device 2, such as the operation mechanism 6, the key switch 14, the emergency stop switch 18, the camera 15, and the information terminal 16. The operation device-side communication device 73 transmits various information from the input device 2 to the outside. The operation device-side communication device 73 also receives various information from the outside. The operation device-side communication device 73 includes a video decoder and an audio decoder. The operation device-side control device 71 outputs various information from the outside to the output device 3, such as the information terminal 16, the speaker 19, the monitor 17, and the seat tilt device 7.

[0050] The operation device side communication device 73 is capable of communicating with the actual work machine 20 and the server 110.

[0051] A user of the operation device 1 can select a real work machine 20 as a connection destination for the operation device 1. An information terminal 16 (see FIG. 2) is used to select a connection destination. When the real work machine 20 is selected as a connection destination for the operation device 1, communication using an identifier is established between the operation device 1 and the real work machine 20. This enables the operation device 1 to remotely operate the real work machine 20 using the input device 2 and the output device 3. The identifier will be described later.

[0052] When the operation device 1 is remotely operating the actual work machine 20, the machine state detection unit 71b causes the image captured by the camera on the work machine 20 side to be displayed on the monitor 17 of the output device 3, and also causes the sound picked up by the microphone 28 on the work machine 20 side to be output from the speaker 19 of the output device 3. In addition, the machine state detection unit 71b causes various types of information from the actual work machine 20 to be output to the output device 3.

[0053] The work machine 20 has a machine-side control device 81 , a machine-side memory device 82 , a machine-side input device 83 , and a machine-side communication device 84 .

[0054] The machine side control device 81 is composed of a drive unit control element 81a and a machine state detection unit 81b. The drive unit control element 81a controls the operation of the actual work machine 20. The machine state detection unit 81b detects the state of the actual work machine 20.

[0055] The information detected by the sensor 50 is input to the machine condition detection unit 81b. Specifically, information on the rotation angle (posture) of the upper rotating body 22 with respect to the lower traveling body 21 detected by the rotation angle sensor 52 is input to the machine condition detection unit 81b. Furthermore, information on the posture of the boom 31 detected by the boom inclination angle sensor 61 is input to the machine condition detection unit 81b. Furthermore, information on the posture of the arm 32 detected by the arm inclination angle sensor 62 is input to the machine condition detection unit 81b. Furthermore, information on the posture of the bucket 33 detected by the bucket inclination angle sensor 63 is input to the machine condition detection unit 81b. Furthermore, information on the inclination of the upper rotating body 22 detected by the inclination angle sensor 55 is input to the machine condition detection unit 81b.

[0056] The drive control element 81 a operates the attachment 30 and the turning device 25 based on the information detected by the sensor 50 and various information from the machine-side input device 83 .

[0057] The machine-side communication device 84 transmits information detected by the sensor 50 to the outside. In addition, the machine-side communication device 84 receives various information from the outside. The machine-side communication device 84 includes a video encoder and an audio encoder. When the work machine 20 is remotely operated, the drive control element 81a operates the attachment 30 and the slewing device 25 based on various information from the outside.

[0058] When the operation device 1 remotely operates the actual work machine 20, a predetermined sequence is used. This sequence is a computer program, and is executed while the operation device side control device 71 of the operation device 1 and the machine side control device 81 of the work machine 20 communicate with each other. By using this sequence between the operation device 1 and the work machine 20, the actual work machine 20 is operated by the operation device 1. Details of the sequence will be described later.

[0059] Here, the machine state detection unit 71b of the operation device 1 transmits a specific signal related to a device equipped in the work machine 20 to the work machine 20 at a predetermined timing. The device equipped in the work machine 20 is, for example, the engine 29. The specific signal is, for example, a signal for checking the operating state of the engine 29. When the machine state detection unit 81b of the work machine 20 receives this signal from the operation device 1, it transmits a response signal according to the state of the device to the operation device 1. The response signal transmitted from the work machine 20 to the operation device 1 is, for example, a response signal indicating that the engine 29 has started or a response signal indicating that the operating state of the engine 29 is normal. For example, when the operating state of the engine 29 is normal, the machine state detection unit 81b of the work machine 20 transmits a response signal indicating that the operating state of the engine 29 is normal to the operation device 1. On the other hand, when the operating state of the engine 29 is abnormal, the machine state detection unit 81b of the work machine 20 transmits a response signal indicating that the operating state of the engine 29 is abnormal to the operation device. In this way, by responding to a specific signal from the operation device 1 with a response signal, the operating status of the operation system 100 can be confirmed.

[0060] The operation device side communication device 73 of the operation device 1 is capable of communicating with the simulator 90.

[0061] A user of the operation device 1 can select the simulator 90 as a connection destination of the operation device 1. That is, the user can select between remote operation of the real work machine 20 and operation of a work machine 120 in a virtual space. When the simulator 90 is selected as a connection destination of the operation device 1, communication using an identifier is established between the operation device 1 and the simulator 90. This enables the operation device 1 to operate the work machine 120 (see FIG. 1) in the virtual space using at least a part of the input device 2 and the output device 3. In this embodiment, all of the input device 2 and the output device 3 are used to operate the work machine 120 in the virtual space, but a configuration in which only a part of the input device 2 or only a part of the output device 3 is used may also be used.

[0062] The simulator 90 has a simulator-side control device 91 , a simulator-side storage device 92 , a simulator-side communication device 93 , and a machine state simulation unit 94 .

[0063] The simulator-side control device 91 is composed of a drive control element 91a and a machine state detection unit 91b. The drive control element 91a simulates the work of the work machine 120 in a virtual space. The machine state detection unit 91b detects the state of the work machine 120 in the virtual space.

[0064] The simulator-side communication device 93 transmits information about the work machine 120 simulated by the drive control element 91a to the outside. The simulator-side communication device 93 also receives various types of information from the outside. The simulator-side communication device 93 includes a video encoder and an audio encoder. The drive control element 91a operates the work machine 120 in the virtual space based on various types of information from the outside. When the operation device 1 is operating the work machine 120 in the virtual space, the drive control element 91a operates the work machine 120 in the virtual space based on various types of information from the operation device 1.

[0065] When the operation device 1 is operating the work machine 120 in the virtual space, the machine state detection unit 71b of the operation device 1 causes the image captured by the imaging device 27 arranged in the virtual space to be displayed on the monitor 17 of the output device 3, and also causes sound data generated in the virtual space to be output as sound from the speaker of the output device 3. In addition, the machine state detection unit 71b of the operation device 1 causes various information from the simulator 90 to be output to the output device 3.

[0066] When the operation device 1 operates the work machine 120 in the virtual space, the same sequence is used as when the operation device 1 remotely operates the real work machine 20. This sequence is performed while the operation device side control device 71 of the operation device 1 and the simulator side control device 91 of the simulator 90 communicate with each other. The work machine 120 in the virtual space is operated by the operation device 1 by using the same sequence between the operation device 1 and the simulator 90 as that used between the real work machine 20 and the operation device 1.

[0067] In this way, the operation device 1 can not only remotely operate the real work machine 20, but also operate the work machine 120 in the virtual space. Therefore, the user can operate the work machine 120 in the virtual space with the same operation or substantially the same operation as when remotely operating the real work machine 20. Therefore, for example, the user can operate the work machine 120 in the virtual space to learn the procedure for remotely operating the real work machine 20 and the method for remotely operating the real work machine 20. Then, after operating the work machine 120 in the virtual space to practice a predetermined task, the user can remotely operate the real work machine 20 to actually perform the predetermined task. This allows the user to utilize the skills acquired in the virtual space in the real work machine 20.

[0068] Furthermore, the user can select between remote operation of the work machine 20 and operation of the work machine 120 in the virtual space. Thus, the user can, for example, practice a predetermined work by selecting operation of the work machine 120 in the virtual space. Furthermore, the user can, for example, perform actual work by selecting remote operation of the real work machine 20.

[0069] Furthermore, the same sequence as that used between the work machine 20 and the operation device 1 is used between the operation device 1 and the simulator 90, so that the work machine 120 in the virtual space is operated by the operation device 1. The user can operate the work machine 120 in the virtual space by operating the same or substantially the same procedure (sequence) as when remotely operating the real work machine 20. Thus, for example, the user can operate the work machine 120 in the virtual space to learn the procedure for remotely operating the real work machine 20. Then, after operating the work machine 120 in the virtual space and practicing a predetermined procedure, the user can remotely operate the real work machine 20 to actually perform the predetermined procedure. This allows the user to utilize the skills acquired in the virtual space in the real work machine 20.

[0070] The server 110 includes a server-side control device 111 , a server-side storage device 112 , and a server-side communication device 113 .

[0071] The server-side storage device 112 stores an identifier. The identifier is, for example, an IP address, and is used to identify the operation device 1, the real work machine 20, and the work machine 120 in the virtual space on the network. An identifier is assigned to each of the devices provided in the operation device 1 (operation device-side communication device 73, monitor 17, speaker 19, etc.). An identifier is also assigned to each of the devices provided in the work machine (communication device, imaging device 27, microphone 28, etc.). Communication between the operation device 1 and the real work machine 20, or communication between the operation device 1 and the simulator 90 is established by using this identifier.

[0072] As described above, the operation device 1 is separate from the simulator 90 that simulates the work of the work machine 120 in the virtual space. When the operation device 1 and the simulator 90 are integrated, that is, when the work of the work machine 120 in the virtual space is simulated by the operation device side control device 71 and the operation device side storage device 72 (see FIG. 4) of the operation device 1, the sequence used when remotely operating the real work machine 20 and the sequence used when operating the work machine 120 in the virtual space are different. This is because, when the operation device 1 and the simulator 90 are separate, a communication connection state between the two is established by applying each other's identifier, whereas when the operation device 1 and the simulator 90 are integrated, such a communication connection state cannot be established. On the other hand, when the operation device 1 is separate from the simulator 90, a communication connection state is established by applying each other's identifier, so that, for example, the sequence used when remotely operating the real work machine 20 and the sequence used when operating the work machine 120 in the virtual space can be made the same. In this case, it is possible to operate the work machine 120 in the virtual space without making any changes to the operation device 1. Also, the processing load on the computer of the operation device 1 can be reduced compared to simulating the work of the work machine 120 in the virtual space using the computer of the operation device 1.

[0073] Furthermore, when operating the work machine 120 in the virtual space, the same sequence is used as when remotely operating the work machine 20. Therefore, there is no need to change the sequence when operating the work machine 120 in the virtual space. Therefore, it is possible to operate the work machine 120 in the virtual space without making any changes to the operation device 1.

[0074] Here, the machine state detection unit 71b of the operation device 1 transmits a specific signal related to a device equipped in the work machine to the simulator 90 at a predetermined timing. An example of the device equipped in the work machine is the engine 29. The specific signal is, for example, a signal for checking the operating state of the engine 29. When the machine state simulation unit 94 of the simulator 90 receives the specific signal from the operation device 1, it transmits a simulated response signal corresponding to the response signal to the operation device 1. In this way, by responding to the specific signal from the operation device 1 with a simulated response signal, the operating status of the operation system 100 can be checked.

[0075] Since the simulator 90 does not have a real engine, the machine state simulation unit 94 of the simulator 90 transmits to the operation device, as a simulated response signal, a response signal indicating that the operating state of the engine 29 is normal, or a response signal indicating that the operating state of the engine 29 is abnormal. If the simulated response signal is a response signal indicating that the operating state of the engine 29 is normal, operation by the simulator 90 can be continued. On the other hand, if the simulated response signal is a response signal indicating that the operating state of the engine 29 is abnormal, an abnormal state of the work machine 120 in the virtual space can be created in a pseudo manner. In this case, it is possible to allow the user to experience an abnormal state of the work machine, and also to allow the user to learn how to respond when an abnormality occurs.

[0076] 4, in this embodiment, the machine state simulation unit 94 is separate from the simulator-side control device 91. Therefore, the exchange between receiving the specific signal and transmitting the simulated response signal does not affect the simulation of the work of the work machine 120 in the virtual space by the simulator-side control device 91. Note that the machine state simulation unit 94 may be a part of the simulator-side control device 91.

[0077] (Operation system operation) Next, the operation of the operation system 100 will be described with reference to Fig. 5 which is a sequence flowchart. This sequence is performed while the operation device side control device 71 of the operation device 1 and the machine side control device 81 of the work machine 20 communicate with each other, or while the operation device side control device 71 of the operation device 1 and the simulator side control device 91 of the simulator 90 communicate with each other. In other words, the same sequence is used when the operation device 1 remotely operates the real work machine 20 and when the operation device 1 operates the work machine 120 in virtual space.

[0078] First, the operation system 100 executes a pre-operation process (step S1).

[0079] A flowchart of the pre-operation process is shown in Fig. 6. First, the operation device side control device 71 of the operation device 1 executes a start preparation process (step S11).

[0080] A flowchart of the start preparation process is shown in Fig. 7. First, the controller device 1 is started (step S31). Specifically, the controller device 1 is started by the user turning on a power button (not shown) of the controller device 1. Next, the controller device side control device 71 of the controller device 1 causes an authentication confirmation screen to be displayed on the information terminal 16 (step S32). On this authentication confirmation screen, the message "Do you want to authenticate?" and an authentication button are displayed.

[0081] Next, the operation device side control device 71 judges whether or not an authentication start operation has been performed (step S33). When the user operates the authentication button, the authentication start operation is performed. If it is judged in step S33 that the authentication start operation has not been performed (S33: NO), the operation device side control device 71 returns to step S32. On the other hand, if it is judged in step S33 that the authentication start operation has been performed (S33: YES), the operation device side control device 71 executes an authentication data acquisition process (step S34). In the authentication data acquisition process, authentication data for identifying the individual user is acquired by the camera 15. Thereafter, the operation device side control device 71 transmits an authentication signal including the authentication data to the server 110 (step S35), and ends this flow.

[0082] 6, when the server 110 receives the authentication signal, the server 110 performs a user authentication process (step S12). In the user authentication process, the user authentication is performed based on the authentication data included in the authentication signal and the authentication data registered in the server 110, and then an authentication result signal is transmitted to the controller device 1.

[0083] Next, the operation device side control device 71 of the operation device 1 executes an authentication result confirmation process (step S13).

[0084] A flowchart of the authentication result confirmation process is shown in FIG. 8. First, the operating device side control device 71 of the operating device 1 judges whether or not the user authentication has been successful (step S41). This judgment is made based on the authentication result data in the authentication result signal. If it is judged in step S41 that the user authentication has failed (S41: NO), the operating device side control device 71 executes a re-authentication process (step S42). In this re-authentication process, a picture indicating that the user authentication has failed and a re-authentication button for performing re-authentication are displayed on the information terminal 16. Then, when the user operates the re-authentication button, steps S32 to S35 of the start preparation process (see FIG. 7) described above are executed. Then, this flow ends.

[0085] On the other hand, when it is determined in step S41 that the user authentication is successful (S41: YES), the operation device side control device 71 causes the information terminal 16 to display an authentication success screen (step S43). On this authentication success screen, the words "Facial Authentication", an emoticon symbol indicating that the facial authentication is successful, and an OK button are displayed.

[0086] Next, the operating device control device 71 judges whether or not an understanding action has been performed (step S44). If the user operates the OK button on the authentication success screen, it is judged that an understanding action has been performed, and otherwise it is judged that an understanding action has not been performed. If it is judged that an understanding action has not been performed in step S44 (S44: NO), the operating device control device 71 returns to step S43. On the other hand, if it is judged that an understanding action has been performed in step S44 (S44: YES), the operating device control device 71 transmits a command signal to the server 110 (step S45) and ends this flow.

[0087] Returning to FIG. 6, when the server 110 receives the command signal, it transmits information about the selected destination to the operation device 1 (step S14). The selected destination is a work machine that is to be operated by the operation device 1. For example, the real work machine 20 that is present in work area A out of multiple work areas and is equipped with a bucket 33 is set as the selected destination. Also, a work machine 120 in the virtual space is set as the selected destination. The real work machine 20 has a means for detecting its position, such as GNSS. The position information and specification information of the real work machine 20 are uploaded to the server 110 via the network, and the real work machine 20 is registered in the server 110 as the selected destination.

[0088] When the operation device side control device 71 of the operation device 1 receives the selection destination information from the server 110, it displays a selection screen on the information terminal 16 (step S15). On this selection screen, a machine selection button showing the contents of multiple work machines selectable by the user and a button for returning to the previous screen are displayed.

[0089] Next, the operating device control device 71 judges whether or not a selection has been made (step S16). When any of the machine selection buttons has been operated by the user, it is judged that a machine selection has been made, and otherwise it is judged that a machine selection has not been made. When it is judged that a selection has not been made in step S16 (S16: NO), the operating device control device 71 returns to step S15. On the other hand, when it is judged that a selection has been made in step S16 (S16: YES), the operating device control device 71 transmits selection information to the server 110 (step S17).

[0090] When the server 110 receives the selection information, it transmits the identifier (step S18). The server 110 transmits the identifier of the work machine to the manipulation device 1, and also transmits the identifier of the manipulation device 1 to the connection destination. When the manipulation device side control device 71 of the manipulation device 1 receives the identifier of the work machine, it executes a communication setting process (step S19). In this communication setting process, various communication setting information such as the IP address of the connection destination is applied to the manipulation device 1. Similarly, when the connection destination receives the identifier of the manipulation device 1, it executes a communication setting process (step S20). In this communication setting process, various communication setting information such as the IP address of the manipulation device 1 is applied to the connection destination. As a result, a communication connection state is established between the manipulation device 1 and the connection destination, and communication between the two becomes possible.

[0091] The connection destination transmits the imaging data captured by the imaging device 27 to the operation device 1 (step S21). When the operation device side control device 71 of the operation device 1 receives the imaging data, it displays the imaging data as an image on the monitor 17 (step S22). Then, the operation device side control device 71 judges whether or not an input indicating that the connection destination is correct has been made (step S23). Whether or not the connection destination is correct is determined by the user confirming the image displayed on the monitor 17. The user inputs whether or not the connection destination is correct to the information terminal 16. If an input indicating that the connection destination is incorrect has been made in step S23 (S23: NO), the operation device side control device 71 returns to step S15. On the other hand, if an input indicating that the connection destination is correct has been made in step S23 (S23: YES), the operation device side control device 71 transmits a command signal to the connection destination (step S24) and ends this flow.

[0092] Returning to FIG. 5, next, the operation system 100 executes a start-up process (step S2).

[0093] A flowchart of the start-up process is shown in Fig. 9. First, the operating device side control device 71 of the operating device 1 determines whether or not the key is turned on (step S51). The key is turned on when the key inserted in the key switch 14 (see Fig. 2) is turned to the key-on position.

[0094] If it is determined in step S51 that the key is not turned on (S51: NO), the operation device side control device 71 repeats step S51. On the other hand, if it is determined in step S51 that the key is turned on (S51: YES), the operation device side control device 71 transmits a key-on signal to the connection destination (step S52). Next, the operation device side control device 71 determines whether the wipers are turned on (step S53). Wiper-on refers to the operation of the wiper operation button displayed on the information terminal 16.

[0095] In step S53, if it is determined that the wipers are not turned on (S53: NO), the operation device side control device 71 repeats step S53. On the other hand, if it is determined that the wipers are turned on (S53: YES), the operation device side control device 71 transmits a wiper operation signal to the connection destination (step S54).

[0096] When the connection destination receives the wiper operation signal, it operates the wipers 26 (step S55). If the connection destination is a real work machine 20, the real wipers 26 are operated. On the other hand, if the connection destination is a work machine 120 in the virtual space, an image of the wipers 26 operating is displayed on the main monitor 17a (see FIG. 3). The image of the wipers 26 operating is transmitted to the operation device 1 as imaging data. The user can check the connection status with the connection destination by checking that the wipers 26 have been operated.

[0097] The operating device side control device 71 of the operating device 1 judges whether the operation of the wiper 26 has been confirmed (step S56). When the confirmation button displayed on the information terminal 16 is operated by the user, it is judged that the operation of the wiper 26 has been confirmed, and otherwise it is judged that the operation of the wiper 26 has not been confirmed. In step S56, when it is judged that the operation of the wiper 26 has not been confirmed (S56: NO), the operating device side control device 71 returns to step S54. As a result, the wiper operation signal is transmitted once more to the connection destination. Note that, if the wiper 26 does not move even if the wiper operation signal is transmitted to the connection destination multiple times, the operating device side control device 71 returns to step S15 of the pre-operation process (see FIG. 6).

[0098] On the other hand, if it is determined in step S56 that the operation of the wipers 26 has been confirmed (S56: YES), the operation device side control device 71 transmits a command signal to the connection destination (step S57). Next, the operation device side control device 71 determines whether or not a key start has been performed (step S58). A key start is when the key inserted in the key switch 14 (see FIG. 2) is rotated to the key start position.

[0099] If it is determined in step S58 that key start has not occurred (S58: NO), the operation device side control device 71 repeats step S58. On the other hand, if it is determined in step S58 that key start has occurred (S58: YES), the operation device side control device 71 transmits a key start signal to the connection destination (step S59).

[0100] When the connected device receives the key start signal, it starts the engine (step S60), transmits an engine start signal to the operation device 1 (step S61), and ends this flow.

[0101] Returning to FIG. 5, next, the operation system 100 executes operation processing (step S3). The operation processing is processing for the user to operate the real work machine 20 or the work machine 120 in the virtual space. In the operation processing, various information for operating the work machine, such as an operation signal from the operation lever 13, is transmitted to the connection destination, and the connected work machine is operated based on this. In addition, information on the state of the work machine, such as the inclination of the upper rotating body 22 with respect to the horizontal plane, the operating status of the wipers 26, and information on remaining fuel, is transmitted from the connection destination to the operation device 1.

[0102] In this operation process, a connection check with the connection destination and a status check of the connection destination are performed at a predetermined timing. The connection check and status check are performed by sending a signal to the connection destination for checking the communication status, operation status, etc., and receiving a signal indicating that the communication status and operation status are normal, information regarding communication or operation failures, etc. from the connection destination as a response. Specifically, the equipment provided in the work machine 20 is checked for power shorts, disconnection abnormalities, communication errors with the control device, connection errors of equipment (cameras, encoders, etc.), etc.

[0103] Next, the operation system 100 executes a termination process (step S4).

[0104] A flowchart of the termination process is shown in Fig. 10. First, the operating device side control device 71 of the operating device 1 determines whether or not the key has been turned off (step S71). Key-off refers to the key inserted in the key switch 14 (see Fig. 2) being turned to the key-off position. If it is determined in step S71 that the key has not been turned off (S71: NO), the operating device side control device 71 repeats step S71. On the other hand, if it is determined in step S71 that the key has been turned off (S71: YES), the operating device side control device 71 transmits a key-off signal to the connection destination (step S72).

[0105] When the connected device receives the key-off signal, it stops the engine 29 (step S73) and transmits an engine stop signal to the operation device 1 (step S74).

[0106] When the operation device side control device 71 of the operation device 1 receives the engine stop signal, it executes an action selection process (step S75).

[0107] A flowchart of the operation selection process is shown in Fig. 11. First, the operation device side control device 71 of the operation device 1 displays an operation selection screen on the information terminal 16 (step S81). This operation selection screen displays a restart button, a machine transfer button, and an operation end button. When the restart button is operated, the same work machine is restarted. When the machine transfer button is operated, transfer to another work machine is performed. When the operation end button is operated, the operation is ended.

[0108] The operation device side control device 71 judges whether or not the operation end is selected (step S82). If it is judged in step S82 that the operation end is selected (S82: YES), the operation device side control device 71 transmits an operation end signal to the server 110 (step S83) and ends this flow.

[0109] On the other hand, if it is determined in step S82 that the operation end has not been selected (S82: NO), the operation device side control device 71 determines whether or not switching to another work machine has been selected (step S84). If it is determined in step S84 that switching to another work machine has been selected (S84: YES), the operation device side control device 71 transmits an operation end signal to the server 110 (step S85). Then, the operation device side control device 71 executes switching processing (step S86) and ends this flow.

[0110] In the transfer process, steps S32 to S35 of the start preparation process (see FIG. 7) are executed, and step S13 and subsequent steps of the pre-operation process (see FIG. 6) are executed. At the same time, the server 110 executes step S12 and subsequent steps of the pre-operation process (see FIG. 6). At the same time, the connection destination executes step S20 and subsequent steps of the pre-operation process (see FIG. 6).

[0111] On the other hand, if it is determined in step S84 that switching to another work machine has not been selected (S84: NO), the operation device side control device 71 determines whether or not restart has been selected (step S87). If it is determined in step S87 that restart has not been selected (S87: NO), the operation device side control device 71 returns to step S81.

[0112] On the other hand, if it is determined in step S84 that restart has been selected (S84: YES), the operation device side control device 71 executes restart processing (step S88) and ends this flow.

[0113] In the restart process, steps S32 to S35 of the start preparation process (see FIG. 7) are executed, as well as step S13 of the pre-operation process (see FIG. 6) and steps S2 and onwards of the sequence (see FIG. 5). At the same time, step S12 of the pre-operation process (see FIG. 6) is executed in the server 110. At the same time, steps S2 and onwards of the sequence (see FIG. 5) are executed in the connected device.

[0114] 10, when the server 110 receives the operation end signal, it executes a communication setting discarding process (step S76). In the communication setting discarding process, a communication setting discarding signal is transmitted from the server 110 to each of the operation device 1 and the connection destination.

[0115] When the operation device 1 receives the communication setting discard signal, the operation device side control device 71 executes a communication setting discard process (step S77). In this communication setting discard process, various communication setting information set in the operation device 1 is discarded. In addition, when the connection destination receives the communication setting discard signal, it executes a communication setting discard process (step S78). In this communication setting discard process, various communication setting information set in the connection destination is discarded. As a result, the connection between the operation device 1 and the connection destination is cut off. Then, this flow ends.

[0116] Returning to FIG. 5, the operation system 100 ends the sequence flow.

[0117] (effect) As described above, according to the operation device 1 of this embodiment, as shown in FIG. 1, in a virtual space in which the work of the work machine 20 is simulated, it is possible to operate the work machine 120 in the virtual space using at least a part of the input device 2. In other words, the operation device 1 can not only remotely operate the real work machine 20, but also operate the work machine 120 in the virtual space. Therefore, the user can operate the work machine 120 in the virtual space with the same operation or substantially the same operation as that for remotely operating the real work machine 20. Therefore, for example, the user can operate the work machine 120 in the virtual space to learn the procedure for remotely operating the real work machine 20 and the method for remotely operating the real work machine 20. Then, after operating the work machine 120 in the virtual space to practice a predetermined work, the user can actually perform the predetermined work by remotely operating the real work machine 20. This allows the user to utilize the skills acquired in the virtual space in the real work machine 20.

[0118] 1, the user can select between remote operation of the work machine 20 and operation of a work machine 120 in a virtual space. Thus, the user can practice a predetermined work by selecting operation of the work machine 120 in the virtual space. The user can also perform actual work by selecting remote operation of the real work machine 20.

[0119] Also, as shown in Fig. 4, the operation device 1 is separate from the simulator 90 that simulates work by the work machine 120 in a virtual space. Therefore, for example, the sequence used when remotely operating the real work machine 20 and the sequence used when operating the work machine 120 in the virtual space can be the same. In this case, the work machine 120 in the virtual space can be operated without any modification to the operation device 1. Also, the processing load on the computer of the operation device 1 can be reduced compared to simulating the work of the work machine 120 in the virtual space using the computer of the operation device 1.

[0120] Furthermore, when operating the work machine 120 in the virtual space, the same sequence is used as when remotely operating the work machine 20. Therefore, there is no need to change the sequence when operating the work machine 120 in the virtual space. Therefore, it is possible to operate the work machine 120 in the virtual space without making any changes to the operation device 1.

[0121] Moreover, according to the operation system 100 of this embodiment, as shown in FIG. 1, the operation device 1 is capable of operating the work machine 120 in a virtual space. That is, the operation device 1 is capable of remotely operating not only the real work machine 20 but also the work machine 120 in the virtual space. Thus, the user can operate the work machine 120 in the virtual space with the same operation or substantially the same operation as that for remotely operating the real work machine 20. Thus, for example, the user can operate the work machine 120 in the virtual space to learn the procedure for remotely operating the real work machine 20 and the method for remotely operating the real work machine 20. Then, after operating the work machine 120 in the virtual space to practice a predetermined task, the user can remotely operate the real work machine 20 to actually perform the predetermined task. This allows the user to utilize the skills acquired in the virtual space in the real work machine 20.

[0122] Furthermore, the same sequence as that used between the work machine 20 and the operation device 1 is used between the operation device 1 and the simulator 90, so that the work machine 120 in the virtual space is operated by the operation device 1. The user can operate the work machine 120 in the virtual space by operating the same or substantially the same procedure (sequence) as when remotely operating the real work machine 20. Thus, for example, the user can operate the work machine 120 in the virtual space to learn the procedure for remotely operating the real work machine 20. Then, after operating the work machine 120 in the virtual space and practicing a predetermined procedure, the user can remotely operate the real work machine 20 to actually perform the predetermined procedure. This allows the user to utilize the skills acquired in the virtual space in the real work machine 20.

[0123] Also, as shown in FIG. 4, when the work machine 20 receives a specific signal related to a device equipped in the work machine 20 from the operation device 1, the work machine 20 transmits a response signal to the operation device 1. When the simulator 90 receives a specific signal from the operation device 1, the simulator 90 transmits a simulated response signal corresponding to the response signal to the operation device 1. In this way, by responding to the specific signal from the operation device 1 with a response signal or a simulated response signal, the operation status of the operation system 100 can be confirmed. Also, for example, the specific signal is a signal for confirming the operation state of the engine 29. In this case, if the operation state of the engine 29 is normal, the work machine 20 transmits a response signal to that effect to the operation device 1, and if the operation state of the engine 29 is abnormal, the simulator 90 transmits a response signal to that effect to the operation device 1. On the other hand, since the simulator 90 does not have a real engine 29, the simulator 90 transmits a response signal to the operation device 1 indicating that the operation state of the engine 29 is normal or a response signal to the operation device 1 indicating that the operation state of the engine 29 is abnormal, as a simulated response signal. If the simulated response signal is a response signal indicating that the operating state of the engine 29 is normal, it is possible to continue operation using the simulator 90. If the simulated response signal is a response signal indicating that the operating state of the engine 29 is abnormal, it is possible to create a pseudo abnormal state of the work machine 120 in the virtual space. In this case, it is possible to allow the user to experience an abnormal state of the work machine, and also to allow the user to learn how to respond when an abnormality occurs.

[0124] Although the embodiments of the present invention have been described above, they are merely illustrative examples and do not limit the present invention, and the specific configurations and the like can be appropriately modified in design. Furthermore, the actions and effects described in the embodiments of the invention are merely a list of the most preferable actions and effects resulting from the present invention, and the actions and effects of the present invention are not limited to those described in the embodiments of the present invention.

[0125] For example, the simulator 90 may select and execute multiple types of simulation software. In this case, work by multiple types of work machines 120 can be simulated by a single simulator 90. The same applies when the server 110 performs the simulation.

[0126] Also, for example, a dump truck may be placed in the virtual space, the imaging device 27 may be placed above the dump truck, and an image captured by the imaging device 27 and overlooking the loading section of the dump truck may be displayed on the monitor 17 (see FIG. 3) of the operation device 1. In this case, when the user practices operation with the simulator 90, the relative positions of the attachment 30 and the dump truck can be easily understood by using the image captured by the imaging device 27 provided in the cab 23 and the overlooking image of the loading section of the dump truck as a reference for operation. This makes it easier for the user to learn the sense of distance when viewed in the same way as the actual work machine 20.

[0127] Also, for example, the imaging device 27 may be placed in a position that allows at least one of a bird's-eye view and a side view of the work machine 120 in the virtual space, and the monitor 17 of the operation device 1 (see FIG. 3) may display an image of a bird's-eye view or a side view of the work machine 120 in the virtual space captured by the imaging device 27. In this case, if the user is a beginner with a low level of proficiency, the user can visually confirm the movement of the entire attachment 30, making it easy to understand the relationship between the operation lever 13 and the operation of the attachment 30, and thus becoming proficient in operation more quickly. [Explanation of symbols]

[0128] 1 Operating device 2 Input Devices 3 Output Device 4 Driver's seat 6 Operating mechanism 7 Seat tilt device 11 Travel lever 12 Travel pedal 13 Operating lever 14 Key Switches 15 Camera 16 Information terminal 17 Monitor 18 Emergency stop switch 19 Speaker 20 Work Machinery 21 Undercarriage 22 Upper rotating body 23 Cab 24 Machine body 25 Swivel 26 Wiper 27 Imaging Device 28. Mike 29 Engine 30 Attachment 31 Boom 32 Arm 33 Bucket 34 Link member 40 Cylinder 41 Boom cylinder 42 Arm Cylinder 43 Bucket Cylinder 50 Sensors 52 Rotation angle sensor 55 Inclination Angle Sensor 60 Inclination Sensor 61 Boom tilt angle sensor 62 Arm inclination angle sensor 63 Bucket tilt angle sensor 71 Control device on the operation device side 72 Controller side storage device 73 Control device side communication device 81 Machine control device 82 Machine side memory device 83 Machine input device 84 Machine communication device 90 Simulator (operation verification device) 91 Simulator side control device 92 Simulator side storage device 93 Simulator side communication device 94 Machine State Simulation Section 100 Operating System 110 Server 111 Server side control device 112 Server side storage device 113 Server side communication device 120 Virtual Machinery

Claims

1. An operation device capable of remotely operating a work machine in response to an operation by a user via an input device, An operation device capable of operating the work machine in a virtual space in which work by the work machine is simulated, using at least a part of the input device.

2. 2. The operation device according to claim 1, wherein a user can select between remote operation of the work machine and operation of the work machine in the virtual space.

3. 3. The operating device according to claim 1, wherein the operating device is separate from an operation verification device that simulates work performed by the work machine in the virtual space.

4. 4. The operating device according to claim 3, wherein, when operating the work machine in the virtual space, the same sequence as when remotely operating the work machine is used.

5. A work machine; an operation device capable of remotely operating the work machine; an operation verification device that simulates work of the work machine in a virtual space; having The operation system, wherein the operation device is capable of operating the work machine in the virtual space.

6. The operation system described in claim 5, characterized in that the work machine in the virtual space is operated by the operation device by using the same sequence between the operation device and the operation verification device as the sequence used between the work machine and the operation device.

7. when the work machine receives a specific signal related to a device equipped in the work machine from the operation device, the work machine transmits a response signal to the operation device; 7. The operation system according to claim 5, wherein the operation verification device transmits a simulated response signal corresponding to the response signal to the operation device when the operation verification device receives the specific signal from the operation device.

Images