Operation device and operation system

Abstract

To provide an operation device and so on capable of preventing a waiting time in processing for establishing communication with an operation target from being wastefully elapsed.SOLUTION: An operation device 20 for operating an operation target comprises: a communication device 250 capable of communicating with the operation target; and a control device 200 capable of executing a communication establishing processing of establishing the communication with the operation target through the communication device, and specified item setting processing of receiving setting of a content of a specified item for supporting the operation of the operation target by the operation device after executing the communication establishing processing, through an input part at least during execution of the communication establishing processing.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an operation device and an operation system for operating an operation object. [Background technology]

[0002] A remote operation system configured to be capable of selectively remotely operating a plurality of target work machines (capable of switching the target work machine from a first work machine to another work machine) has been disclosed (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-139144 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in an operation system such as that exemplified in Prior Art 1, it takes a certain amount of time to establish communication between an operation device and an operation target so that they can be operated. While the process for establishing the communication is being executed, the operator has to wait for the establishment of communication to be completed, resulting in wasted time.

[0005] An object of the present invention is to provide an operation device and the like that can prevent a waiting time from passing unnecessarily in a process for establishing communication with an operation target. [Means for solving the problem]

[0006] A first aspect of the present invention provides an operating device for operating an operating object, the operating device including a communication device capable of communicating with the operating object, a control device capable of executing a communication establishment process for establishing the communication with the operating object through the communication device, and a designated item setting process for receiving, at least during the communication establishment process, through an input unit, the setting of the contents of designated items for supporting the operation of the operating object by the operating device after the communication establishment process is executed.

[0007] According to the operation device having the above configuration, during the execution of the communication establishment process, it is possible to set the contents of the designated items for supporting the operation of the operation target by the operation device after the completion of the communication establishment process, thereby making use of the waiting time during the execution of the communication establishment process.

[0008] An operating device according to a second aspect of the present invention is the operating device according to the first aspect, wherein, when the operating target is switched from one operating target to another operating target, the control device executes a communication disconnection process to disconnect communication established between the communication device and the one operating target, and a communication establishment process to establish communication between the operating device and the other operating target, and it is preferable that the control device accepts setting of the content of the specified item through the input unit when at least one of the communication disconnection process and the communication establishment process is executed in conjunction with switching of the operating target through the communication device.

[0009] According to the operation device having this configuration, when the operation target is switched from one operation target to another operation target, a communication disconnection process and a communication establishment process are executed. The "communication disconnection process" is a process for disconnecting the communication established between the operation device and the one operation target. The "communication establishment process" here is a process for establishing communication between the operation device and the other operation target. When at least one of the "communication disconnection process" and "the communication establishment process" is executed, it becomes possible to set the contents of the specified items via the input unit. Therefore, the waiting time during the execution of the process for switching the operation target is utilized.

[0010] In an operating device according to a third aspect of the present invention, in the operating device according to the first or second aspect, it is preferable that, when the operating object is switched from one operating object to another operating object having specifications different from those of the one operating object, the control device accepts, via the input unit, a setting of the content of the specified item corresponding to the difference in the specifications.

[0011] According to the operation device having the above configuration, when an operation target is switched from one operation target to another operation target having specifications different from those of the first operation target, it is possible to set, via the input unit, the specified items corresponding to the differences in the specifications, which enables smooth operation of the other operation target after the switching of the operation target.

[0012] An operating device according to a fourth aspect of the present invention is an operating device according to any one of the first to third aspects, further comprising a simulation device for operating a virtual operating object corresponding to the operating target in a virtual space, and it is preferable that the control device assists in operating the virtual operating object in the virtual space based on the setting contents of the specified items received through the input unit.

[0013] According to the operation device having the above configuration, it is possible to virtually experience, through the simulation device, the support mode and / or operation mode of the operation target that is predicted to occur according to the setting contents of the items specified through the input unit. Based on the experience, it is possible to reset or change the settings of the items specified through the input unit.

[0014] An operating device according to a fifth aspect of the present invention is preferably an operating device according to any one of the first to fourth aspects, further comprising a storage device that stores setting contents of the designated items, and an output unit that outputs the setting contents of the designated items stored in the storage device.

[0015] According to the operation device having the above configuration, the operator can confirm the current setting contents of the designated items output to the output unit. Based on the output contents, the designated items can be reset or changed via the input unit.

[0016] An operating device according to a sixth aspect of the present invention is the operating device according to the fifth aspect, and preferably has a function of authenticating an operator, and the output unit outputs setting contents of the specified items stored in the storage device in association with the authenticated operator.

[0017] According to the operation device having this configuration, the contents of the designated items last set by each operator (or the contents of the designated items appropriate for each operator) are stored and output to the output unit. The output contents may be defined as the default contents of the designated items.

[0018] An operating device according to a seventh aspect of the present invention is the operating device according to the fifth aspect, further comprising a function of authenticating an operator, and it is preferable that the storage device stores the setting contents of the specified items received through the input unit in association with the authenticated operator.

[0019] According to the operation device having the above configuration, the contents of the designation items last set by each operator can be stored in the storage device.

[0020] In an operating device according to an eighth aspect of the present invention, in the operating device according to any one of the first to seventh aspects, it is preferable that the control device accepts settings via the input unit regarding the content of a specified item selected from among the plurality of specified items.

[0021] According to the operation device having the above-described configuration, only the contents of one or more designated items selected (through the input unit) from among the multiple designated items are set through the input unit.

[0022] What is provided by the present invention is an operation system, which is configured by an operation device according to any one of the first to eighth aspects and the operation target. [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 is a configuration diagram of an operation system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a perspective view of a remote control device according to one embodiment of the present invention. [Diagram 3] FIG. 3 is a perspective view of a work machine according to one embodiment of the present invention. [Figure 4] FIG. 4 is a flowchart of an operation system according to an embodiment of the present invention. [Diagram 5] FIG. 5 is a flowchart of an authentication processing function of the operation system according to one embodiment of the present invention. [Figure 6] FIG. 6 is a flowchart of an operation processing function of an operation system according to an embodiment of the present invention. [Figure 7A] FIG. 7A is a diagram showing a first authentication screen for an operator in the operation system according to one embodiment of the present invention. [Figure 7B] FIG. 7B is a diagram showing a second authentication screen for an operator in the operation system according to one embodiment of the present invention. [Figure 8A] FIG. 8A is a diagram showing a work content designation screen in the operation system according to one embodiment of the present invention. [Figure 8B] FIG. 8B is a diagram showing an operation target designation screen in the operation system according to one embodiment of the present invention. [Figure 8C] FIG. 8C is a diagram showing a confirmation screen for the work content and the operation target in the operation system according to one embodiment of the present invention. [Figure 9A] FIG. 9A is a diagram showing a first lever pattern setting screen in the operation system according to one embodiment of the present invention. [Figure 9B]FIG. 9B is a diagram showing a second lever pattern setting screen in the operation system according to one embodiment of the present invention. [Figure 10] FIG. 10 is a diagram showing a lever sensitivity setting screen in the operation system according to one embodiment of the present invention. [Figure 11] FIG. 11 is a diagram showing a screen for setting the seat vibration mode in the operation system according to one embodiment of the present invention. [Figure 12] FIG. 12 is a diagram showing a setting screen for operation assistance in the operation system according to one embodiment of the present invention. [Figure 13] FIG. 13 is a diagram showing a guidance setting screen in the operation system according to one embodiment of the present invention. [Figure 14] FIG. 14 is a diagram showing a depth sense image in the operation system according to one embodiment of the present invention. [Figure 15] FIG. 15 is a diagram showing a terrain image in an operation system according to one embodiment of the present invention. [Figure 16A] FIG. 16A is a diagram showing a first button function setting screen in the operation system according to one embodiment of the present invention. [Figure 16B] FIG. 16B is a diagram showing a second button function setting screen in the operation system according to one embodiment of the present invention. [Figure 17] FIG. 17 is a diagram showing an operation selection screen in an operation system according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] (Operation System Configuration) FIG. 1 is a configuration diagram of an operation system according to an embodiment of the present invention. This operation system has an information processing terminal 10 and a remote control device 20 (operation device), and is configured to remotely operate a work machine 40 (operation target) using the remote control device 20. In another embodiment, the operation system may include the work machine 40. The information processing terminal 10 and the remote control device 20 are configured to be able to communicate with each other via a first communication network. The remote control device 20 and the work machine 40 are configured to be able to communicate with each other via a second communication network. The first communication network and the second communication network may have the same communication standard or different standards. The information processing terminal 10 may be omitted. The remote control device 20 may have the functions of the information processing terminal 10 built in. The "operation target" may be any device that has a function of performing a work task by remote control, such as a vehicle, a tiller, or an airplane, in addition to the work machine 40.

[0025] (Configuration of information processing terminal) As shown in FIG. 1, the information processing terminal 10 includes a terminal control device 100 (control device, control unit), a terminal input interface 110 (input interface, input unit), a terminal output interface 120 (output interface, output unit), and a terminal wireless communication device 150 (communication device) for communicating via a first communication network. The information processing terminal 10 is configured by a smartphone, tablet terminal, or notebook computer carried by an operator or mounted on a remote control device 20. The terminal control device 100 is configured by an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing according to the software on the data. The terminal input interface 110 and the terminal output interface 120 are configured by, for example, a touch panel display.

[0026] As described above, the information processing terminal 10 may be a component of the remote control device 20. In this case, the information processing terminal 10 and the remote control device 20 may be configured to be capable of wired or wireless communication, the terminal input interface 110 may constitute a part of the remote input interface 210, and the terminal output interface 120 may constitute a part of the remote output interface 220.

[0027] (Configuration of remote control device) As shown in Fig. 1, the remote operation device 20 has a remote control device 200 (control device, control unit), a remote input interface 210 (input interface, input unit), a remote output interface 220 (output interface, output unit), and a remote wireless communication device 250 (communication device) capable of communicating with the information processing terminal 10 or the work machine 40 via a first communication network and a second communication network. The remote wireless communication device 250 is assigned an operation device identifier (IP address) for identifying the remote wireless communication device 250 in the second communication network and setting it as a transmission destination. The remote control device 200 is composed of an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing according to the software on the data.

[0028] The remote input interface 210 includes a remote operation mechanism 211 and a face authentication sensor 212. The remote output interface 220 includes a remote image output device 221, a remote audio output device 222, and a seat drive device 224.

[0029] FIG. 2 is a perspective view of the remote control mechanism 211 according to this embodiment. The remote control mechanism 211 has the devices and members shown in FIG. 2, and includes a traveling operation device, a swivel operation device, a boom operation device, an arm operation device, and an end attachment operation device. Each operation device has an operation lever that receives a rotation operation. The operation lever (travel lever) of the traveling operation device is operated to drive a hydraulic traveling motor (actuator) that constitutes the traveling drive mechanism in the lower travel body 41 of the work machine 40. The travel lever may also serve as a traveling pedal. For example, a traveling pedal fixed to the base or lower end of the travel lever may be provided. The operation lever (swivel lever) of the swing operation device is operated to move a hydraulic swing motor (actuator) that constitutes the swing drive mechanism of the work machine 40. The operation lever (boom lever) of the boom operation device is operated to move the boom cylinder 442 of the work machine 40. The operating lever (arm lever) of the arm operating device is operated to move the arm cylinder 444 of the work machine 40. The operating lever (tip attachment lever) of the end attachment operating device is operated to move the end attachment cylinder 446 of the work machine 40. An optional pedal that is operated to move the end attachment of the work machine 40, etc. may be disposed on the side of the travel pedal.

[0030] The operation levers constituting the remote operation mechanism 211 are arranged around the seat 24 on which the operator sits, as shown in FIG. 2, for example. The seat 24 has a seat part 241 installed on the upper side of a seat base 240 and a backrest part 242, and is in the form of a high-back chair with armrests. The seat 24 may be in the form of a low-back chair without a headrest, or in the form of a chair without the backrest part 242, or in any other form on which the operator can sit. The seat 24 and / or the side of the seat 24 may be configured so that the information processing terminal 10 can be mounted. In this case, it is preferable that the touch screen of the information processing terminal 10 is mounted facing the seat 24 so that the display of the information processing terminal 10 can be easily operated and viewed by the operator seated on the seat 24. In addition, a cutoff lever (not shown) may be mounted on the seat 24 and / or the side of the seat 24. The cutoff lever has a function of prohibiting the operation of the work machine (stopping the actuator operation) when tilted from the unlocked position to the locked position.

[0031] Left and right travel levers 2110 corresponding to the left and right crawlers are arranged side by side in front of the seat 24. One operation lever may serve as multiple operation levers. For example, the left operation lever 2111 provided in front of the left frame of the seat 24 shown in FIG. 2 may function as an arm lever when operated in the front-rear direction, and as a rotation lever when operated in the left-right direction. Similarly, the right operation lever 2112 provided in front of the right frame of the seat shown in FIG. 2 may function as a boom lever when operated in the front-rear direction, and as a tip attachment lever when operated in the left-right direction. Such a lever pattern may be arbitrarily changed by an operation instruction from the operator.

[0032] 2, the face authentication sensor 212 is configured with a camera and / or a distance measuring sensor arranged in front of the seat 24. An authentication process for the operator is executed based on an image of the operator's face captured by the camera and / or a three-dimensional shape of the operator's face captured by the distance measuring sensor.

[0033] The remote image output device 221 is composed of a single image output device or a plurality of image output devices. For example, as shown in FIG. 2, the remote image output device 221 is composed of a central remote image output device 2210, a left remote image output device 2211, and a right remote image output device 2212, each having a substantially rectangular screen, which are arranged in front of the seat 24, diagonally forward to the left, and diagonally forward to the right. The shapes and sizes of the screens (image display areas) of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be the same or different. The remote image output device 221 may be composed of a single curved or bendable image output device, or two or four or more image output devices arranged to surround the front of the seat 24.

[0034] 2, the right edge of the left remote image output device 2211 is adjacent to the left edge of the central remote image output device 2210 so that the screens of the central remote image output device 2210 and the left remote image output device 2211 form a first specified angle (e.g., 120° to 150°). As shown in FIG. 2, the left edge of the right remote image output device 2212 is adjacent to the right edge of the central remote image output device 2210 so that the screens of the central remote image output device 2210 and the right remote image output device 2212 form a second specified angle (e.g., 120° to 150°).

[0035] The screens of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be parallel to the vertical direction or may be inclined to the vertical direction. At least one of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be composed of a plurality of divided image output devices. For example, the central remote image output device 2210 may be composed of vertically adjacent image output devices each having a substantially rectangular screen.

[0036] The remote audio output device 222 is composed of one or more speakers, and is composed of audio output devices arranged, for example, behind the seat 24, at the rear of the left armrest, and at the rear of the right armrest. The specifications of the audio output devices may be the same or different. The remote audio output device 222 may be configured to output the sound of the work machine 40 collected by a microphone mounted on the work machine 40.

[0037] The seat drive device 224 constitutes a "seat device" together with the seat 24. The seat drive device 224 is configured to tilt or swing the seat 24 relative to the seat base 240 (or integrally with the seat base 240) about the pitch axis and / or the roll axis (about the roll axis as necessary). The seat drive device 224 may be configured to oscillate the seat 24 relative to the seat base 240 (or integrally with the seat base 240) in at least one direction of the pitch axis, the roll axis, and the yaw axis.

[0038] (Work machine configuration) As shown in FIG. 1, the work machine 40 has a real machine control device 400, a real machine input interface 410, a real machine output interface 420, and a real machine wireless communication device 450 for communicating via the second communication network. The real machine wireless communication device 450 is assigned a real machine identifier (IP address) for identifying the real machine wireless communication device 450 in the second communication network and making it a transmission destination. Each of the components of the real machine control device 400 is composed of an arithmetic processing device (a single-core processor or a multi-core processor or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing according to the software on the data. The remote control device 200 communicates with a specific work machine 40 selected from a plurality of work machines 40 via the remote wireless communication device 250, thereby enabling the specific work machine 40 to be operated by the remote input interface 210.

[0039] Fig. 3 is a perspective view of a work machine 40 according to this embodiment. The work machine 40 is, for example, a hydraulic excavator, and as shown in Fig. 3, includes a crawler-type lower traveling body 41 constituting a traveling drive mechanism, and an upper rotating body 42 rotatably mounted on the lower traveling body 41 via a swing drive mechanism. A cab 43 (operator's cab) is provided on the front left side of the upper rotating body 42. A work mechanism 44 is provided on the front right side of the upper rotating body 42. The cab 43 may be configured to be driven to move up and down relative to the upper rotating body 42 by a lifting drive mechanism (not shown) having a link mechanism and an actuator, and may be configured to change the tilt angle relative to the upper rotating body 42 by a tilt drive mechanism (not shown) having an actuator.

[0040] The real machine input interface 410 has a real machine operation mechanism 411, a real machine image pickup device 412, and a real machine sensor group 416. The real machine operation mechanism 411 has a plurality of operation levers arranged around a seat arranged inside the cab 43 in the same manner as the remote operation mechanism 211. An actuation mechanism or a robot that receives a signal according to the operation mode of the remote operation lever and moves the real machine operation lever based on the received signal is provided in the cab 43. The real machine operation mechanism 411 may be configured by an actuation mechanism that moves a control valve that constitutes a hydraulic circuit mounted on the work machine 40. The real machine image pickup device 412 is installed, for example, inside the cab 43, and picks up an image of the environment in front of the cab 43 including at least a part of the working mechanism 44 through the front window and the left and right side windows. A part or all of the front window and the side windows may be omitted. The real machine image pickup device 412 may be mounted outside the cab 43 as long as it picks up an image of the environment in front of the cab 43 including at least a part of the working mechanism 44.

[0041] The actual machine sensor group 416 includes various sensors configured to output signals to the actual machine control device 400 according to parameter values ​​related to the state of the work machine 40. The state of the work machine 40 includes the actual machine attitude detected by the actual machine attitude sensor, that is, the tilt angle (pitch angle) about the pitch axis and / or the tilt angle (roll angle) about the roll axis with respect to the vertical direction of the upper rotating body 42. The actual machine attitude sensor is configured, for example, with a three-dimensional acceleration sensor. The state of the work machine 40 may include whether or not the prime mover (equipment that generates vibration, such as an engine or a work oil pump) is operating, as well as the hoisting angle of the boom 441 relative to the upper rotating body 42, the rotation angle of the arm 443 relative to the boom 441, the rotation angle of the tip attachment 445 (lift magnet) relative to the arm 443, the operating oil pressure, the remaining amount of fuel, and the like. The state of the work machine 40 may include the sound of the work machine 40 collected by a microphone mounted on the work machine 40.

[0042] 3, the working mechanism 44 (operating mechanism) has a boom 441 having a base end movably attached to the upper rotating body 42, an arm 443 having a base end rotatably connected to the tip of the boom 441, and a tip attachment 445 having a base end rotatably connected to the tip of the arm 443. The working mechanism 44 is equipped with a boom cylinder 442, an arm cylinder 444, and a tip attachment cylinder 446 as actuators each composed of an extendable hydraulic cylinder.

[0043] As the tip attachment 445, in addition to a bucket having a container for storing objects such as soil and sand, a lifting magnet, a grapple that grips an object by opening and closing multiple opposing gripping claws, a bucket with a hook that can hook an object to a hook attached to the back and lift it, and a breaker having a chisel that is supported on the breaker body so as to be capable of reciprocating and striking the work object may be used. The chisel of the breaker may be configured to be operable by a switch. In addition, a swing mechanism that swings the tip end left and right relative to each base end, and a slide arm that is provided with a slidable extension mechanism on the arm 443 and is configured to be extendable and retractable on the arm 443 may be provided at the base end of the boom 441 and the base end of the arm 443, and may be configured to be operable by a switch or an optional pedal. A dozer having a blade lifting drive mechanism that drives the blade to lift and lower by a dozer cylinder is provided at the front side of the lower traveling body 41, and may be configured to be operable by a switch or an optional pedal.

[0044] The boom cylinder 442 is interposed between the boom 441 and the upper rotating body 42 so as to extend and retract when supplied with hydraulic oil to rotate the boom 441 in the hoisting direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441 so as to extend and retract when supplied with hydraulic oil to rotate the arm 443 about a horizontal axis relative to the boom 441. The tip attachment cylinder 446 is interposed between the tip attachment 445 and the arm 443 so as to extend and retract when supplied with hydraulic oil to rotate the tip attachment 445 about a horizontal axis relative to the arm 443.

[0045] The real machine control device 400 includes a switch that can switch between the presence and absence of power supply to the real machine control device 400. The switch switches from a state in which there is no power supply to the real machine control device 400 to a state in which there is power supply to the real machine control device 400, and the real machine control device 400 starts operating. When the real machine control device 400 starts operating, the real machine control device 400 starts the real machine wireless communication device 450, and the real machine wireless communication device 450 connects to the first communication network and the second communication network. In addition, the real machine control device 400 that has started operating starts the real machine imaging device 412 and the real machine sensor group 416. In a state in which there is no power supply to the real machine control device 400 due to the switch, the real machine control device 400 stops operating, the real machine wireless communication device 450 is stopped, and communication with the information processing terminal 10 or the work machine 40 via the first communication network and the second communication network is not possible. In addition, when the real machine control device 400 stops operating, the operations of the real machine imaging device 412 and the real machine sensor group 416 stop.

[0046] (function) Fig. 4 is a flowchart of an operation system according to an embodiment of the present invention. Fig. 5 is a flowchart of an authentication processing function of the operation system according to this embodiment. Fig. 6 is a flowchart of an operation processing function of the operation system according to this embodiment. Fig. 7A is a diagram showing a first authentication screen for an operator in the operation system according to this embodiment. Fig. 7B is a diagram showing a second authentication screen for an operator in the operation system according to this embodiment. The functions of the remote operation device 20 and the remote operation system having the above configuration will be described with reference to Figs. 4 to 7B.

[0047] When the operation system starts processing, first, the remote control device 200 executes authentication processing (FIG. 4 / STEP 200). Specifically, as shown in FIG. 7A, a message Q01 prompting the operator to sit in the seat 24 and an authentication button Q02 are output to the remote output interface 220 (remote image output device 221) (output, display by the remote output interface 220 (same hereinafter)). The various buttons output to the remote output interface 220 (remote image output device 221) form a touch panel type input interface.

[0048] When the authentication button Q02 is tapped, etc., a face authentication button Q03 and an identifier authentication button Q04 are output to the remote output interface 220 (remote image output device 221) as shown in FIG. 7B. When the face authentication button Q03 is tapped, for example, a camera and / or a distance measuring sensor mounted on the remote operation device 20 and / or the information processing terminal is used to perform face authentication of the operator (and further by querying a database). When the identifier authentication button Q04 is tapped, the operator authentication is performed using the operator identifier (or the operator identifier (user ID) and passcode) input through the remote input interface 210. When the authentication is successful, an OK button Q05 is output to the remote output interface 220 (remote image output device 221) together with information indicating that the authentication is successful. When the authentication is unsuccessful, information indicating that the authentication is unsuccessful is output to the remote output interface 220 (remote image output device 221). If the authentication fails, the face authentication button Q03 and the identifier authentication button Q04 may be output again to the remote output interface 220 (remote image output device 221).

[0049] Details of the authentication process executed by the remote control device 200 will be described with reference to the flowchart of Fig. 5. The authentication process may be executed steadily or periodically as background processing. Specifically, first, an operator identifier (including captured image data and / or distance measurement data for face authentication) is acquired (Fig. 5 / STEP 202). If an operator identifier based on face authentication is not acquired for a certain period of time, an operation history indicating that the operator is away from his / her seat (not seated in the seat 24) is acquired together with the last acquired operator identifier.

[0050] It is determined whether or not an operator change notification has been accepted through the remote input interface 210 (or the change notification input interface (touch button) that constitutes the remote input interface) (FIG. 5 / STEP 240). If the determination result is positive (YES in FIG. 5 / STEP 240), a first flag f1 indicating the presence or absence of a change notification is set to "1" (FIG. 5 / STEP 241). On the other hand, if the determination result is negative (NO in FIG. 5 / STEP 240), the first flag f1 is set to "0" (FIG. 5 / STEP 242).

[0051] Next, it is determined whether the operator identifier has changed (whether the operator identifier over the most recent multiple times is different from the operator identifier from the past) (FIG. 5 / STEP 250). If the determination result is positive (FIG. 5 / YES in STEP 250), a second flag f2 indicating whether the operator identifier has changed is set to "1" (FIG. 5 / STEP 251). For example, if an operator leaves his / her seat and another operator sits in the seat 24, the newly acquired operator identifier is different from the last acquired operator identifier, so the determination result is positive. On the other hand, if the determination result is negative (FIG. 5 / NO in STEP 250), the second flag f2 is set to "0" (FIG. 5 / STEP 252).

[0052] Furthermore, it is determined whether the first flag f1 and the second flag f2 are equal (FIG. 5 / STEP 254). If the determination result is negative (FIG. 5 / NO in STEP 254), that is, if an operator change report has been made but the operator identifier has not changed, or if an operator change report has not been made but the operator identifier has changed, the authentication flag f indicating the success or failure of authentication is set to "0" (FIG. 5 / STEP 262).

[0053] On the other hand, if the determination result is positive (FIG. 5 / STEP 254: YES), that is, if an operator change report is made and the operator identifier has changed, or if an operator change report is not made and the operator identifier has not changed, the success or failure of authentication is determined (FIG. 5 / STEP 260). Specifically, the success or failure of authentication is determined depending on whether the latest operator identifier is registered in the storage device constituting the remote control device 200 and / or an external database. If the determination result is positive (FIG. 5 / STEP 260: YES), the authentication flag f is set to "1" (FIG. 5 / STEP 261). On the other hand, if the determination result is negative (FIG. 5 / STEP 260: NO), the authentication flag f is set to "0" (FIG. 5 / STEP 262). A series of authentication processes is terminated by setting the authentication flag f.

[0054] The order of the process of determining whether or not an operator change notification has been received and the associated process related to the first flag f1 (see FIG. 5 / STEP 240, STEP 241, and STEP 242) and the process of determining whether or not an operator identifier has changed and the associated process related to the second flag f2 (see FIG. 5 / STEP 250, STEP 251, and STEP 252) may be reversed, or may be parallel processes. The process of determining whether or not an operator change notification has been received through the remote input interface 210 and the associated process related to the first flag f1 may be omitted (see FIG. 5 / STEP 240, STEP 241, STEP 242, and STEP 254). The process of determining whether or not an operator identifier has changed and the associated process related to the second flag f2 may be omitted (see FIG. 5 / STEP 250, STEP 251, and STEP 252). The process of comparing the first flag f1 and the second flag f2 may be omitted (see FIG. 5 / STEP 254).

[0055] FIG. 8A is a diagram showing a work content designation screen in the operation system according to this embodiment. After the operator is authenticated by the authentication process in STEP 200 in FIG. 4, when the OK button Q05 is tapped, the operation target and work content (scheduled work content) are designated or selected (FIG. 4 / STEP 204). Specifically, when the OK button Q05 is tapped, as shown in FIG. 8A, work content selection windows Q11 and Q12 and a back button Q13 are output to the remote output interface 220 (remote image output device 221). In each of the multiple (for example, two) work content selection windows Q11 and Q12, for example, the work name, the name of the office involved in the work, the work location, the characteristics of the work target (materials, etc.), and the work period are displayed. When the back button Q13 is tapped, etc., the screen shown in FIG. 7A or FIG. 7B is output to the remote output interface 220 (remote image output device 221).

[0056] FIG. 8B is a diagram showing an operation target designation screen in the operation system according to this embodiment. When one of the above-mentioned work content selection windows Q11 and Q12 is tapped or the like, as shown in FIG. 8B, operation target candidate selection windows Q14 and Q15 and a back button Q16 are output to the remote output interface 220 (remote image output device 221). For example, the model number, name and specification information of the work machine 40, which is the operation target candidate, are displayed in each of the multiple (for example, two) operation target candidate selection windows Q14 and Q15. The model number, name and specification information of the operation target candidate are associated with the actual machine identifier for identifying the actual machine wireless communication device 450 possessed by the work machine 40 (operation target), and are registered in the storage device constituting the remote control device 200 and / or an external database. Of the work machines 40 to be operated, only operation target candidates that have been switched by the switch to a state in which power is being supplied to the actual machine control device 400 can be selected, and operation target candidates that have not been switched to a state in which power is being supplied are displayed in a grayed-out state, and windows Q14 and Q15 are output to the remote output interface 220 (remote image output device 221) so as not to accept selection even if they are tapped, etc. When the back button Q16 is tapped, etc., the screen shown in Fig. 8A is output to the remote output interface 220 (remote image output device 221).

[0057] Depending on the operator authenticated by executing the authentication process (see FIG. 4 / STEP 200), a plurality of operation target candidates may be searched for from the storage device constituting the remote control device 200 and / or an external database, and a plurality of operation target candidate selection windows showing the model numbers and the like of the plurality of operation target candidates may be output to the remote output interface 220 (remote image output device 221). As a result, a new operation target may be designated by the operator through the remote input interface 210 according to his / her intention from among the operation target candidates output to the remote output interface 220 (remote image output device 221) according to the operator. Note that the operation target candidates to be used by each operator according to the operator identified by the operator identifier are registered in advance in the storage device and / or the external database.

[0058] A plurality of operation target candidates may be searched for in the storage device constituting the remote control device 200 and / or an external database according to the type of work content (scheduled work content) selected from the plurality of work content selection windows, and a plurality of operation target candidate selection windows showing the model numbers and the like of the plurality of operation target candidates may be output to the remote output interface 220 (remote image output device 221). As a result, a new operation target may be designated by the operator through the remote input interface 210 according to his / her intention from among the operation target candidates output to the remote output interface 220 (remote image output device 221) according to the type of the planned work content. Note that the storage device and / or the external database have preregistered therein the planned work content of each operator and the operation target candidates to be used when executing the planned work content according to the type of operator identified by the operator identifier.

[0059] Fig. 8C is a diagram showing a confirmation screen for work content and operation target in the operation system according to this embodiment. When one of the operation target selection windows Q14 and Q15 is tapped or the like, a selection result confirmation window Q17, a back button Q18, and an operation start button Q19 are output to the remote output interface 220 (remote image output device 221) as shown in Fig. 8C. For example, the selected work content and information on the work machine 40 that is the operation target are displayed in the selection result confirmation window Q17. When the back button Q18 is tapped or the like, the screen shown in Fig. 8B is output to the remote output interface 220 (remote image output device 221).

[0060] Next, an operation to start using the operation target is accepted (FIG. 4 / STEP 206). Specifically, the operation start button Q19 shown in FIG. 8C is tapped or the like.

[0061] In response to this, it is determined whether there is an existing operation target for the remote control device 200 (FIG. 4 / STEP 208). Specifically, it is determined whether there is a work machine 40 that is an operation target and in a state in which communication is established with the remote operation device 20. When communication with the operation target is established and / or when the established communication is released, the value of the flag may be changed, and the value of the flag may be used for this determination.

[0062] If it is determined that there is no existing operation target (FIG. 4 / STEP 208: NO), the remote control device 200 starts the execution of a communication establishment process as an "operation preparation process" (FIG. 4 / STEP 210). First, the actual machine identifier corresponding to the work machine 40 selected as the operation target by the designation command is obtained from the actual machine identifiers associated with each of a plurality of operation target candidates registered in the storage device and / or an external database.

[0063] Next, the remote control device 200 transmits a communication establishment request signal to a destination identified and indicated by the actual machine identifier corresponding to the actual wireless communication device 450 of the work machine 40 selected as the operation target, requesting the establishment of communication with the work machine 40 that is the operation target. The communication establishment request signal includes the operation device identifier corresponding to the remote wireless communication device 250 possessed by the remote operation device 20.

[0064] When the actual machine control device 400 mounted on the work machine 40 to be operated receives the communication establishment request signal through the actual machine wireless communication device 450, it transmits an approval signal indicating that the request for communication establishment has been approved to the remote wireless communication device 250 possessed by the remote operation device 20, which is the destination identified and indicated by the operation device identifier contained in the communication establishment request signal.

[0065] When the remote control device 200 receives the approval signal through the remote wireless communication device 250, it transmits an activation signal to activate the actual machine imaging device 412 and the actual machine sensor group 416 mounted on the work machine 40 to the destination identified and indicated by the actual machine identifier, and waits until it receives a signal corresponding to the actual machine operating status regarding the image acquired by the actual machine imaging device 412 through the remote wireless communication device 250 and the status of the work machine 40 acquired by the actual machine sensor group 416.

[0066] When the real machine control device 400 receives the activation signal through the real machine wireless communication device 450, it activates the real machine imaging device 412 and the real machine sensor group 416, starts transmitting signals corresponding to the real machine operation status relating to the captured images acquired by the real machine imaging device 412 and the state of the work machine 40 acquired by the real machine sensor group 416 to a destination identified and indicated by the operation device identifier, and starts receiving the signals corresponding to the operation mode of the remote control lever for controlling the operation of the actuation mechanism through the real machine wireless communication device 450. Note that the signals corresponding to the real machine operation status contain information related to the real machine identifier corresponding to the real machine wireless communication device 450.

[0067] The remote control device 200 outputs the captured images and the actual machine operation status received through the remote wireless communication device 250 to the remote output interface 220. For example, the remote control device 200 outputs the received captured images to a remote image output device 221, outputs the sound of the work machine 40 included in the received actual machine operation status to a remote sound output device 222, and controls the operation of the seat drive device 224 according to the inclination angle included in the received actual machine operation status.

[0068] As described above, the actual machine control device 400 executes a process for controlling at least the communication device and establishing the communication with the operation target as an operation preparation process. By executing the communication establishment process, communication is established between the remote wireless communication device 250 of the work machine 40 selected as the operation target and the remote wireless communication device 250 for remotely operating the work machine 40 (operation target) by the remote operation device 20.

[0069] If there is a plan to perform communication establishment processing between the work machine 40 to be operated and the remote operation device 20, the remote control device 200 may transmit a communication establishment request signal before the time to execute the communication establishment processing. In this case, the communication establishment request signal includes the operation device identifier corresponding to the remote wireless communication device 250 possessed by the remote operation device 20 and information regarding the time to establish the communication establishment processing. The actual machine control device 400 mounted on the work machine 40 to be operated stores the operation device identifier contained in the communication establishment request signal received through the remote wireless communication device 250 in a storage device mounted on the work machine 40, reads out the operation device identifier from the storage device when it is time to establish the communication establishment processing, and transmits the approval signal indicating that the request for communication establishment has been approved to the remote wireless communication device 250 possessed by the remote operation device 20 that is the transmission destination identified and indicated by the read out operation device identifier.

[0070] Further, the operating device identifier and the actual machine identifier may be assigned something other than an IP address as long as they can identify the remote wireless communication device 250 and the actual machine wireless communication device 450 in the second communication network. For example, a wireless LAN access point or the like may be assigned to the remote wireless communication device 250 and the actual machine wireless communication device 450, such as an SSID (Service Set Identifier). The operating device identifier and the actual machine identifier may be configured to correspond to, for example, a plurality of letters, numbers, and / or symbols representing the serial numbers of components such as an engine and / or a battery mounted on the work machine 40 together with the actual machine wireless communication device 450, so that the remote wireless communication device 250 and the actual machine wireless communication device 450 can be identified in the second communication network. In communication, an authenticator is generated according to a designation method based on the operating device identifier and the actual machine identifier. An authenticator may be generated according to a different designation method depending on the difference in the data or letters, numbers, and symbols constituting the operating device identifier and the actual machine identifier. Then, the remote control device 200 uses the generated authenticator to establish wireless communication with the remote wireless communication device 250 and the actual wireless communication device 450 identified by the communication identifier.

[0071] On the other hand, if it is determined that there is an existing operation target for the remote control device 200 (YES in FIG. 4 / STEP 208), the remote control device 200 starts executing a communication disconnection process as an "operation preparation process" in addition to the communication establishment process (FIG. 4 / STEP 212).

[0072] In the communication establishment process in the operation preparation process, the remote control device 200 first acquires the actual machine identifier corresponding to the work machine 40 (other operation target) selected as the operation target by the designation command from the actual machine identifiers associated with each of the multiple operation target candidates registered in the storage device and / or external database.

[0073] Next, the remote control device 200 transmits a communication establishment request signal to a destination identified and indicated by the actual machine identifier corresponding to the actual machine wireless communication device 450 of the work machine 40 selected as the target to be operated, requesting the establishment of communication with the work machine 40 to be operated.

[0074] When the actual machine control device 400 mounted on the work machine 40 to be operated receives the communication establishment request signal through the actual machine wireless communication device 450, it transmits an approval signal indicating that the request for communication establishment has been approved to the remote wireless communication device 250 possessed by the remote operation device 20, which is the destination identified and indicated by the operation device identifier contained in the communication establishment request signal.

[0075] When the remote control device 200 receives the approval signal through the remote wireless communication device 250, it transmits an activation signal to activate the actual machine imaging device 412 and the actual machine sensor group 416 mounted on the work machine 40 to the destination identified and indicated by the actual machine identifier, and waits until it receives a signal corresponding to the actual machine operating status regarding the image acquired by the actual machine imaging device 412 through the remote wireless communication device 250 and the status of the work machine 40 acquired by the actual machine sensor group 416.

[0076] When the real machine control device 400 receives the activation signal through the real machine wireless communication device 450, it activates the real machine imaging device 412 and the real machine sensor group 416, starts transmitting signals corresponding to the real machine operation status relating to the captured images acquired by the real machine imaging device 412 and the state of the work machine 40 acquired by the real machine sensor group 416 to a destination identified and indicated by the operation device identifier, and starts receiving the signals corresponding to the operation mode of the remote control lever for controlling the operation of the actuation mechanism through the real machine wireless communication device 450. Note that the signals corresponding to the real machine operation status contain information related to the real machine identifier corresponding to the real machine wireless communication device 450.

[0077] The remote control device 200 outputs the captured images and the actual machine operation status received through the remote wireless communication device 250 to the remote output interface 220 .

[0078] As described above, as the communication establishment process in the operation preparation process, the remote control device 200 executes the process of establishing the communication with the operation target through the communication device.

[0079] In the communication disconnection processing in the operation preparation processing, the remote control device 200 first stops outputting to the remote output interface 220 captured images and actual operating status of the work machine 40 (one work machine) that is the existing operation target, which are received through the remote wireless communication device 250, and further stops transmitting to the work machine 40 (one work machine) a signal corresponding to the operation mode of the remote operation lever to the work machine 40 (one work machine).

[0080] Next, the remote control device 200 transmits a communication disconnection request signal to a destination identified and indicated by the actual machine identifier corresponding to the actual machine wireless communication device 450 of the work machine 40 that is the existing target of operation, requesting that the established communication be disconnected.

[0081] When the actual machine control device 400 mounted on the existing work machine 40 to be operated receives the communication disconnection request signal via the actual machine wireless communication device 450, it executes processing to turn off the power to the actual machine imaging device 412 and the actual machine sensor group 416 that have been activated. Then, the actual machine control device 400 executes processing to cut off the power supply being supplied to the actual machine control device 400.

[0082] As described above, the real machine control device 400 executes a communication disconnection process as an operation preparation process to cancel the established communication state between the remote operation device 20 and the work machine 40 in the established communication state. Note that, during this communication disconnection process, the energized state to the real machine control device 400 mounted on the work machine 40 may be maintained (processing to cut off the power supply to the real machine control device 400 may not be performed). This allows the real machine control device 400 to receive the signal according to the operation mode of the remote operation lever for controlling the operation of the operating mechanism through the real machine wireless communication device 450, and when the real machine imaging device 412 and the real machine sensor group 416 are started again, a state in which it is possible to transmit the captured image acquired by the real machine imaging device 412 and the signal according to the real machine operating status regarding the state of the work machine 40 acquired by the real machine sensor group 416 to the remote operation device 20 through the real machine wireless communication device 450 may be maintained.

[0083] The communication establishment process and the communication disconnection process may be processed in parallel or in parallel. When the communication establishment process and the communication disconnection process are processed in parallel or in parallel, the remote control device 200 performs the process of acquiring the actual machine identifier corresponding to the work machine 40 (another operation target), while simultaneously starting the process of stopping the output of the captured image and the actual machine operation status of the work machine 40 (one work machine) to the remote output interface 220 and the process of stopping the transmission to the work machine 40 (one work machine) of a signal corresponding to the operation mode of the remote operation lever to the work machine 40 (one work machine). Then, the remote control device 200 proceeds with the communication establishment process and the communication disconnection process by performing the respective processes in parallel or in parallel.

[0084] Here, either the process of turning off the power supply of the real machine imaging device 412 and the real machine sensor group 416 of the work machine 40 (one work machine) that is an existing operation target or the process of starting up the real machine imaging device 412 and the real machine sensor group 416 of the work machine 40 (another operation target) may be executed first. That is, the remote control device 200 may be temporarily in a state of receiving captured images and real machine operation status from both the work machine 40 (one work machine) that is an existing operation target and the work machine 40 (another operation target). In this case, the remote control device 200 selects the captured image and the real machine operation status of the work machine 40 (another operation target) by reading the real machine identifier of the transmission source attached to the signal indicating the received captured image and real machine operation status, and outputs it to the remote output interface 220. In addition, the start time of the communication establishment process and the start time of the communication disconnection process may be the same or different. The end time of the communication establishment process and the end time of the communication disconnection process may be the same or different. The communication establishment process and the communication disconnection process may be serially processed (executed sequentially).

[0085] After the operation preparation process starts, a designated item setting process is executed to set the contents of the designated items for supporting the operation of the work machine 40, which is the operation target by the remote operation device 20 (FIG. 4 / STEP 214). If it is determined that there is no existing operation target (NO in FIG. 4 / STEP 208), the designated item setting process is executed in parallel or in parallel with the execution of the communication establishment process (FIG. 4 / STEP 210) as the operation preparation process, and is started, for example, at the same time as the execution of the communication establishment process starts. If it is determined that there is an existing operation target (YES in FIG. 4 / STEP 208), the designated item setting process is executed in parallel or in parallel with the execution of the communication establishment process and the communication disconnection process (FIG. 4 / STEP 212) as the operation preparation process, and is started, for example, at the same time as the execution of the communication establishment process and the communication disconnection process starts. Next, the remote control device 200 determines whether the operation preparation process is completed (FIG. 4 / STEP 216). If it is determined that the operation preparation process has not been completed (NO in FIG. 4 / STEP 216), the designated item setting process continues to be executed (FIG. 4 / STEP 214).

[0086] The designated item setting process will be described in detail below. Fig. 9A is a diagram showing a first lever pattern setting screen in the operation system according to this embodiment. Fig. 9B is a diagram showing a second lever pattern setting screen in the operation system according to this embodiment. In the designated item setting process, first, for example, a menu screen for selecting multiple designated items is output to the remote output interface 220 (remote image output device 221). When "lever pattern" is selected as a designated item on the menu screen, the first lever pattern setting screen shown in Fig. 9A is output to the remote output interface 220 (remote image output device 221).

[0087] As shown in FIG. 9A, the first lever pattern setting screen includes a first operation guide window Q21, a lever pattern window Q22, an OK button Q23, and a change button Q24. The first operation guide window Q21 displays information guiding the operation method for setting the lever pattern. The lever pattern window Q22 displays a provisionally set lever pattern. When the OK button Q23 is tapped or the like by the operator, the lever pattern displayed in the lever pattern window Q22 is set for real. When the change button Q24 is tapped or the like by the operator, the second lever pattern setting screen shown in FIG. 9B is output to the remote output interface 220 (remote image output device 221).

[0088] The finally set lever pattern may be associated with an operator identifier (including image and distance measurement data for face authentication) and / or an actual machine identifier and stored in a storage device and / or a database constituting the remote control device 200. The operator identifier used in the authentication process and / or the actual machine identifier used in the designation of the operation target may be used to search or read out the last finally set lever pattern from the storage device, and then displayed as a default in the lever pattern window Q22 as a provisionally set lever pattern. The finally set lever pattern may simply be stored in a storage device and / or an external database constituting the remote control device 200, and then read out from the storage device or the like, and then displayed as a default in the lever pattern window Q22 as a provisionally set lever pattern.

[0089] As shown in FIG. 9B, the second lever pattern setting screen includes a second operation guide window Q25 and a plurality of (for example, four) lever pattern candidate windows Q26, Q27, Q28, and Q29. The second operation guide window Q21 displays information for guiding the selection of one of the lever pattern candidate windows Q26, Q27, Q28, and Q29 in order to set a lever pattern. Each of the lever pattern windows Q26, Q27, Q28, and Q29 displays a lever pattern candidate different from the provisionally set lever pattern. In response to a selective tap or the like on one of the lever pattern candidate windows Q26, Q27, Q28, and Q29, the first lever pattern setting screen is output to the remote output interface 220 (remote image output device 221) (see FIG. 9A). At this time, the selected lever pattern candidate is displayed in the lever pattern window Q22 as a provisionally set lever pattern.

[0090] The first lever pattern setting screen and the second lever pattern setting screen may be output to the remote output interface 220 (remote image output device 221) as a single screen or as three or more transitional screens.

[0091] FIG. 10 is a diagram showing a screen for setting the lever sensitivity in the operation system according to this embodiment. When "lever sensitivity" is selected as a designated item on the menu screen, the lever sensitivity setting screen shown in FIG. 10 is output to the remote output interface 220 (remote image output device 221). As shown in FIG. 10, the lever sensitivity setting screen includes an operation guide window Q31, a lever sensitivity adjustment bar Q32, and an OK button Q33. The operation guide window Q31 displays information that guides the operation method for setting the lever sensitivity. The lever sensitivity adjustment bar Q32 includes a measure indicating the lever sensitivity (which determines the amount of displacement of the operation target according to the amount of lever operation or the speed of displacement) and, for example, a triangular slider, and the lever sensitivity is provisionally set by adjusting the position of the slider along the measure. When the OK button Q33 is tapped, the lever sensitivity displayed on the lever sensitivity adjustment bar Q32 is set for real.

[0092] The finally set lever sensitivity may be associated with the operator identifier and / or the actual machine identifier and stored in a storage device constituting the remote control device 200 and / or an external database. The operator identifier used in the authentication process and / or the actual machine identifier used in the designation of the operation target may be used to search or read out the last finally set lever sensitivity from the storage device as a provisionally set lever sensitivity, and the default position of the slider in the lever sensitivity adjustment bar Q32 may be adjusted. The finally set lever sensitivity may simply be stored in a storage device constituting the remote control device 200 and / or an external database, and the default position of the slider in the lever sensitivity adjustment bar Q32 may be adjusted after being read out from the storage device or the like.

[0093] FIG. 11 is a diagram showing a screen for setting the seat vibration mode in the operation system according to this embodiment. When "seat vibration" is selected as a designated item on the menu screen, the seat vibration setting screen shown in FIG. 11 is output to the remote output interface 220 (remote image output device 221). As shown in FIG. 11, the seat vibration setting screen includes an operation guide window Q41, a seat vibration strength selection window Q42, and an OK button Q43. The operation guide window Q41 displays information that guides an operation method for setting the strength of vibration (including the presence or absence of vibration) of the seat 24 (or the seat device). The seat vibration strength selection window Q42 includes a plurality of buttons, and the strength of the vibration of the seat device is provisionally set by selectively tapping one of the plurality of buttons. When the OK button Q43 is tapped, the strength of the vibration of the seat device corresponding to the one button is set.

[0094] The actually set vibration intensity may be associated with the operator identifier and / or the actual machine identifier and stored in a storage device constituting the remote control device 200 and / or an external database. The operator identifier used in the authentication process and / or the actual machine identifier used in the designation of the operation target may be used to search or read out the last actually set vibration intensity from the storage device, and then displayed as a provisional vibration intensity by default in the seat vibration intensity selection window Q42. The actually set vibration intensity may simply be stored in a storage device constituting the remote control device 200 and / or an external database, and then read out from the storage device or the like, and then displayed as a provisional vibration intensity by default in the seat vibration intensity selection window Q42.

[0095] FIG. 12 is a diagram showing a screen for setting the operation assist in the operation system according to this embodiment. When "operation assist" is selected as a designated item on the menu screen, the operation assist setting screen shown in FIG. 12 is output to the remote output interface 220 (remote image output device 221). As shown in FIG. 12, the operation assist setting screen includes an operation guide window Q51, an operation assist window Q52, and an OK / no button Q53. The operation guide window Q51 displays information that guides an operation method for setting the presence or absence of operation assist. The operation assist window Q52 includes an ON button and an OFF button for provisionally setting the presence or absence of operation assist or ON / OFF. When the OK button Q53 is tapped, the operation assist is set for real. When the operation assist is set for real to ON, the operation content assisted by the operation assist may be output to the remote output interface 220 (remote image output device 221). The operation content is, for example, a horizontal pulling operation in which the bucket, which is the tip attachment 445, is moved linearly along the ground on which the lower running body 41 of the work machine 40 is in contact, and an operation in which the bucket is moved from a first position away from the upper rotating body 42 in the forward direction to a second position closer to the upper rotating body 42 than the first position. There may be multiple types of operation content assisted by the operation assist, and the configuration may be such that the presence or absence of operation assist can be set for the multiple types of operation content.

[0096] Also, a simulation image showing an operating situation in which the work machine 40 is assisted by the operation assist may be output to the remote output interface 220 (remote image output device 221). The remote control device 200 executes a process of recognizing an operating mode of the remote input interface 210 (remote operation mechanism 211) by the operator, a process of correcting the recognized operating mode so as to approach a target operating mode, a process of recognizing a motion mode of the work machine 40 in the virtual space based on the corrected operating mode, and a process of outputting an image that simulates a state in which a change in the attitude of the work machine 40 is reflected according to the recognized motion mode to the remote output interface 220 (remote image output device 221) as the simulation image. For example, when the OK button Q53 is tapped, etc., a simulation image showing an operating situation in which the work machine 40, which is the operation target in the virtual space, is assisted by the operation assist is output to the remote output interface 220 (remote image output device 221). Then, when the operator operates the work machine 40 in the virtual space through the remote input interface 210 (remote operation mechanism 211), the work machine 40 moves in accordance with the operation mode of the remote input interface 210 (remote operation mechanism 211) in the simulation image based on the operation mode of the remote input interface 210 (remote operation mechanism 211), and the position and attitude of the tip attachment 445 and the like are shown to change. Therefore, the remote control device 200, the remote input interface 210 (remote operation mechanism 211) and the remote output interface 220 (remote image output device 221) constitute a "simulation device." Here, when the operator operates the bucket through the remote input interface 210 (remote operation mechanism 211) to move the bucket closer to a second position that is closer to the upper rotating body 42 than the first position in the virtual space, the simulation image shows the tip of the bucket moving linearly from the first position to the second position.Specifically, when the operator performs an operation of rotating the boom 441 upward while rotating the arm 443 downward and further rotating the bucket tip so that it faces downward through the remote input interface 210 (remote operation mechanism 211), the operation is corrected (the deviation amount of the bucket tip from the target trajectory is reduced) so that the bucket tip does not deviate from the target trajectory that linearly connects the first position to the second position by more than a predetermined distance, and is reflected in the simulation image. The first position, the second position, and the target trajectory may be visualized and shown in the simulation image, and the first position and the second position may be set to any position by the operator's operation (for example, by moving the bucket tip to a position to be the first position or the second position and pressing a decision button to set the position). In addition, after communication is established between the remote operation device 20 and the work machine 40 to be operated and the remote operation of the work machine 40 by the remote operation device 20 becomes possible, a remote operation simulation of the work machine 40 in the virtual space may be permitted in a state in which the remote operation of the work machine 40 is stopped or temporarily stopped.

[0097] FIG. 13 is a diagram showing a guidance setting screen in the operation system according to this embodiment. When "Guidance (operation support image)" is selected as a designated item on the menu screen, the operation support image setting screen shown in FIG. 13 is output to the remote output interface 220 (remote image output device 221). As shown in FIG. 13, the operation support image setting screen includes an operation guide window Q61, a depth sense image window Q62, a terrain image window Q63, and an OK button Q64. The operation guide window Q61 displays information that guides an operation method for setting the display mode of the operation support image in the remote output interface 220 (remote image output device 221). The depth sense image window Q62 includes an ON button and an OFF button for provisionally setting the presence or absence or ON / OFF of the display of the depth sense image in the remote output interface 220 (remote image output device 221). The terrain image window Q63 includes an ON button and an OFF button for provisionally setting the presence or absence or ON / OFF of the display of the terrain image in the remote output interface 220 (remote image output device 221). When the OK button Q64 is tapped or the like, the display or non-display of the depth sense image and the display or non-display of the terrain image are set.

[0098] The set information on whether or not to display the depth sense image and / or the terrain image may be stored in a storage device constituting the remote control device 200 and / or an external database in association with an operator identifier and / or an actual machine identifier. The operator identifier used in the authentication process and / or the actual machine identifier used in the designation of the operation target may be used to search or read out the last set information on whether or not to display (ON / OFF) from the storage device, and then displayed as a default in the depth sense image window Q62 and / or the terrain image window Q63. The set information on whether or not to display the depth sense image and / or the terrain image may simply be stored in a storage device constituting the remote control device 200 and / or an external database, and then read out from the storage device or the like, and then displayed as a default in the depth sense image window Q62 and / or the terrain image window Q63.

[0099] Also, a simulation image showing a depth sense image and / or a terrain image may be output to the remote output interface 220 (remote image output device 221). For example, when the display of the depth sense image is set to ON, a simulation image showing an operating situation assisted by the operation assist in the work machine 40, which is the operation target in the virtual space, is output to the remote output interface 220 (remote image output device 221). Then, when the operator operates the work machine 40 in the virtual space through the remote input interface 210 (remote operation mechanism 211), the simulation image shows how the work machine 40 operates according to the operation mode, and the position and attitude of the tip attachment 445 and the like change. For this reason, the remote control device 200, the remote input interface 210 (remote operation mechanism 211) and the remote output interface 220 (remote image output device 221) constitute a "simulation device." Here, in the simulation image showing the operating situation of the work machine 40, a depth sense image for showing the position or depth sense of one or more locations defined in the work mechanism 44 is displayed. Fig. 14 is a diagram showing a depth perception image in the operation system according to this embodiment. For example, as shown in Fig. 14, a mark M is displayed on the simulation image. A mark M is displayed indicating the position of a second specified point p2 as a result of vertically projecting a first specified point p1, which is the left-right center point of the base end of the tip attachment 445. The first specified point p1 may be specified as a point other than the left-right center point of the base end of the tip attachment 445 (such as the tip of the tip attachment 445). The second specified point p2 may be displayed at a predetermined distance from the first specified point p1 in the virtual space, or may be displayed at any height in the virtual space.

[0100] Fig. 15 is a diagram showing a terrain image in the operation system according to this embodiment. When the display of the terrain image is set to ON, a terrain image showing the terrain around (forward of) the work machine 40 is superimposed on a simulation image showing the operating status of the work machine 40. This causes, for example, as shown in Fig. 15, a mesh-like terrain image showing the terrain or unevenness of the ground surface to be displayed in the simulation image.

[0101] Fig. 16A is a diagram showing a first button function setting screen in the operation system according to this embodiment. When "button assignment" is selected as a specification item on the menu screen, the first button function setting screen shown in Fig. 16A is output to the remote output interface 220 (remote image output device 221). As shown in Fig. 16A, the first button function setting screen includes an operation guide window Q71, a button layout window Q72, a function assignment status window Q73, and an OK button Q74. The operation guide window Q71 displays information that guides the user on how to assign functions to multiple buttons that make up the remote operation mechanism 211. Functions that can be assigned to the buttons include a function for adjusting the angle of the real machine imaging device 412 in the left-right direction, a function for adjusting the angle of the real machine imaging device 412 in the up-down direction, a zoom-in function and a zoom-out function for the real machine imaging device 412, a function for switching on / off an audio output device (for example, a horn that issues an alarm sound to the surroundings) mounted on the work machine 40, a setting function such as calibration of the operation lever that constitutes the remote operation mechanism 211, a function for switching between increasing and decreasing the pressure of the hydraulic oil for the work mechanism 44, a function for switching between exciting and releasing the lifting magnet as the end attachment 445, and a function for canceling the temporary stop state of the work machine 40. The button layout window Q72 shows the layout of multiple buttons to which functions can be assigned. For example, the button layout window Q72 shows an arrangement of a total of ten buttons, with buttons L1, L2, and L3 arranged on the left travel lever 2110, buttons L4 and L5 arranged on the left operation lever 2111, buttons R1, R2, and R3 arranged on the right travel lever 2110, and buttons R4 and R5 arranged on the right operation lever 2112. The function assignment status window Q73 shows a chart showing the functions assigned to each of the multiple buttons. When the OK button Q74 is tapped, etc., the functions assigned to each button shown in the function assignment status window Q73 are finally set.

[0102] The functions assigned to each button in the actual setting may be associated with the operator identifier and / or the actual device identifier and stored in a storage device constituting the remote control device 200 and / or an external database. The operator identifier used in the authentication process and / or the actual device identifier used in the operation target designation may be used to search or read out the last function assigned to each button from the storage device, and then displayed as a default in the function assignment status window Q73. The functions assigned to each button in the actual setting may simply be stored in a storage device constituting the remote control device 200 and / or an external database, and then read out from the storage device or the like and displayed as a default in the function assignment status window Q73.

[0103] FIG. 16B is a diagram showing a second button function setting screen in the operation system according to this embodiment. When one of the buttons shown in the button layout window Q72 (or the function assignment status window Q73) is selectively tapped, the second button function setting screen shown in FIG. 16B is output to the remote output interface 220 (remote image output device 221). As shown in FIG. 16B, the second button function setting screen includes a button window Q75, a button function window Q76, a button function selection window Q77, and an OK button Q78. The button window Q75 displays (highlights as appropriate) the name and / or position of one button selected from the multiple buttons. The button function window Q76 displays the function assigned to the one button. The button function selection window Q77 displays a list of functions that can be assigned to the one button (in the form of a pull-down menu, etc.). When the OK button Q78 is tapped or the like, a first button function setting screen is output to the remote output interface 220 (remote image output device 221) (see FIG. 16A).

[0104] In addition, when the work machine 40 is equipped with multiple actual machine imaging devices 412 with different imaging ranges (which may partially overlap), "screen allocation" may be selected as a specification item on the menu screen, and a correspondence may be set between each of the multiple image output devices (displays or monitors) that make up the remote image output device 221 and the captured images obtained through each of the multiple actual machine imaging devices 412 displayed on each image output device.

[0105] As described above, various designated item setting processes are executed. If it is determined that the operation preparation process being executed has been completed (FIG. 4 / STEP 216: YES), the remote control device 200 in the remote operation device 20 executes remote operation process for remotely operating the work machine 40 (FIG. 4 / STEP 218). If the designated item setting process is in progress for a certain designated item when the operation preparation process is completed, the designated item setting process may be executed continuously (FIG. 4 / STEP 214).

[0106] On the condition that communication for remotely operating a work machine 40 (operation target) by the remote operation device 20 has been established, the actual machine control device 400 in the work machine 40 transmits a captured image acquired via the actual machine imaging device 412 to the remote operation device 20. The transmitted captured image (or captured image data) may be not only the captured image itself, but also image data representing a simulated environmental image generated based on the captured image.

[0107] For example, in the remote operation device 20, the remote control device 200 may establish communication for remotely operating the work machine 40 (operation target) in response to transmitting a designation signal to the work machine 40 via the remote wireless communication device 250. When a plurality of work machines 40 are a plurality of operation target candidates, a designation signal may be transmitted to each of the plurality of work machines 40. The presence or absence of a designation operation via the remote input interface 210 may be determined by the operator, and when the determination result is positive, a designation signal may be transmitted. The "designation operation" is, for example, an operation such as tapping on the remote input interface 210 for the operator to designate the work machine 40 that he or she intends to remotely operate.

[0108] When the remote operation device 20 receives a captured image through the remote wireless communication device 250, the captured image or the work environment image is output to the remote image output device 221 (display device) (FIG. 6 / STEP 280). As a result, for example, a captured image showing the ground spreading out in front of the cab 43, a bucket corresponding to the tip attachment 445 that is a part of the work mechanism 44, the ground, and the like is output to the remote image output device 221.

[0109] When the designated item "button assignment" is selected in the designated item setting process and the function assignment of each button is set, the assigned function is applied. As a result, the range and / or scale of the captured image can be adjusted by the operator operating the button to which the angle adjustment and / or zoom-in / zoom-out functions of the real imaging device 412 are assigned (see FIG. 4 / STEP 214, FIG. 16A, and FIG. 16B).

[0110] When the designated item "guidance" is selected in the designated item setting process and the display of the depth image is set to "Yes (ON)", a mark M may be superimposed on the captured image as a depth image (see FIG. 4 / STEP 214, FIG. 13, and FIG. 14). For example, the remote control device 200 calculates the position coordinates of a first designated point p1, which is the left-right center point of the base end of the tip attachment 445, from the hoisting angle of the boom 441, the rotation angle of the arm 443, and the rotation angle of the tip attachment 445, and displays the mark M superimposed on the position of a second designated point p2 obtained by projecting the position coordinates vertically downward. When the designated item "guidance" is selected in the designated item setting process and the display of the terrain image is set to "Yes (ON)", a mesh may be superimposed on the captured image as a terrain image (see FIG. 4 / STEP 214, FIG. 13, and FIG. 15). The mesh to be superimposed is generated, for example, based on the distance from the work machine 40 to the surrounding ground surface obtained by a sensing device mounted on the work machine 40 that measures the distance to surrounding objects.

[0111] Next, the remote control device 200 recognizes the operation mode of the remote operation mechanism 211 (FIG. 6 / STEP 281). Specifically, when the designated item "lever pattern" is selected in the designated item setting process and the lever pattern is set, the operation mode of the remote operation mechanism 211 or the operating levers 2110, 2111, and 2112 constituting the remote operation mechanism 211 is recognized according to the lever pattern (see FIG. 4 / STEP 214, FIG. 9A, and FIG. 9B). When the designated item "lever sensitivity" is selected in the designated item setting process and the lever sensitivity is set, the operation mode of the remote operation mechanism 211 or the operating levers 2110, 2111, and 2112 constituting the remote operation mechanism 211 is recognized according to the lever sensitivity (see FIG. 4 / STEP 214 and FIG. 10).

[0112] Next, the remote control device 200 judges whether the authentication flag f (see FIG. 5 / STEP 261 and STEP 262) is 1 or not (FIG. 6 / STEP 282). When the authentication flag f is set to "1" in response to the success of the authentication process of the operator (YES in FIG. 6 / STEP 282 (YES in FIG. 5 / STEP 260 → see STEP 261)), a remote operation command corresponding to the operation mode is transmitted to the work machine 40 through the remote wireless communication device 250 (FIG. 6 / STEP 284). When the authentication flag f is set to "0" in response to the failure of the authentication process of the operator (NO in FIG. 6 / STEP 282 (NO in FIG. 5 / STEP 254, NO in STEP 260 → see STEP 262)), the transmission of the remote operation command corresponding to the operation mode to the work machine 40 through the remote wireless communication device 250 is suspended or only a part of the remote operation command is transmitted (FIG. 6 / STEP 286).

[0113] Then, a time series of the operation modes of the remote operation mechanism 211 by the operator is sequentially acquired as an operation history of the work machine 40 in association with the operator identifier and / or the actual machine identifier, and is stored and held in a storage device constituting the remote control device 200 and / or an external database (FIG. 6 / STEP 288). If the execution of remote operation processing for remotely operating the work machine 40 by the remote control device 200 (FIG. 4 / STEP 218) is continued, the processing of the operation processing function (FIG. 6 / STEP 280 → . . . → FIG. 6 / STEP 288) is repeated.

[0114] In the work machine 40, when the actual machine control device 400 receives a remote operation command via the actual machine wireless communication device 450, the operation of the work mechanism 44 and the like is controlled in accordance with the remote operation command. For example, while holding an object such as soil or sand with a bucket as the tip attachment 445, a predetermined task is performed involving the operation of the work mechanism 44 and the like in order to displace the real space position of the bucket.

[0115] When the designated item "operation assist" is selected in the designated item setting process and the operation assist is set to "ON", specific operation content is assisted based on the setting. For example, when a horizontal pulling operation is assisted based on the setting, when the operator operates the bucket to approach a second position closer to the upper rotating body 42 of the work machine 40 (operation target) from a first position away from the upper rotating body 42 in the forward direction through the remote input interface 210 (remote operation mechanism 211), the tip of the bucket moves linearly from the first position to the second position in the work machine 40, and a captured image or a work environment image capturing this state is output to the remote image output device 221 (display device). Specifically, when an operation of rotating the boom 441 upward while rotating the arm 443 downward and further rotating the bucket tip so that it faces downward is performed by the operator through the remote input interface 210 (remote operation mechanism 211), the operation is corrected (the deviation of the bucket tip from the target trajectory is reduced) so that the bucket tip does not deviate by more than a predetermined distance from a target trajectory that linearly connects the first position to the second position, and is reflected in the operation of the work mechanism 44. The first position, the second position, and the target trajectory may be superimposed on a captured image or a work environment image and output to the remote image output device 221 (display device), and the first position and the second position may be set to any position by the operator's operation (for example, the position can be set by moving the bucket tip to a position desired to be the first position or the second position and pressing a decision button).

[0116] In the work machine 40, the actual machine control device 400 transmits the actual machine operation status acquired through the actual machine sensor group 416 to the remote operation device 20. The actual machine operation status includes the actual machine attitude (tilt angle around the pitch axis and / or roll axis with respect to the vertical direction of the upper rotating body 42), whether the prime mover is operating, the attitude of the work mechanism 44, etc. When the remote operation device 20 receives the actual machine operation status through the remote wireless communication device 250, the remote control device 200 controls the operation of the seat drive device 224 to adjust the attitude of the seat 24 so that the seat attitude is aligned or coincident with the actual machine attitude. At this time, among the frequency components representing the time-varying manner of the actual machine attitude, high-frequency components exceeding a threshold value may be cut, and the time-varying manner of the seat attitude may be adjusted according to only low-frequency components below the threshold value.

[0117] When the remote control device 200 determines, based on the actual machine operating status, that the prime mover of the work machine 40 is operating, the seat 24 may be vibrated by the seat drive device 224. The seat 24 vibrates around the pitch axis and / or the roll axis with a predetermined minute amplitude and at a frequency corresponding to the frequency of the prime mover, so that the attitude of the seat 24 changes slightly with respect to the designated attitude. Alternatively, the seat 24 vibrates in a direction parallel to at least one of the pitch axis, the roll axis, and the yaw axis with a predetermined minute amplitude and at a frequency corresponding to the frequency of the prime mover, so that the attitude of the seat 24 changes slightly in at least one of the forward / backward, left / right, and up / down directions while being maintained in the designated attitude.

[0118] When the designated item "seat vibration" is selected in the designated item setting process and the vibration intensity of the seat 24 (or seat device) is set, the operating speed and amplitude of the seat drive device 224, and ultimately the speed and amount of change in the posture of the seat 24, are adjusted according to the intensity (see Figure 4 / STEP 214 and Figure 11).

[0119] In the remote operation device 20, the remote control device 200 determines whether or not an operation end condition for the work machine 40 has been satisfied (FIG. 4 / STEP 220). The operation end condition is a condition for determining that work is not being performed using the work machine 40 that is the operation target.

[0120] The operation end condition includes, for example, a condition that the operation of the prime mover of the work machine 40 has been stopped (or an operation to stop the operation of the prime mover of the work machine 40 has been performed via the remote input interface 210). The operation end condition may also include a condition that the operation lever, travel pedal, and / or option pedal constituting the remote input interface 210 have been returned to the neutral position or the initial position (and that state has been maintained for a certain period of time) to cause the work machine 40 to end a predetermined operation of the operating mechanisms such as the work mechanism 44. In addition, the operation end condition may include conditions such as whether or not the cut-off lever has been operated to the lock position, whether or not an operation to stop the operation of the prime mover of the work machine 40 has been performed via the remote input interface 210, and whether or not the switch constituting the remote input interface 210 has been switched from the ON state to the OFF state to cause the work machine 40 to end a predetermined operation of the operating mechanisms such as the work mechanism 44, and may be detected depending on whether or not a plurality of conditions are satisfied.

[0121] If it is determined that the operation end condition is not satisfied (NO in FIG. 4 / STEP 220), the remote operation of the work machine 40 by the remote operation device 20 continues (FIG. 4 / STEP 218).

[0122] On the other hand, if it is determined that the operation end condition is satisfied (YES in FIG. 4 / STEP 220), the remote control device 200 determines the operation content by the operator via the remote input interface 210 (FIG. 4 / STEP 222).

[0123] FIG. 17 is a diagram showing an operation selection screen in the operation system according to this embodiment. Specifically, the operation selection screen shown in FIG. 17 is output to the remote output interface 220. The operation selection screen shows an operation restart button Q81, an operation switching button Q82, and an operation end button Q83 which constitute the remote input interface 210. The operation restart button Q81 constitutes an input interface for restarting remote operation, via the remote operation device 20, of the work machine 40 which is the operation target and in a state of communication establishment with the remote operation device 20. The operation switching button Q82 constitutes an input interface for starting remote operation, via the remote operation device 20, of a work machine 40 (second work machine) which is the operation target different from the work machine 40 (first work machine) which is the operation target and in a state of communication establishment with the remote operation device 20. The operation end button Q83 constitutes an input interface for ending remote operation, via the remote operation device 20, of the work machine 40 which is the operation target and in a state of communication establishment with the remote operation device 20.

[0124] When the operator taps the operation resume button Q81, etc. (when the operation content is an operation to resume remote operation of the work machine 40, which is the operation target and in a state of established communication with the remote control device 20, via the remote control device 20) (FIG. 4 / STEP 222-1), remote operation of the work machine 40 by the remote control device 20 is continued (FIG. 4 / STEP 218).

[0125] When the operation switching button Q82 is tapped or the like (when the operation content is an operation for starting remote operation, via the remote operation device 20, of a work machine 40 (second work machine) that is an operation target other than the work machine 40 (first work machine) that is an operation target and has established communication with the remote operation device 20) (FIG. 4 / STEP 222-2), the authentication process (FIG. 4 / STEP 200) and subsequent processes are repeated. In this case, since there is a work machine 40 (first work machine) that is an operation target and has established communication with the remote operation device 20, communication disconnection process with the first work machine and communication establishment process with the new operation target work machine 40 (second work machine) are executed as "operation preparation process" (FIG. 4 / STEP 208: YES → see STEP 212).

[0126] If the authentication flag f is set to “0” during the authentication process (NO in STEP 254 in FIG. 5, NO in STEP 260 → see STEP 262), the operation resume button Q81 and / or the operation switch button Q82 may not be displayed, or operations such as tapping may be prohibited.

[0127] When the operation end button Q83 is tapped or the like (when the operation content is an operation to end remote operation of the work machine 40, which is the operation target and in which communication has been established with the remote control device 20, via the remote control device 20) (FIG. 4 / STEP 222-3), communication disconnection processing between the remote control device 20 and the work machine 40, which is the operation target, is executed (FIG. 4 / STEP 224), and remote operation of the work machine 40 by the remote control device 20 ends.

[0128] (Action and effect) According to the remote control device 20 that exerts the above-mentioned functions and the operation system configured by the remote control device 20, during the execution of the communication establishment process as the operation preparation process, it becomes possible to set the contents of the designated items for supporting the operation of the work machine 40, which is the object of operation by the remote control device 20 after the completion of the communication establishment process (see NO in FIG. 4 / STEP 208→STEP 210→STEP 214, FIG. 9A, FIG. 9B, FIG. 10, FIG. 11, FIG. 12, FIG. 13, FIG. 16A and FIG. 16B). Therefore, the waiting time during the execution of the communication establishment process between the remote control device 20 and the work machine 40, which is the object of operation, can be utilized. As a result, it is also possible to reduce the stress of the operator during the waiting time. Note that the setting of the contents of the designated items (designated item setting process) may be executed at least during the execution of the communication establishment process.

[0129] When the operation target is switched from one work machine 40 (first operation target / first work machine) to another work machine 40 (second operation target / second work machine) through the communication device, a communication disconnection process and a communication establishment process are executed as an operation preparation process (see FIG. 4 / STEP 208: YES → STEP 212). When the communication disconnection process and the communication establishment process are executed, it becomes possible to set the contents of the designated items through the remote input interface 210 (see FIG. 4 / STEP 214). Therefore, the waiting time during the execution of the communication disconnection process and the communication establishment process for switching the operation target is utilized.

[0130] It becomes possible to virtually experience, through the simulation device, the assistance mode and / or operation mode of the work machine 40 to be operated, which is predicted to occur according to the settings of the specified items through the remote input interface 210 (see Figs. 14 and 15). It becomes possible to reset or change the settings of the specified items through the remote input interface 210 based on the experience. As described above, the simulation device is composed of the remote control device 200, the remote input interface 210 (remote operation mechanism 211), and the remote output interface 220 (remote image output device 221).

[0131] The setting contents of the specified items are stored in a storage device or the like constituting the remote control device 200, and the setting contents of the specified items stored in the storage device or the like are output to the remote output interface 220 (remote image output device 221) (see Fig. 9A / Q22, Fig. 10 / Q32, Fig. 11 / Q42, Fig. 12 / Q52, Fig. 13 / Q62, Q63, Fig. 16A / Q73). The current setting contents of the specified items output to the remote output interface 220 can be confirmed by an operator or the like. Based on the output contents, it becomes possible to reset or change the setting contents of the specified items through the remote input interface 210 (see Figs. 9B / Q26 to Q29, Fig. 10 / Q32, Fig. 11 / Q42, Fig. 12 / Q52, Fig. 13 / Q62, Q63, Fig. 16B / Q75 to Q78).

[0132] According to the remote operation device 20 that performs the above functions and the operation system configured thereby, when the operation target is switched from a work machine 40 (first work machine) as one operation target to a work machine 40 (second work machine) as another operation target, communication disconnection processing and communication establishment processing are executed in parallel (see FIG. 4 / STEP 208: YES → STEP 210). The "communication disconnection processing" is processing for disconnecting communication that was established between the remote operation device 20 and the work machine 40 as the one operation target. The "communication establishment processing" is processing for establishing communication between the remote operation device 20 and the work machine 40 as the other operation target. For this reason, switching of the operation target can be executed smoothly.

[0133] The operator can, in accordance with his / her intention, designate a work machine 40 as a new operation target through the remote input interface 210. More specifically, the operator can, in accordance with his / her intention, cancel the designation of the work machine 40 that was the previous operation target through the remote input interface 210, and then designate a work machine 40 as a new operation target (see Fig. 4 / STEP 222-2 →STEP 204 and Fig. 17 / Q82 → ... → Fig. 8B). This can avoid a situation in which the operation target is erroneously switched when the operator does not intend to switch the operation target.

[0134] When a work machine 40 that is an operation target with established communication is specified as a new operation target through the remote input interface 210, the execution of the communication disconnection processing and communication establishment processing is put on hold (FIG. 4 / STEP 222 1 →STEP 218 and FIG. 17 / Q81). When the work machine 40 specified as the new operation target is the same as the work machine 40 that was previously specified, the communication already established with that work machine 40 continues to be used. This makes it unnecessary to execute the communication disconnection processing and communication establishment processing again when specifying (re-specifying) a work machine 40 that is a new operation target, and operation of the operation target can be smoothly resumed.

[0135] When the operator gives an instruction to end operation of the work machine 40 to be operated through the remote input interface 210 according to his / her intention, communication disconnection processing is executed (see FIG. 4 / STEP 222-3 →STEP 224 and FIG. 17 / Q83). This can prevent a situation in which communication established between the remote operation device 20 and the work machine 40 to be operated is erroneously disconnected.

[0136] According to the remote operation device 20 that performs the above-mentioned functions and the operation system configured therewith, the operation history of the work machine 40 that is the operation target can be acquired separately for each operator identifier and, in turn, each operator identified by the operator identifier, and can be stored and retained in a database or the like (see FIG. 6 / STEP 288). When the operation target of the remote operation device 20 is switched from one work machine 40 (first work machine) to another work machine 40 (second work machine) (see FIG. 4 / STEP 222-2 → . . . →STEP 204 and FIG. 8B / Q14, Q15), the operation history of the other work machine 40 is acquired together with the operator identifier (see FIG. 6 / STEP 288). This allows the operation history of each different operation target (work machine 40) by the operator to be acquired separately.

[0137] Furthermore, the presence or absence of an instruction to switch the operation target is determined on the condition that no work is being performed using the work machine 40 that is the operation target (see FIG. 4 / STEP 220 -> STEP 222). This makes it possible to avoid a situation in which, while work is being performed using a work machine 40 that is an operation target, the operation target is switched to another operation target, and the work is stopped against the operator's intention.

[0138] When an operator is replaced by another operator, the operation history of the work machine 40 to be operated is acquired together with an operator identifier for identifying the new operator (FIG. 5 / STEP 202->...->YES in STEP 250, see FIG. 6 / STEP 288). This makes it possible to distinguish and acquire the operation history of the work machine 40 to be operated for each different operator.

[0139] Furthermore, when a notification of a changeover from one operator to another operator is accepted through the remote input interface 210 (and further, when the operator is actually changed), the operation history of the operation target is acquired together with the operator identifier of the other operator (see FIG. 5 / STEP 202->...->YES in STEP 240, and FIG. 6 / STEP 288). This allows the operation history of the work machine 40, which is the operation target, to be distinguished and acquired for each different operator. Furthermore, when a switching instruction from one work machine 40 (first work machine) to another work machine 40 (second work machine) is accepted through the remote input interface 210 (and further, when the operation target is actually switched), the operation history of the other work machine 40 is acquired together with the operator identifier (see FIG. 5 / STEP 202->...->YES in STEP 240, and FIG. 6 / STEP 288). This allows the operation history of each work machine 40, which is the operation target, to be distinguished and acquired for each different operator.

[0140] When an operator is replaced by another operator, if authentication of the other operator fails (see FIG. 5 / STEP 250: YES → STEP 251 → ... → STEP 260 ... NO → STEP 262), the other operator is prohibited or restricted from operating the work machine 40 as the operation target through the remote operation mechanism 211 (see FIG. 6 / STEP 282: NO → STEP 286). As a result, the prohibited or restricted operation history of the other operator and the work machine 40 as the operation target are both acquired and stored in a database or the like.

[0141] When an operator is replaced by another operator, if authentication of the other operator fails (see FIG. 5 / STEP 250: YES → STEP 251 → ... → STEP 260: NO → STEP 262), switching of the work machine 40 as the operation target in response to a switching instruction from the other operator is prohibited. On the other hand, if authentication of the other operator is successful, switching of the work machine 40 as the operation target in response to a switching instruction from the other operator is permitted.

[0142] (Another embodiment of the present invention) When the operation target is switched from one operation target work machine 40 to another operation target work machine 40 having specifications different from the one operation target, the setting of the designated items corresponding to the differences in the specifications may be accepted by the remote control device 200 through the remote input interface 210. For example, when the types of the tip attachments 445 are different, it may be possible to set the assignment mode to each button so that a function specific to the operation of the tip attachment 445 (e.g., a lifting magnet) of the work machine 40 after the switch is assigned to the button. Specifically, when a function specific to the operation of the tip attachment 445, which is a difference in the specifications of the work machine 40 after the switch, is not assigned to the button, it is determined that the operation preparation process is not completed (NO in FIG. 4 / STEP 216), and the designated item setting process may be continuously executed (FIG. 4 / STEP 214). This ensures that a function specific to the operation corresponding to the differences in the specifications is assigned to the button, enabling smooth operation of the other work machine 40 after the switch of the operation target.

[0143] In the above embodiment, the "simulation device" presents a simulation image showing the operating status of the work machine 40 by displaying an operation assist, a depth sense image, and a terrain image specified on the menu screen, but the "simulation device" may also be made to allow the user to experience other designated item functions that can be specified on the menu screen. For example, when the designated item "lever pattern" is selected in the designated item setting process and the lever pattern is set, the operation mode of the remote control mechanism 211 or the operating levers 2110, 2111, and 2112 that configure the remote control mechanism 211 is recognized in accordance with the lever pattern, and the operation of the work mechanism 44, etc. of the work machine 40 in the virtual space is controlled in accordance with the recognized operation mode. When the designated item "lever sensitivity" is selected in the designated item setting process and the lever sensitivity is set, the operation of the work mechanism 44, etc. of the work machine 40 in the virtual space is controlled in accordance with the operation mode recognized according to the lever sensitivity. When the designated item "seat vibration" is selected in the designated item setting process and the strength of the seat vibration is finally set, the operating speed and amplitude of the seat drive device 224, and further the speed and amount of change in the attitude of the seat 24, are adjusted according to the operating status of the work machine 40 in the virtual space in response to the intensity. The operating status of the work machine 40 in the virtual space includes the tilt angle around the pitch axis and / or roll axis of the upper rotating body 42 of the work machine 40 relative to the vertical direction in the virtual space, whether or not the prime mover of the work machine 40 is operating in the virtual space, and the attitude of the work mechanism 44 of the work machine 40 in the virtual space. When the designated item "button assignment" is selected in the designated item setting process and the degree of button assignment is finally set, for example, a button to which an angle adjustment and / or a zoom-in function / zoom-out function of the actual machine imaging device 412 of the work machine 40 in the virtual space is assigned is operated by the operator, and a virtual captured image or work environment image with the range and / or scale of the captured image adjusted is output to the remote image output device 221 (display device).

[0144] Of the functions of the remote operation device 20, except for functions specific to the remote operation device 20, such as communication establishment processing (see FIG. 4 / STEP 210 and STEP 212), communication disconnection processing (see FIG. 4 / STEP 212), and remote operation processing (see FIG. 4 / STEP 218), authentication processing (see FIG. 4 / STEP 200), setting processing of the operation target and work content (see FIG. 4 / STEP 204), and designated item setting processing (see FIG. 4 / STEP 214), may be executed by the information processing terminal 10. The functions of the remote operation device 20, such as communication establishment processing (see FIG. 4 / STEP 210 and STEP 212), communication disconnection processing (see FIG. 4 / STEP 212), and remote operation processing (see FIG. 4 / STEP 218), may be executed by a server (management device) capable of communicating with the remote operation device 20 and the work machine 40 via the second communication network.

[0145] Even after a communication disconnection process is executed between the remote operation device 20 and a work machine 40 that is an operation target, the operating status (such as a powered state) of the real machine control device 400 and the real machine wireless communication device 450 of the work machine 40 may be maintained as it was before the communication disconnection process. This makes it unnecessary to perform a process for restoring the operating status of the real machine control device 400 and the real machine wireless communication device 450 to the status before the communication disconnection process (such as a process for energizing the real machine wireless communication device 450 to return it to a state where the communication establishment process can be executed) when communication is established again between the remote operation device 20 and the work machine 40 that is an operation target, and therefore the communication establishment process can be executed smoothly and quickly. For example, when the communication disconnection process is executed, the power supply state to the real machine control device 400 mounted on the work machine 40 is maintained (the process of cutting off the power supply to the real machine control device 400 is not performed), and the power supply state to the real machine imaging device 412 and the real machine sensor group 416 is maintained (the process of cutting off the power supply to the real machine imaging device 412 and the real machine sensor group 416 is not performed).The real machine control device 400 then transmits signals corresponding to the real machine operating status relating to the captured images acquired by the real machine imaging device 412 and the status of the work machine 40 acquired by the real machine sensor group 416 to a destination identified and indicated by the operation device identifier, and continues the process of receiving the signals corresponding to the operation mode of the remote operation lever for controlling the operation of the operating mechanism through the real machine wireless communication device 450. In this case, a signal according to the actual machine operating status is transmitted to a destination identified and indicated by the operation device identifier, and the remote operation device 20, which is the destination, can determine whether or not information related to the actual machine identifier contained in the received signal corresponds to that of the work machine 40 currently being operated by the remote operation device 20, thereby determining that the received signal is not that of the current operation target of the remote operation device 20. Then, since the received signal is not that of the current operation target of the remote operation device 20, a process is performed to discard the received signal. Note that, when the control device in the present invention includes the actual machine control device 400 of the work machine 40, the actual machine control device 400 may perform control so that the operating status (such as a power-on state) of the actual machine control device 400 and the actual machine wireless communication device 450, etc., are maintained as they were before the communication disconnection process.

[0146] Execution of the communication disconnection process may disconnect a part of the communication established between the remote operation device 20 and one of the work machines 40 to be operated. For example, when the communication disconnection process is executed, a process is executed to turn off the power to the actual machine imaging device 412 and the actual machine sensor group 416 that have been activated, but the power supply to the actual machine control device 400 mounted on the work machine 40 is maintained (a process to cut off the power supply to the actual machine control device 400 is not executed). The actual machine control device 400 then transmits signals corresponding to the actual machine operating status relating to the captured images acquired by the actual machine imaging device 412 and the status of the work machine 40 acquired by the actual machine sensor group 416 to a destination identified and indicated by the operation device identifier, and continues the process of receiving the signals corresponding to the operation mode of the remote operation lever for controlling the operation of the operating mechanism through the actual machine wireless communication device 450. In this case, the signal corresponding to the actual machine operation status regarding the state of the work machine 40 acquired by the actual machine sensor group 416 becomes an empty signal with no information, and the signal corresponding to the operation mode of the remote control lever is not received until the work machine 40 becomes an operation target again. The empty signal may be replaced with a signal indicating information indicating that the signal is empty. The empty signal may be transmitted at a transmission cycle lower than the transmission cycle in which the signal corresponding to the actual machine operation status acquired by the activated actual machine imaging device 412 and the actual machine sensor group 416 is transmitted to the transmission destination. In this case, the signal corresponding to the actual machine operation status is transmitted to the transmission destination identified and indicated by the operation device identifier, and the remote operation device 20, which is the transmission destination, can determine whether or not the information related to the actual machine identifier included in the received signal corresponds to that of the work machine 40 currently operated by the remote operation device 20, thereby determining that the received signal is not the current operation target of the remote operation device 20. Then, since the received signal is not the current operation target of the remote operation device 20, a process of discarding the received signal is performed.As a result, the communication volume between the remote control device 20 and the work machine 40 is maintained, although it is low compared to the communication established state, so that when communication is established again between the remote control device 20 and the work machine 40 as the one operation target, the process of restoring the operating status of the real machine control device 400 and the real machine wireless communication device 450, etc. to the status before the communication disconnection process is unnecessary or can be simplified. For example, when the real machine control device 400 receives a communication establishment request signal from the remote control device 200, the real machine control device 400 can transmit a signal including an imaged image and the real machine operating status to the remote control device 200 simply by activating the real machine imaging device 412 and the real machine sensor group 416, and can restore the status before the communication disconnection process. Also, for example, the real machine control device 400 may enable the power supply to the real machine imaging device 412 and the real machine sensor group 416 by not performing the process of turning off the power of the activated real machine imaging device 412 and the real machine sensor group 416 when the communication disconnection process is executed. In this case, the remote control device 200 receives a signal including the captured image and the actual machine operation status before transmitting a communication establishment request signal to the actual machine control device 400, and when executing the communication establishment process, outputs the captured image and the actual machine operation status to the remote output interface 220. As described above, the communication establishment process can be executed smoothly and quickly.

[0147] On the condition that there is no operator change report through the remote input interface 210 (the first flag f1 is set to "0") (see FIG. 5 / STEP 240: NO → STEP 242), the presence or absence of an instruction to switch the work machine 40 as an operation target through the remote input interface 210 may be determined while maintaining the communication established with the work machine 40 as one operation target (see FIG. 4 / STEP 222). As a result, when the work machine 40 designated as a new operation target is the same as the previous operation target (see FIG. 4 / STEP 222-1), the communication already established between the remote operation device 20 and the work machine 40 continues to be used, eliminating the need to execute communication disconnection processing and communication establishment processing again, and operation of the work machine 40 can be smoothly resumed.

[0148] The presence or absence of an instruction to switch the work machine 40 as an operation target through the remote input interface 210 may be determined (see FIG. 4 / STEP 222) based on the requirements that an operator change report has been made through the remote input interface 210 (the first flag f1 is set to "1") and that the acquired operator identifier has changed from one operator identifier to another operator identifier (the second flag f2 is set to "1") (see FIG. 5 / STEP 240: YES → STEP 241 and STEP 250: YES → STEP 251). This prevents the one operator from designating the work machine 40 as an operation target to be remotely operated by the other operator when the operator is changed over to another operator, and allows the other operator to designate the work machine 40 as an operation target to be remotely operated by himself / herself according to his / her will.

[0149] The communication established between the remote operation device 20 and the work machine 40 as the operation target may be disconnected by executing the communication disconnection process on the condition that the change of operator has been reported through the remote input interface 210 (the first flag f1 is set to "1") (see FIG. 5 / STEP 240 -> STEP 241). This prevents a situation in which the communication establishment state between the remote operation device 20 and the work machine 40 as the operation target designated by one operator before the changeover remains, causing the other operator after the changeover to feel uneasy. For example, a captured image or a work environment image received by the remote operation device 20 from the work machine 40 as the operation target designated by the one operator is output to the remote image output device 221. The captured image or the work environment image is unnecessary information that is unrelated to the remote operation by the other operator himself, so if the captured image or the work environment image continues to be output to the remote image output device 221, the other operator will feel uneasy. Here, when a change of operator is notified, the communication established between the remote control device 20 and the work machine 40 to be operated is cut off, and the captured image or work environment image is no longer output to the remote image output device 221, thereby preventing the other operator from feeling uncomfortable.

[0150] In the above embodiment, when it is determined that an existing operation target exists (YES in FIG. 4 / STEP 208) on the condition that an instruction to switch the work machine 40 as the operation target has been given through the remote input interface 210 (see FIG. 4 / STEP 222-2 and FIG. 17 / Q82), the remote control device 200 starts to execute a communication disconnection process as an "operation preparation process" (FIG. 4 / STEP 212) in addition to the communication establishment process, but this is not limited to this. For example, after communication established with one work machine 40 (first work machine) is disconnected in response to an operation by the operator through the remote input interface 210, on the condition that an instruction to switch the work machine 40 as the operation target has been given through the remote input interface 210 by the operator (see FIG. 4 / STEP 222-2 and FIG. 17 / Q82), communication establishment process may be executed to establish communication with another work machine 40 (second work machine). This makes it possible to execute communication establishment processing for the work machine 40 (first work machine) and communication disconnection processing for the work machine 40 (second work machine) in accordance with the operator's intention.

[0151] When work is being carried out using the work machine 40 as the object of operation (FIG. 4 / STEP 218), provided that a change of operator is reported from one operator to another operator via the remote input interface 210 (FIG. 5 / STEP 240: YES → see STEP 241), the operation history of the work machine 40 as the object of operation may be acquired together with an operator identifier that identifies the other operator (see FIG. 6 / STEP 288).

[0152] A first aspect of the present invention provides an operating device for operating an operating object, the operating device including a communication device capable of communicating with the operating object, a control device capable of executing a communication establishment process for establishing the communication with the operating object through the communication device, and a designated item setting process for receiving, at least during the communication establishment process, through an input unit, the setting of the contents of designated items for supporting the operation of the operating object by the operating device after the communication establishment process is executed.

[0153] According to the operation device having the above configuration, during the execution of the communication establishment process, it is possible to set the contents of the designated items for supporting the operation of the operation target by the operation device after the completion of the communication establishment process, thereby making use of the waiting time during the execution of the communication establishment process.

[0154] An operating device according to a second aspect of the present invention is the operating device according to the first aspect, wherein, when the operating target is switched from one operating target to another operating target, the control device executes a communication disconnection process to disconnect communication established between the communication device and the one operating target, and a communication establishment process to establish communication between the operating device and the other operating target, and it is preferable that the control device accepts setting of the content of the specified item through the input unit when at least one of the communication disconnection process and the communication establishment process is executed in conjunction with switching of the operating target through the communication device.

[0155] According to the operation device having this configuration, when the operation target is switched from one operation target to another operation target, a communication disconnection process and a communication establishment process are executed. The "communication disconnection process" is a process for disconnecting the communication established between the operation device and the one operation target. The "communication establishment process" here is a process for establishing communication between the operation device and the other operation target. When at least one of the "communication disconnection process" and "the communication establishment process" is executed, it becomes possible to set the contents of the specified items via the input unit. Therefore, the waiting time during the execution of the process for switching the operation target is utilized.

[0156] In an operating device according to a third aspect of the present invention, in the operating device according to the first or second aspect, it is preferable that, when the operating object is switched from one operating object to another operating object having specifications different from those of the one operating object, the control device accepts, via the input unit, a setting of the content of the specified item corresponding to the difference in the specifications.

[0157] According to the operation device having the above configuration, when an operation target is switched from one operation target to another operation target having specifications different from those of the first operation target, it is possible to set, via the input unit, the specified items corresponding to the differences in the specifications, which enables smooth operation of the other operation target after the switching of the operation target.

[0158] An operating device according to a fourth aspect of the present invention is an operating device according to any one of the first to third aspects, further comprising a simulation device for operating a virtual operating object corresponding to the operating target in a virtual space, and it is preferable that the control device assists in operating the virtual operating object in the virtual space based on the setting contents of the specified items received through the input unit.

[0159] According to the operation device having the above configuration, it is possible to virtually experience, through the simulation device, the support mode and / or operation mode of the operation target that is predicted to occur according to the setting contents of the items specified through the input unit. Based on the experience, it is possible to reset or change the settings of the items specified through the input unit.

[0160] An operating device according to a fifth aspect of the present invention is preferably an operating device according to any one of the first to fourth aspects, further comprising a storage device that stores setting contents of the designated items, and an output unit that outputs the setting contents of the designated items stored in the storage device.

[0161] According to the operation device having the above configuration, the operator can confirm the current setting contents of the designated items output to the output unit. Based on the output contents, the designated items can be reset or changed via the input unit.

[0162] An operating device according to a sixth aspect of the present invention is the operating device according to the fifth aspect, and preferably has a function of authenticating an operator, and the output unit outputs setting contents of the specified items stored in the storage device in association with the authenticated operator.

[0163] According to the operation device having this configuration, the contents of the designated items last set by each operator (or the contents of the designated items appropriate for each operator) are stored and output to the output unit. The output contents may be defined as the default contents of the designated items.

[0164] An operating device according to a seventh aspect of the present invention is the operating device according to the fifth aspect, further comprising a function of authenticating an operator, and it is preferable that the storage device stores the setting contents of the specified items received through the input unit in association with the authenticated operator.

[0165] According to the operation device having the above configuration, the contents of the designation items last set by each operator can be stored in the storage device.

[0166] In an operating device according to an eighth aspect of the present invention, in the operating device according to any one of the first to seventh aspects, it is preferable that the control device accepts settings via the input unit regarding the content of a specified item selected from among the plurality of specified items.

[0167] According to the operation device having the above-described configuration, only the contents of one or more designated items selected (through the input unit) from among the multiple designated items are set through the input unit.

[0168] What is provided by the present invention is an operation system, which is configured by an operation device according to any one of the first to eighth aspects and the operation target. [Explanation of symbols]

[0169] 10. Information processing terminal 100. Terminal control device 110...Terminal input interface 120...Terminal output interface 150. Terminal wireless communication equipment 20. Remote control device (operation device) 24 seats 200. Remote control device (control device) 210 Remote input interface 211. Remote control mechanism 220 Remote output interface 221 Remote image output device 222. Remote audio output device 224. Seat drive mechanism 250 Remote wireless communication equipment (communication devices) 40. Work machinery 44‥Working mechanism 400... Actual machine control device 410... Actual machine input interface 420... Actual machine output interface

Claims

1. An operation device for operating an operation object, A communication device capable of communicating with the operation target; a control device capable of executing a communication establishment process for establishing the communication with the operation target through the communication device, and a designated item setting process for receiving, at least during the execution of the communication establishment process, through an input unit, a setting of a designated item for supporting the operation of the operation target by the operation device after the execution of the communication establishment process; An operating device comprising:

2. 2. The operating device according to claim 1, The control device includes: When the operation target is switched from one operation target to another operation target, a communication disconnection process is executed to disconnect a communication established between the communication device and the one operation target, and a communication establishment process is executed to establish a communication between the communication device and the other operation target; an operation device that accepts a setting of the content of the specified item through the input unit when at least one of the communication disconnection process and the communication establishment process is executed in association with switching of the operation target through the communication device.

3. 3. The operating device according to claim 2, The control device accepts, when the operation object is switched from one operation object to another operation object having specifications different from those of the one operation object, the setting of the content of the specified item corresponding to the differences in the specifications through the input unit.

4. 2. The operating device according to claim 1, A simulation device for operating a virtual operation object corresponding to the operation object in a virtual space, The control device is an operation device that assists in operating the virtual operation target in the virtual space based on the setting contents of the specified items accepted through the input unit.

5. 2. The operating device according to claim 1, A storage device that stores the setting contents of the specified items; an output unit that outputs the setting contents of the specified items stored in the storage device; An operating device comprising:

6. 6. The operating device according to claim 5, It has a function to authenticate the operator, The output unit outputs setting contents of the specified items that are associated with the authenticated operator and stored in the storage device.

7. 6. The operating device according to claim 5, It has a function to authenticate the operator, The storage device stores the setting contents of the specified items received through the input unit in association with the authenticated operator.

8. 2. The operating device according to claim 1, The control device is an operation device that accepts settings, via the input unit, for the content of the designated item selected from the plurality of designated items.

9. 1. An operating system, comprising: An operating device according to any one of claims 1 to 8; The operation target; An operating system comprising:

Images