Operation device and operation system

Abstract

This operation device for operating an object to be operated comprises: a communication device capable of performing communication with the object to be operated; and a control device capable of executing communication establishment processing for establishing the communication with the object to be operated through the communication device, and designation item setting processing for receiving, through the input part, setting of content of a designation item for supporting operation of the object to be operated by the operation device after execution of the communication establishment processing during at least execution of the communication establishment processing.

Description

Operating device and operating system

[0001] The present invention relates to an operation device and an operation system for operating an operation object.

[0002] A remote control system is disclosed that is configured to be able to selectively remotely control a plurality of work machines to be operated (to be able to switch the work machine to be operated from a first work machine to another work machine) (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2021-139144

[0004] However, in an operation system such as that disclosed in Patent Document 1, it takes a certain amount of time to establish communication between the operation device and the operation target so that they can be operated. While the process for establishing the communication is being executed, the operator has to wait for the communication establishment to be completed, resulting in wasted time.

[0005] An object of the present invention is to provide an operation device or the like that can prevent a waiting time from passing unnecessarily in a process for establishing communication with an operation target.

[0006] The present invention provides an operating device for operating an operating object, the operating device including a communication device capable of communicating with the operating object, a control device capable of executing a communication establishment process for establishing the communication with the operating object via the communication device, and a designated item setting process for receiving, at least during the communication establishment process, through an input unit, the setting of designated items for assisting the operating device in operating the operating object after the communication establishment process is executed.

[0007] The present invention provides an operation system, which is configured by the operation device described above and the operation target.

[0008] FIG. 1 is a configuration diagram of an operation system according to one embodiment of the present invention. FIG. 2 is a perspective view of a remote control device according to one embodiment of the present invention. FIG. 3 is a perspective view of a work machine according to one embodiment of the present invention. FIG. 4 is a flowchart of an operation system according to one embodiment of the present invention. FIG. 5 is a flowchart of an authentication processing function of an operation system according to one embodiment of the present invention. FIG. 6 is a flowchart of an operation processing function of an operation system according to one embodiment of the present invention. FIG. 7A is a diagram showing a first authentication screen for an operator in an operation system according to one embodiment of the present invention. FIG. 7B is a diagram showing a second authentication screen for an operator in an operation system according to one embodiment of the present invention. FIG. 8A is a diagram showing a work content designation screen in an operation system according to one embodiment of the present invention. FIG. 8B is a diagram showing an operation target designation screen in an operation system according to one embodiment of the present invention. FIG. 8C is a diagram showing a work content and operation target confirmation screen in an operation system according to one embodiment of the present invention. FIG. 9A is a diagram showing a first lever pattern setting screen in an operation system according to one embodiment of the present invention. FIG. 9B is a diagram showing a second lever pattern setting screen in an operation system according to one embodiment of the present invention. FIG. 10 is a diagram showing a lever sensitivity setting screen in an operation system according to one embodiment of the present invention. FIG. 11 is a diagram showing a seat vibration mode setting screen in an operation system according to one embodiment of the present invention. Fig. 12 is a diagram showing a screen for setting operation assistance in an operation system according to one embodiment of the present invention. Fig. 13 is a diagram showing a screen for setting guidance in an operation system according to one embodiment of the present invention. Fig. 14 is a diagram showing a depth perception image in an operation system according to one embodiment of the present invention. Fig. 15 is a diagram showing a terrain image in an operation system according to one embodiment of the present invention. Fig. 16A is a diagram showing a screen for setting a first button function in an operation system according to one embodiment of the present invention. Fig. 16B is a diagram showing a screen for setting a second button function in an operation system according to one embodiment of the present invention. Fig. 17 is a diagram showing an operation selection screen in an operation system according to one embodiment of the present invention.

[0009] (Configuration of operation system) FIG. 1 is a configuration diagram of an operation system according to one embodiment of the present invention. This operation system has an information processing terminal 10 and a remote operation device 20 (operation device), and is configured to remotely operate a work machine 40 (operation target) using the remote operation device 20. Note that in other embodiments, the operation system may also include the work machine 40. The information processing terminal 10 and the remote operation device 20 are configured to be able to communicate with each other via a first communication network. The remote operation device 20 and the work machine 40 are configured to be able to communicate with each other via a second communication network. The first communication network and the second communication network may have the same or different communication standards, etc. The information processing terminal 10 may be omitted. Furthermore, the remote operation device 20 may have the functions of the information processing terminal 10 built in. The "operation target" may be a work machine 40, or any device that has the function of performing work or tasks by remote control, such as a vehicle, tiller, or airplane.

[0010] (Configuration of Information Processing Terminal) As shown in FIG. 1, the information processing terminal 10 includes a terminal control device 100 (control device, control unit), a terminal input interface 110 (input interface, input unit), a terminal output interface 120 (output interface, output unit), and a terminal wireless communication device 150 (communication device) for communicating via a first communication network. The information processing terminal 10 is configured as a smartphone, tablet terminal, laptop, or the like carried by an operator or installed in a remote control device 20. The terminal control device 100 is configured as a calculation processing device (a single-core processor or a multi-core processor, or a processor core constituting the same), reads necessary data and software from a storage device such as a memory, and performs calculation processing on the data in accordance with the software. The terminal input interface 110 and the terminal output interface 120 are configured, for example, as touch panel displays.

[0011] Furthermore, as described above, the information processing terminal 10 may be a component of the remote control device 20. In this case, the information processing terminal 10 and the remote control device 20 may be configured to be capable of wired or wireless communication, the terminal input interface 110 may constitute a part of the remote input interface 210, and the terminal output interface 120 may constitute a part of the remote output interface 220.

[0012] 1, the remote operation device 20 has a remote control device 200 (control device, control unit), a remote input interface 210 (input interface, input unit), a remote output interface 220 (output interface, output unit), and a remote wireless communication device 250 (communication device) that is capable of communicating with the information processing terminal 10 or the work machine 40 via a first communication network and a second communication network. The remote wireless communication device 250 is assigned an operation device identifier (IP address) that identifies the remote wireless communication device 250 in the second communication network and uses it as a transmission destination. The remote control device 200 is composed of an arithmetic processing device (a single-core processor or a multi-core processor or processor cores that make up the same), reads necessary data and software from a storage device such as a memory, and executes arithmetic processing on the data in accordance with the software.

[0013] The remote input interface 210 has a remote operation mechanism 211 and a face authentication sensor 212. The remote output interface 220 has a remote image output device 221, a remote sound output device 222, and a seat drive device 224.

[0014] FIG. 2 is a perspective view of the remote control mechanism 211 according to this embodiment. The remote control mechanism 211 has the devices and components shown in FIG. 2 , including a travel control device, a swing control device, a boom control device, an arm control device, and an end-attachment control device. Each control device has an operation lever that is rotated. The operation lever (travel lever) of the travel control device is operated to drive a hydraulic travel motor (actuator) that constitutes the travel drive mechanism in the undercarriage 41 of the work machine 40. The travel lever may also serve as a travel pedal. For example, a travel pedal may be fixed to the base or lower end of the travel lever. The operation lever (swing lever) of the swing control device is operated to operate the hydraulic swing motor (actuator) that constitutes the swing drive mechanism of the work machine 40. The operation lever (boom lever) of the boom control device is operated to operate the boom cylinder 442 of the work machine 40. The control lever (arm lever) of the arm control device is operated to move the arm cylinder 444 of the work machine 40. The control lever (tip attachment lever) of the tip attachment control device is operated to move the tip attachment cylinder 446 of the work machine 40. An optional pedal that is operated to move the tip attachment of the work machine 40, etc., may be arranged on the side of the travel pedal.

[0015] The operating levers constituting the remote operation mechanism 211 are arranged around the seat 24 on which the operator sits, as shown in FIG. 2 , for example. The seat 24 has a seat portion 241 installed on the upper side of a seat base 240 and a backrest portion 242, and is configured like a high-back chair with armrests. The seat 24 may be configured as a low-back chair without a headrest, or a chair without the backrest portion 242, or any other seating configuration suitable for the operator. The information processing terminal 10 may be configured to be mountable on the seat 24 and / or on a side of the seat 24. In this case, it is preferable that the touch screen of the information processing terminal 10 be mounted facing the seat 24 so that the display of the information processing terminal 10 can be easily operated and viewed by the operator seated on the seat 24. A shutoff lever (not shown) may also be mounted on the seat 24 and / or on a side of the seat 24. The shutoff lever has a function of prohibiting operation of the work machine (stopping actuator operation) when tilted from the unlocked position to the locked position.

[0016] Left and right travel levers 2110 corresponding to the left and right crawlers are arranged side by side in front of the seat 24. One operation lever may serve as multiple operation levers. For example, the left operation lever 2111 provided in front of the left frame of the seat 24 shown in FIG. 2 may function as an arm lever when operated in the forward / backward direction, and as a swing lever when operated in the left / right direction. Similarly, the right operation lever 2112 provided in front of the right frame of the seat shown in FIG. 2 may function as a boom lever when operated in the forward / backward direction, and as a tip attachment lever when operated in the left / right direction. Such lever patterns may be changed as desired by an operator's operation command.

[0017] 2, the face authentication sensor 212 is configured with a camera and / or a distance measurement sensor disposed in front of the seat 24. An authentication process for the operator is executed based on an image of the operator's face captured by the camera and / or a three-dimensional shape of the operator's face captured by the distance measurement sensor.

[0018] The remote image output device 221 may be composed of a single image output device or multiple image output devices. For example, as shown in FIG. 2 , the remote image output device 221 may be composed of a central remote image output device 2210, a left remote image output device 2211, and a right remote image output device 2212, each having a substantially rectangular screen, and arranged in front of, the seat 24, diagonally forward to the left, and diagonally forward to the right. The shapes and sizes of the screens (image display areas) of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be the same or different. The remote image output device 221 may be composed of a single curved or bendable image output device, or two or four or more image output devices arranged to surround the front of the seat 24.

[0019] As shown in Fig. 2, the right edge of the left remote image output device 2211 is adjacent to the left edge of the central remote image output device 2210 so that the screens of the central remote image output device 2210 and the left remote image output device 2211 form a first specified angle (e.g., 120° to 150°). As shown in Fig. 2, the left edge of the right remote image output device 2212 is adjacent to the right edge of the central remote image output device 2210 so that the screens of the central remote image output device 2210 and the right remote image output device 2212 form a second specified angle (e.g., 120° to 150°).

[0020] The screens of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be parallel to the vertical direction or may be tilted relative to the vertical direction. At least one of the central remote image output device 2210, the left remote image output device 2211, and the right remote image output device 2212 may be composed of multiple divided image output devices. For example, the central remote image output device 2210 may be composed of adjacent image output devices above and below, each having a substantially rectangular screen.

[0021] The remote audio output device 222 is made up of one or more speakers, and is made up of audio output devices located, for example, behind the seat 24, behind the left armrest, and behind the right armrest. The specifications of the audio output devices may be the same or different. The remote audio output device 222 may be configured to output audio from the work machine 40 that is collected by a microphone mounted on the work machine 40.

[0022] The seat drive device 224, together with the seat 24, constitutes a "seat device." The seat drive device 224 is configured to tilt or oscillate the seat 24 relative to the seat base 240 (or integrally with the seat base 240) about the pitch axis and / or the roll axis (about the roll axis as necessary). The seat drive device 224 may be configured to oscillate the seat 24 relative to the seat base 240 (or integrally with the seat base 240) in at least one direction among the pitch axis, the roll axis, and the yaw axis.

[0023] (Configuration of Work Machine) As shown in FIG. 1 , the work machine 40 has a real machine control device 400, a real machine input interface 410, a real machine output interface 420, and a real machine wireless communication device 450 for communicating via the second communication network. The real machine wireless communication device 450 is assigned a real machine identifier (IP address) that identifies the real machine wireless communication device 450 on the second communication network and makes it the transmission destination. Each of the components of the real machine control device 400 is made up of an arithmetic processing device (a single-core processor or a multi-core processor or a processor core that makes up such a processor), reads the necessary data and software from a storage device such as memory, and executes arithmetic processing on the data in accordance with the software. The remote control device 200 communicates with a specific work machine 40 selected from a plurality of work machines 40 via the remote wireless communication device 250, making it possible to operate the specific work machine 40 using the remote input interface 210.

[0024] Figure 3 is a perspective view of a work machine 40 according to this embodiment. The work machine 40 is, for example, a hydraulic excavator, and as shown in Figure 3, is equipped with a crawler-type undercarriage 41 that constitutes a traveling drive mechanism, and an upper rotating body 42 that is rotatably mounted on the undercarriage 41 via a swing drive mechanism. A cab 43 (operator's compartment) is provided on the front left side of the upper rotating body 42. A work mechanism 44 is provided on the front right side of the upper rotating body 42. The cab 43 may be configured to be driven to move up and down relative to the upper rotating body 42 by an elevation drive mechanism (not shown) having a link mechanism and an actuator, and may be configured so that the tilt angle of the cab 43 relative to the upper rotating body 42 can be changed by a tilt drive mechanism (not shown) having an actuator.

[0025] The real machine input interface 410 includes a real machine operation mechanism 411, a real machine imaging device 412, and a real machine sensor group 416. The real machine operation mechanism 411 has multiple operation levers arranged around a seat inside the cab 43, similar to the remote operation mechanism 211. An actuation mechanism or robot is provided in the cab 43 that receives signals corresponding to the operation mode of the remote operation levers and moves the real machine operation levers based on the received signals. The real machine operation mechanism 411 may be configured as an actuation mechanism that operates a control valve constituting a hydraulic circuit mounted on the work machine 40. The real machine imaging device 412 is installed inside the cab 43, for example, and captures images of the environment in front of the cab 43, including at least a portion of the work mechanism 44, through the front window and left and right side windows. Some or all of the front window and side windows may be omitted. The real machine imaging device 412 may be mounted outside the cab 43 as long as it captures images of the environment in front of the cab 43, including at least a portion of the work mechanism 44.

[0026] The actual machine sensor group 416 includes various sensors configured to output signals to the actual machine control device 400 according to parameter values ​​related to the state of the work machine 40. The state of the work machine 40 includes the actual machine attitude detected by an actual machine attitude sensor, i.e., the tilt angle around the pitch axis (pitch angle) and / or the tilt angle around the roll axis (roll angle) relative to the vertical direction of the upper rotating body 42. The actual machine attitude sensor is configured, for example, with a three-dimensional acceleration sensor. The state of the work machine 40 may include whether or not the prime mover (equipment that generates vibration, such as an engine or work oil pump) is operating, as well as the hoisting angle of the boom 441 relative to the upper rotating body 42, the rotation angle of the arm 443 relative to the boom 441, the rotation angle of the tip attachment 445 (lift magnet) relative to the arm 443, the hydraulic oil pressure, the remaining fuel amount, etc. The state of the work machine 40 may also include sound from the work machine 40 collected by a microphone mounted on the work machine 40.

[0027] 3, the working mechanism 44 (operating mechanism) has a boom 441 whose base end is mounted on the upper rotating body 42 so as to be able to be raised and lowered, an arm 443 whose base end is rotatably connected to the tip of the boom 441, and a tip attachment 445 whose base end is rotatably connected to the tip of the arm 443. The working mechanism 44 is equipped with a boom cylinder 442, an arm cylinder 444, and a tip attachment cylinder 446 as actuators each composed of an extendable and retractable hydraulic cylinder.

[0028] The tip attachment 445 may be a bucket with a container for holding objects such as soil and sand, a lifting magnet, a grapple that grips objects by opening and closing multiple opposing gripping claws, a bucket with a hook that can lift an object by engaging a hook attached to the back, or a breaker with a chisel that is reciprocally supported on the breaker body and strikes the work object. The breaker's chisel may be configured to be operable by a switch. Furthermore, the base end of the boom 441 or the base end of the arm 443 may be provided with a swing mechanism that swings the tip end left and right relative to each base end, or a slide arm that has a slidable extension mechanism on the arm 443 to extend and retract the arm 443, and may be configured to be operable by a switch or optional pedal. A dozer with a blade lifting mechanism that drives the blade up and down using a dozer cylinder may be provided on the front side of the lower traveling body 41, and may be configured to be operable by a switch or optional pedal.

[0029] The boom cylinder 442 is interposed between the boom 441 and the upper rotating body 42 so as to extend and retract by receiving a supply of hydraulic oil to rotate the boom 441 in the hoisting direction. The arm cylinder 444 is interposed between the arm 443 and the boom 441 so as to extend and retract by receiving a supply of hydraulic oil to rotate the arm 443 about a horizontal axis relative to the boom 441. The tip attachment cylinder 446 is interposed between the tip attachment 445 and the arm 443 so as to extend and retract by receiving a supply of hydraulic oil to rotate the tip attachment 445 about a horizontal axis relative to the arm 443.

[0030] The actual machine control device 400 is equipped with a switch that can switch between supplying power to the actual machine control device 400 and not supplying power to the actual machine control device 400. The switch switches from a state in which no power is supplied to the actual machine control device 400 to a state in which power is supplied to the actual machine control device 400, causing the actual machine control device 400 to start operating. When the actual machine control device 400 starts operating, the actual machine control device 400 activates the actual machine wireless communication device 450, which connects to the first communication network and the second communication network. Furthermore, the actual machine control device 400 that has started operating also activates the actual machine imaging device 412 and the actual machine sensor group 416. When the switch prevents power from being supplied to the actual machine control device 400, the actual machine control device 400 stops operating, the actual machine wireless communication device 450 is stopped, and communication with the information processing terminal 10 or the work machine 40 via the first communication network and the second communication network is not possible. Furthermore, when the actual machine control device 400 stops operating, the actual machine imaging device 412 and the actual machine sensor group 416 also stop operating.

[0031] (Functions) Fig. 4 is a flowchart of an operation system according to one embodiment of the present invention. Fig. 5 is a flowchart of an authentication processing function of the operation system according to this embodiment. Fig. 6 is a flowchart of an operation processing function of the operation system according to this embodiment. Fig. 7A is a diagram showing a first authentication screen for an operator in the operation system according to this embodiment. Fig. 7B is a diagram showing a second authentication screen for an operator in the operation system according to this embodiment. The functions of the remote operation device 20 and remote operation system configured as described above will be explained using Figs. 4 to 7B.

[0032] When the operation system starts processing, the remote control device 200 first executes authentication processing (FIG. 4 / STEP 200). Specifically, as shown in FIG. 7A, a message Q01 prompting the operator to sit in the seat 24 and an authentication button Q02 are output to the remote output interface 220 (remote image output device 221) (output, displayed by the remote output interface 220 (same hereinafter)). The various buttons output to the remote output interface 220 (remote image output device 221) form a touch panel input interface.

[0033] When the authentication button Q02 is tapped, for example, a face authentication button Q03 and an identifier authentication button Q04 are output to the remote output interface 220 (remote image output device 221), as shown in FIG. 7B . When the face authentication button Q03 is tapped, for example, a camera and / or a distance measurement sensor mounted on the remote operation device 20 and / or the information processing terminal is used to perform face authentication of the operator (and furthermore, by querying a database). When the identifier authentication button Q04 is tapped, the operator is authenticated using the operator identifier (or operator identifier (user ID) and passcode) input through the remote input interface 210. If the authentication is successful, an OK button Q05 is output to the remote output interface 220 (remote image output device 221) together with information indicating that the authentication was successful. If the authentication is unsuccessful, information indicating that the authentication was unsuccessful is output to the remote output interface 220 (remote image output device 221). If the authentication fails, the face authentication button Q03 and the identifier authentication button Q04 may be output again to the remote output interface 220 (remote image output device 221).

[0034] Details of the authentication process executed by the remote control device 200 will be described using the flowchart in Fig. 5. The authentication process may be executed steadily or periodically as background processing. Specifically, first, an operator identifier (including captured image data and / or distance measurement data for facial authentication) is acquired (Fig. 5 / STEP 202). If an operator identifier based on facial authentication is not acquired for a certain period of time, the last acquired operator identifier is acquired along with an operation history indicating that the operator is away from his / her seat (not seated in the seat 24).

[0035] It is determined whether or not an operator change notification has been received via the remote input interface 210 (or the change notification input interface (touch button) that constitutes the remote input interface 210) (FIG. 5 / STEP 240). If the determination result is affirmative (YES in FIG. 5 / STEP 240), a first flag f1 indicating whether or not a change notification has been received is set to "1" (FIG. 5 / STEP 241). On the other hand, if the determination result is negative (NO in FIG. 5 / STEP 240), the first flag f1 is set to "0" (FIG. 5 / STEP 242).

[0036] Next, it is determined whether the operator identifier has changed (whether the operator identifiers used over the most recent multiple times are different from the operator identifiers used in the past) (FIG. 5 / STEP 250). If the determination result is positive (YES in FIG. 5 / STEP 250), a second flag f2 indicating whether the operator identifier has changed is set to "1" (FIG. 5 / STEP 251). For example, if an operator leaves the seat and another operator sits in the seat 24, the newly acquired operator identifier will be different from the last acquired operator identifier, and the determination result will be positive. On the other hand, if the determination result is negative (NO in FIG. 5 / STEP 250), the second flag f2 is set to "0" (FIG. 5 / STEP 252).

[0037] Furthermore, it is determined whether the first flag f1 and the second flag f2 are equal (FIG. 5 / STEP 254). If the determination result is negative (NO in FIG. 5 / STEP 254), that is, if an operator change notification has been made but the operator identifier has not changed, or if an operator change notification has not been made but the operator identifier has changed, the authentication flag f, which indicates whether authentication has succeeded or failed, is set to "0" (FIG. 5 / STEP 262).

[0038] On the other hand, if the determination result is positive (FIG. 5 / STEP 254: YES), that is, if an operator change has been reported and the operator identifier has changed, or if an operator change has not been reported and the operator identifier has not changed, the success or failure of authentication is determined (FIG. 5 / STEP 260). Specifically, the success or failure of authentication is determined based on whether the latest operator identifier is registered in the storage device constituting the remote control device 200 and / or an external database. If the determination result is positive (FIG. 5 / STEP 260: YES), the authentication flag f is set to "1" (FIG. 5 / STEP 261). On the other hand, if the determination result is negative (FIG. 5 / STEP 260: NO), the authentication flag f is set to "0" (FIG. 5 / STEP 262). Setting the authentication flag f ends the authentication process.

[0039] The order of the process for determining whether an operator change notification has been received and the associated process for the first flag f1 (see FIG. 5 / STEP 240, STEP 241, and STEP 242) and the process for determining whether an operator identifier has changed and the associated process for the second flag f2 (see FIG. 5 / STEP 250, STEP 251, and STEP 252) may be reversed, or they may be performed in parallel. The process for determining whether an operator change notification has been received via the remote input interface 210 and the associated process for the first flag f1 may be omitted (see FIG. 5 / STEP 240, STEP 241, STEP 242, and STEP 254). The process for determining whether an operator identifier has changed and the associated process for the second flag f2 may be omitted (see FIG. 5 / STEP 250, STEP 251, and STEP 252). The comparison process between the first flag f1 and the second flag f2 may be omitted (see FIG. 5 / STEP 254).

[0040] FIG. 8A is a diagram showing a work content specification screen in the operation system according to this embodiment. After the operator is authenticated in STEP 200 of FIG. 4 , tapping the OK button Q05 specifies or selects the operation target and work content (planned work content) ( FIG. 4 / STEP 204). Specifically, when the OK button Q05 is tapped, work content selection windows Q11 and Q12 and a back button Q13 are output to the remote output interface 220 (remote image output device 221) as shown in FIG. 8A . Each of the multiple (e.g., two) work content selection windows Q11 and Q12 displays, for example, the work name, the name of the office involved in the work, the work location, the characteristics of the work target (e.g., materials), and the work period. When the back button Q13 is tapped, the screen shown in FIG. 7A or 7B is output to the remote output interface 220 (remote image output device 221).

[0041] Figure 8B is a diagram showing an operation target designation screen in the operation system according to this embodiment. When one of the above-described work content selection windows Q11 and Q12 is tapped, as shown in Figure 8B, operation target candidate selection windows Q14 and Q15 and a back button Q16 are output to the remote output interface 220 (remote image output device 221). Each of the multiple (e.g., two) operation target candidate selection windows Q14 and Q15 displays, for example, the model number, name, and specification information of the work machine 40 that is the operation target candidate. The model number, name, specification information, etc. of the operation target candidate are associated with the actual machine identifier for identifying the actual wireless communication device 450 possessed by the work machine 40 (operation target), and are registered in a storage device constituting the remote control device 200 and / or an external database. Of the work machines 40 to be operated, only operation target candidates that have been switched by the switch to a state in which power is being supplied to the actual machine control device 400 can be selected, and operation target candidates that have not been switched to a state in which power is being supplied are displayed grayed out, and windows Q14 and Q15 are output to the remote output interface 220 (remote image output device 221) so that they will not accept selection even if they are tapped, etc. When the back button Q16 is tapped, etc., the screen shown in Fig. 8A is output to the remote output interface 220 (remote image output device 221).

[0042] Depending on the operator authenticated by the authentication process (see FIG. 4 / STEP 200), a plurality of operation target candidates may be searched for in a storage device constituting the remote control device 200 and / or an external database, and a plurality of operation target candidate selection windows showing the respective model numbers of the plurality of operation target candidates may be output to the remote output interface 220 (remote image output device 221). As a result, a new operation target can be designated by the operator through the remote input interface 210 according to his or her will from among the operation target candidates output to the remote output interface 220 (remote image output device 221) according to the operator. Note that the operation target candidates to be used by each operator are pre-registered in the storage device and / or external database according to the operator identified by the operator identifier.

[0043] A plurality of operation target candidates may be searched for in a storage device constituting the remote control device 200 and / or an external database according to the type of work content (planned work content) selected from the plurality of work content selection windows, and a plurality of operation target candidate selection windows showing the model numbers and the like of the plurality of operation target candidates may be output to the remote output interface 220 (remote image output device 221). As a result, a new operation target can be designated by the operator through the remote input interface 210 according to his / her intention from the operation target candidates output to the remote output interface 220 (remote image output device 221) according to the type of planned work content. Note that the storage device and / or external database pre-register the planned work content of each operator and the operation target candidates to be used when executing the planned work content according to the type of operator identified by the operator identifier.

[0044] Figure 8C is a diagram showing a confirmation screen for work content and operation target in the operation system according to this embodiment. When one of the operation target selection windows Q14 and Q15 is tapped, etc., a selection result confirmation window Q17, a back button Q18, and an operation start button Q19 are output to the remote output interface 220 (remote image output device 221), as shown in Figure 8C. The selection result confirmation window Q17 displays, for example, the selected work content and information about the work machine 40 that is the operation target. When the back button Q18 is tapped, etc., the screen shown in Figure 8B is output to the remote output interface 220 (remote image output device 221).

[0045] Next, an operation to start using the operation target is accepted (FIG. 4 / STEP 206). Specifically, the operation start button Q19 shown in FIG. 8C is tapped, for example.

[0046] In response to this, it is determined whether there is an existing operation target for the remote control device 200 (FIG. 4 / STEP 208). Specifically, it is determined whether there is a work machine 40 that is an operation target and for which communication has been established with the remote operation device 20. When communication with the operation target is established and / or when the established communication is released, the value of the flag may be changed, and the value of the flag may be used for this determination.

[0047] If it is determined that there is no existing operation target (NO in FIG. 4 / STEP 208), the remote control device 200 starts the execution of the communication establishment process as an "operation preparation process" (FIG. 4 / STEP 210). First, the actual machine identifier corresponding to the work machine 40 selected as the operation target by the designation command is obtained from the actual machine identifiers associated with each of a plurality of operation target candidates registered in the storage device and / or an external database.

[0048] Next, the remote control device 200 transmits a communication establishment request signal to the destination identified and indicated by the actual machine identifier corresponding to the actual machine wireless communication device 450 of the work machine 40 selected as the operation target, requesting the establishment of communication with the work machine 40 that is the operation target. The communication establishment request signal includes the operation device identifier corresponding to the remote wireless communication device 250 possessed by the remote operation device 20.

[0049] When the actual machine control device 400 mounted on the work machine 40 to be operated receives the communication establishment request signal through the actual machine wireless communication device 450, it transmits an approval signal indicating that the request for communication establishment has been approved to the remote wireless communication device 250 possessed by the remote control device 20, which is the destination identified and indicated by the operation device identifier included in the communication establishment request signal.

[0050] When the remote control device 200 receives the approval signal through the remote wireless communication device 250, it transmits an activation signal to activate the actual machine imaging device 412 and the actual machine sensor group 416 mounted on the work machine 40 to the destination identified and indicated by the actual machine identifier, and waits until it receives a signal corresponding to the actual machine operating status regarding the image acquired by the actual machine imaging device 412 through the remote wireless communication device 250 and the status of the work machine 40 acquired by the actual machine sensor group 416.

[0051] When the actual machine control device 400 receives the activation signal via the actual machine wireless communication device 450, it activates the actual machine imaging device 412 and the actual machine sensor group 416, and begins transmitting, to a destination identified and indicated by the operation device identifier, signals corresponding to the actual machine operating status relating to the images acquired by the actual machine imaging device 412 and the state of the work machine 40 acquired by the actual machine sensor group 416, and begins receiving, via the actual machine wireless communication device 450, the signals corresponding to the operation mode of the remote control lever for controlling the operation of the operating mechanism. Note that the signals corresponding to the actual machine operating status include information related to the actual machine identifier corresponding to the actual machine wireless communication device 450.

[0052] The remote control device 200 outputs the captured images and actual machine operation status received via the remote wireless communication device 250 to the remote output interface 220. The remote control device 200, for example, outputs the received captured images to a remote image output device 221, outputs the sound of the work machine 40 included in the received actual machine operation status to a remote sound output device 222, and controls the operation of the seat drive device 224 in accordance with the tilt angle included in the received actual machine operation status.

[0053] As described above, the actual machine control device 400 executes processing to control at least the communication device and establish communication with the operation target as operation preparation processing. By executing this communication establishment processing, communication is established between the remote wireless communication device 250 of the work machine 40 selected as the operation target and the remote wireless communication device 250 for remotely operating the work machine 40 (operation target) by the remote operation device 20.

[0054] Note that if there is a plan to establish communication between the work machine 40 to be operated and the remote operation device 20, the remote control device 200 may transmit a communication establishment request signal before the time to execute the communication establishment process. In this case, the communication establishment request signal includes the operation device identifier corresponding to the remote wireless communication device 250 possessed by the remote operation device 20 and information regarding the time to establish the communication establishment process. The actual machine control device 400 mounted on the work machine 40 to be operated saves the operation device identifier included in the communication establishment request signal received via the remote wireless communication device 250 in a storage device mounted on the work machine 40, reads the operation device identifier from the storage device when it is time to establish the communication establishment process, and transmits the approval signal indicating that the request for communication establishment has been approved to the remote wireless communication device 250 possessed by the remote operation device 20 that is the transmission destination identified and indicated by the read operation device identifier.

[0055] Furthermore, the operating device identifier and the actual device identifier may be assigned something other than an IP address as long as they can identify the remote wireless communication device 250 and the actual wireless communication device 450 in the second communication network. For example, a wireless LAN access point, such as an SSID (Service Set Identifier), may be assigned to the remote wireless communication device 250 and the actual wireless communication device 450. The operating device identifier and the actual device identifier may be configured to correspond to, for example, a plurality of letters, numbers, and / or symbols representing the serial number of a component, such as an engine and / or battery, that is installed on the work machine 40 together with the actual wireless communication device 450, so that the remote wireless communication device 250 and the actual wireless communication device 450 can be identified in the second communication network. For communication, an authenticator is generated in accordance with a designation method based on the operating device identifier and the actual device identifier. The authenticator may be generated in accordance with a different designation method depending on differences in the data or letters, numbers, and symbols that make up the operating device identifier and the actual device identifier. Then, the remote control device 200 uses the generated authenticator to establish wireless communication with the remote wireless communication device 250 and the actual wireless communication device 450 identified by the communication identifier.

[0056] On the other hand, if it is determined that there is an existing operation target for the remote control device 200 (YES in FIG. 4 / STEP 208), the remote control device 200 starts the execution of a communication disconnection process as an "operation preparation process" in addition to the communication establishment process (FIG. 4 / STEP 212).

[0057] In the communication establishment process in the operation preparation process, the remote control device 200 first obtains the actual machine identifier corresponding to the work machine 40 (other operation target) selected as the operation target by the specified command from the actual machine identifiers associated with each of the multiple operation target candidates registered in the storage device and / or external database.

[0058] Next, the remote control device 200 transmits a communication establishment request signal to the destination identified and indicated by the actual machine identifier corresponding to the actual wireless communication device 450 of the work machine 40 selected as the object to be operated, requesting the establishment of communication with the work machine 40 that is the object to be operated.

[0059] When the actual machine control device 400 mounted on the work machine 40 to be operated receives the communication establishment request signal through the actual machine wireless communication device 450, it transmits an approval signal indicating that the request for communication establishment has been approved to the remote wireless communication device 250 possessed by the remote control device 20, which is the destination identified and indicated by the operation device identifier included in the communication establishment request signal.

[0060] When the remote control device 200 receives the approval signal through the remote wireless communication device 250, it transmits an activation signal to activate the actual machine imaging device 412 and the actual machine sensor group 416 mounted on the work machine 40 to the destination identified and indicated by the actual machine identifier, and waits until it receives a signal corresponding to the actual machine operating status regarding the image acquired by the actual machine imaging device 412 through the remote wireless communication device 250 and the status of the work machine 40 acquired by the actual machine sensor group 416.

[0061] When the actual machine control device 400 receives the activation signal via the actual machine wireless communication device 450, it activates the actual machine imaging device 412 and the actual machine sensor group 416, and begins transmitting, to a destination identified and indicated by the operation device identifier, signals corresponding to the actual machine operating status relating to the images acquired by the actual machine imaging device 412 and the state of the work machine 40 acquired by the actual machine sensor group 416, and begins receiving, via the actual machine wireless communication device 450, the signals corresponding to the operation mode of the remote control lever for controlling the operation of the operating mechanism. Note that the signals corresponding to the actual machine operating status include information related to the actual machine identifier corresponding to the actual machine wireless communication device 450.

[0062] The remote control device 200 outputs the captured image and the actual machine operating status received through the remote wireless communication device 250 to the remote output interface 220 .

[0063] As described above, as the communication establishment process in the operation preparation process, the remote control device 200 executes the process of establishing the communication with the operation target through the communication device.

[0064] In the communication disconnection process in the operation preparation process, the remote control device 200 first stops outputting to the remote output interface 220 captured images and actual machine operating status of the work machine 40 (first work machine) that is the existing operation target, which have been received via the remote wireless communication device 250, and further stops transmitting to the work machine 40 (first work machine) a signal corresponding to the operation mode of the remote control lever to the work machine 40 (first work machine).

[0065] Next, the remote control device 200 transmits a communication disconnection request signal to the destination identified and indicated by the actual machine identifier corresponding to the actual wireless communication device 450 of the work machine 40 that is the existing target of operation, requesting that the established communication be disconnected.

[0066] When the actual machine control device 400 mounted on the existing work machine 40 to be operated receives the communication disconnection request signal via the actual machine wireless communication device 450, it executes processing to turn off the power to the actual machine imaging device 412 and actual machine sensor group 416 that have been activated. Then, the actual machine control device 400 executes processing to cut off the power supply being supplied to the actual machine control device 400.

[0067] As described above, the actual machine control device 400 executes communication disconnection processing as operation preparation processing, which cancels the established communication state between the remote operation device 20 and the work machine 40, with which communication is established. Note that during this communication disconnection processing, a powered state to the actual machine control device 400 mounted on the work machine 40 may be maintained (processing to cut off the power supply to the actual machine control device 400 may not be performed). This allows the actual machine control device 400 to receive, via the actual machine wireless communication device 450, the signals corresponding to the operation mode of the remote operation lever for controlling the operation of the actuation mechanism, and when the actual machine imaging device 412 and the actual machine sensor group 416 are started again, a state may be maintained in which it is possible to transmit, to the remote operation device 20, via the actual machine wireless communication device 450, images acquired by the actual machine imaging device 412 and signals corresponding to the actual machine operating status relating to the state of the work machine 40 acquired by the actual machine sensor group 416.

[0068] The communication establishment processing and the communication disconnection processing may be processed in parallel or in parallel. When the communication establishment processing and the communication disconnection processing are processed in parallel or in parallel, the remote control device 200, while performing the processing to acquire the actual machine identifier corresponding to the work machine 40 (another operation target), simultaneously starts processing to stop the output of captured images and actual machine operating status of the work machine 40 (one work machine) to the remote output interface 220 and to stop the transmission to the work machine 40 (one work machine) of signals corresponding to the operation mode of the remote operation lever for the work machine 40 (one work machine). The remote control device 200 then proceeds with the communication establishment processing and the communication disconnection processing in parallel or in parallel.

[0069] Here, it does not matter which is executed first: the process of turning off the power to the actual machine imaging device 412 and the actual machine sensor group 416 of the work machine 40 (first work machine), which is the existing operation target, or the process of starting up the actual machine imaging device 412 and the actual machine sensor group 416 of the work machine 40 (second operation target). In other words, the remote control device 200 may temporarily be in a state where it receives captured images and actual machine operating statuses from both the work machine 40 (first work machine), which is the existing operation target, and the work machine 40 (second operation target). In this case, the remote control device 200 selects the captured images and actual machine operating statuses of the work machine 40 (second operation target) by reading the actual machine identifier of the sender attached to the signal representing the received captured images and actual machine operating statuses, and outputs them to the remote output interface 220. Furthermore, the start time of the communication establishment process and the start time of the communication disconnection process may be the same or different. The end time of the communication establishment process and the end time of the communication disconnection process may be the same or different. The communication establishment process and the communication disconnection process may be serially processed (executed sequentially).

[0070] After the execution of the operation preparation process has begun, a designated item setting process is executed in which the contents of designated items for assisting the operation of the work machine 40, which is the operation target of the remote control device 20, are set (FIG. 4 / STEP 214). If it is determined that no existing operation target exists (FIG. 4 / STEP 208: NO), the designated item setting process is executed in parallel or in parallel with the execution of the communication establishment process (FIG. 4 / STEP 210) as the operation preparation process, and for example, is started simultaneously with the start of the execution of the communication establishment process. If it is determined that an existing operation target exists (FIG. 4 / STEP 208: YES), the designated item setting process is executed in parallel or in parallel with the execution of the communication establishment process and communication disconnection process (FIG. 4 / STEP 212) as the operation preparation process, and for example, is started simultaneously with the start of the execution of the communication establishment process and communication disconnection process. Next, the remote control device 200 determines whether the operation preparation process has been completed (FIG. 4 / STEP 216). If it is determined that the operation preparation process has not been completed (NO in FIG. 4 / STEP 216), the designated item setting process continues to be executed (FIG. 4 / STEP 214).

[0071] The designated item setting process will be described in detail below. Fig. 9A is a diagram showing a first lever pattern setting screen in the operation system according to this embodiment. Fig. 9B is a diagram showing a second lever pattern setting screen in the operation system according to this embodiment. In the designated item setting process, first, for example, a menu screen for selecting multiple designated items is output to the remote output interface 220 (remote image output device 221). When "lever pattern" is selected as a designated item on the menu screen, the first lever pattern setting screen shown in Fig. 9A is output to the remote output interface 220 (remote image output device 221).

[0072] As shown in Fig. 9A, the first lever pattern setting screen includes a first operation guide window Q21, a lever pattern window Q22, an OK button Q23, and a change button Q24. The first operation guide window Q21 displays information guiding the operation method for setting a lever pattern. The lever pattern window Q22 displays a provisionally set lever pattern. When the OK button Q23 is tapped by the operator, the lever pattern displayed in the lever pattern window Q22 is set permanently. When the change button Q24 is tapped by the operator, the second lever pattern setting screen shown in Fig. 9B is output to the remote output interface 220 (remote image output device 221).

[0073] The finally set lever pattern may be associated with an operator identifier (including image and distance measurement data for face authentication) and / or an actual machine identifier and stored in a storage device and / or database constituting the remote control device 200. The operator identifier used in the authentication process and / or the actual machine identifier used in specifying the operation target may be used to search for or read out the last finally set lever pattern from the storage device and then displayed as a temporary lever pattern by default in the lever pattern window Q22. The finally set lever pattern may simply be stored in a storage device and / or an external database constituting the remote control device 200, read out from the storage device, etc., and then displayed as a temporary lever pattern by default in the lever pattern window Q22.

[0074] As shown in FIG. 9B , the second lever pattern setting screen includes a second operation guide window Q25 and multiple (e.g., four) lever pattern candidate windows Q26, Q27, Q28, and Q29. The second operation guide window Q25 displays information guiding the selection of one of the lever pattern candidate windows Q26, Q27, Q28, and Q29 to set a lever pattern. Each of the lever pattern windows Q26, Q27, Q28, and Q29 displays a lever pattern candidate different from the provisionally set lever pattern. In response to a selective tap or the like on one of the lever pattern candidate windows Q26, Q27, Q28, and Q29, the first lever pattern setting screen is output to the remote output interface 220 (remote image output device 221) (see FIG. 9A ). At this time, the selected lever pattern candidate is displayed in the lever pattern window Q22 as the provisionally set lever pattern.

[0075] In addition, the first lever pattern setting screen and the second lever pattern setting screen may be output to the remote output interface 220 (remote image output device 221) as a single screen or as three or more transitional screens.

[0076] FIG. 10 is a diagram showing a lever sensitivity setting screen in the control system according to this embodiment. When "lever sensitivity" is selected as a specification item on the menu screen, the lever sensitivity setting screen shown in FIG. 10 is output to the remote output interface 220 (remote image output device 221). As shown in FIG. 10, the lever sensitivity setting screen includes an operation guide window Q31, a lever sensitivity adjustment bar Q32, and an OK button Q33. The operation guide window Q31 displays information guiding the operation method for setting the lever sensitivity. The lever sensitivity adjustment bar Q32 includes a measure indicating the lever sensitivity (which determines the magnitude of the displacement amount or the displacement speed of the controlled object according to the lever operation amount) and, for example, a triangular slider. The lever sensitivity is provisionally set by adjusting the position of the slider along the measure. When the OK button Q33 is tapped, the lever sensitivity displayed in the lever sensitivity adjustment bar Q32 is set permanently.

[0077] The finally set lever sensitivity may be associated with the operator identifier and / or the actual device identifier and stored in a storage device constituting the remote control device 200 and / or an external database. The operator identifier used in the authentication process and / or the actual device identifier used in specifying the operation target may be used to search for or read out the last actually set lever sensitivity from the storage device as a provisionally set lever sensitivity, and the default position of the slider in the lever sensitivity adjustment bar Q32 may be adjusted. The finally set lever sensitivity may simply be stored in a storage device constituting the remote control device 200 and / or an external database, and may be read out from the storage device or the like to adjust the default position of the slider in the lever sensitivity adjustment bar Q32.

[0078] FIG. 11 is a diagram showing a screen for setting the seat vibration mode in the operation system according to this embodiment. When "Seat Vibration" is selected as a specification item on the menu screen, the seat vibration setting screen shown in FIG. 11 is output to the remote output interface 220 (remote image output device 221). As shown in FIG. 11, the seat vibration setting screen includes an operation guide window Q41, a seat vibration intensity selection window Q42, and an OK button Q43. The operation guide window Q41 displays information guiding the operation method for setting the vibration intensity (including whether or not vibration is to be applied) of the seat 24 (or the seat device). The seat vibration intensity selection window Q42 includes multiple buttons, and by selectively tapping one of the multiple buttons, the vibration intensity of the seat device is provisionally set. When the OK button Q43 is tapped, the vibration intensity of the seat device corresponding to that button is finally set.

[0079] The actually set vibration intensity may be associated with the operator identifier and / or the actual machine identifier and stored in a storage device constituting the remote control device 200 and / or an external database. The operator identifier used in the authentication process and / or the actual machine identifier used in specifying the operation target may be used to search for or read out the last actually set vibration intensity from the storage device, and then displayed as a provisional vibration intensity by default in the seat vibration intensity selection window Q42. The actually set vibration intensity may simply be stored in a storage device constituting the remote control device 200 and / or an external database, and then read out from the storage device or the like and displayed as a provisional vibration intensity by default in the seat vibration intensity selection window Q42.

[0080] FIG. 12 is a diagram showing a setting screen for operation assistance in the operation system according to this embodiment. When "operation assistance" is selected as a specification item on the menu screen, the operation assistance setting screen shown in FIG. 12 is output to the remote output interface 220 (remote image output device 221). As shown in FIG. 12, the operation assistance setting screen includes an operation guide window Q51, an operation assistance window Q52, and an OK / ON / OFF button Q53. The operation guide window Q51 displays information guiding the operation method for setting whether or not operation assistance is enabled. The operation assistance window Q52 includes an ON button and an OFF button for temporarily setting whether or not operation assistance is enabled or whether or not it is enabled. When the OK button Q53 is tapped, for example, operation assistance is officially enabled. When operation assistance is officially enabled, the operation content assisted by operation assistance may be output to the remote output interface 220 (remote image output device 221). The operation content is, for example, a horizontal pulling operation in which the bucket, which is the tip attachment 445, is moved linearly along the ground on which the undercarriage 41 of the work machine 40 is in contact, and is an operation in which the bucket is moved from a first position that is away from the upper rotating body 42 in the forward direction to a second position that is closer to the upper rotating body 42 than the first position. There may be multiple types of operation content assisted by the operation assist, and the configuration may be such that it is possible to set whether or not operation assist is provided for the multiple types of operation content.

[0081] Furthermore, a simulation image showing an operating situation in which the work machine 40 is assisted by operation assistance may be output to the remote output interface 220 (remote image output device 221). The remote control device 200 executes the following processes: a process of recognizing the operation mode of the remote input interface 210 (remote operation mechanism 211) by the operator; a process of correcting the recognized operation mode so as to approach a target operation mode; a process of recognizing a motion mode of the work machine 40 in virtual space based on the corrected operation mode; and a process of outputting an image that simulates a state in which a change in posture of the work machine 40 is reflected in accordance with the recognized motion mode to the remote output interface 220 (remote image output device 221) as the simulation image. For example, when the OK button Q53 is tapped, etc., a simulation image showing an operating situation in which the work machine 40, which is the operation target in virtual space, is assisted by operation assistance, is output to the remote output interface 220 (remote image output device 221). When the operator operates work machine 40 in the virtual space via remote input interface 210 (remote operation mechanism 211), the work machine 40 moves in accordance with the operation mode of remote input interface 210 (remote operation mechanism 211) in the simulation image, and changes in the position and attitude of tip attachment 445 and the like are shown. Therefore, remote control device 200, remote input interface 210 (remote operation mechanism 211), and remote output interface 220 (remote image output device 221) constitute a "simulation device." Here, when the operator operates via remote input interface 210 (remote operation mechanism 211) to move the bucket closer to a second position that is closer to the upper rotating body 42 than the first position in the virtual space, the simulation image shows the tip of the bucket moving linearly from the first position to the second position.Specifically, when the operator performs an operation via remote input interface 210 (remote operation mechanism 211) to rotate boom 441 upward while rotating arm 443 downward and further rotate the bucket so that the tip of the bucket faces downward, the operation is corrected (the amount of deviation of the bucket tip from the target trajectory is reduced) so that the tip of the bucket does not deviate more than a predetermined distance from a target trajectory that linearly connects the first position to the second position, and this is reflected in the simulation image. The first position, second position, and target trajectory may be visualized and shown in the simulation image, and the first position and second position may be set to any position by operation of the operator (for example, by moving the tip of the bucket to a position desired to be the first position or the second position and pressing a confirm button to set the position). Furthermore, after communication is established between remote operation device 20 and work machine 40 that is the target of operation and a state in which remote operation of work machine 40 by remote operation device 20 is possible, a remote operation simulation of work machine 40 in virtual space may be permitted while the remote operation of the work machine 40 is stopped or temporarily stopped.

[0082] FIG. 13 is a diagram showing a guidance setting screen in the operation system according to this embodiment. When "Guidance (Operation Assistance Image)" is selected as a specification item on the menu screen, the operation assistance image setting screen shown in FIG. 13 is output to the remote output interface 220 (remote image output device 221). As shown in FIG. 13, the operation assistance image setting screen includes an operation guide window Q61, a depth perception image window Q62, a terrain image window Q63, and an OK button Q64. The operation guide window Q61 displays information guiding the operation method for setting the display mode of the operation assistance image in the remote output interface 220 (remote image output device 221). The depth perception image window Q62 includes an ON button and an OFF button for temporarily setting whether or not to display the depth perception image in the remote output interface 220 (remote image output device 221) or whether or not to display it. The terrain image window Q63 includes an ON button and an OFF button for temporarily setting whether or not to display a terrain image in the remote output interface 220 (remote image output device 221) or whether or not to display the terrain image. When the OK button Q64 is tapped, etc., the setting of whether or not to display a depth perception image and whether or not to display a terrain image is permanently set.

[0083] The currently set display status of the depth sense image and / or terrain image may be associated with the operator identifier and / or the actual machine identifier and stored in a storage device constituting the remote control device 200 and / or an external database. The operator identifier used in the authentication process and / or the actual machine identifier used in specifying the operation target may be used to search or read the last currently set display status (ON / OFF) from the storage device, and then displayed by default in the depth sense image window Q62 and / or the terrain image window Q63. The currently set display status of the depth sense image and / or the terrain image may simply be stored in a storage device constituting the remote control device 200 and / or an external database, and then read from the storage device, etc., and displayed by default in the depth sense image window Q62 and / or the terrain image window Q63.

[0084] Furthermore, a simulation image showing a depth perception image and / or a terrain image may be output to the remote output interface 220 (remote image output device 221). For example, when the display of the depth perception image is set to ON, a simulation image showing the operating status of the work machine 40, which is the operation target in the virtual space, assisted by the operation assist is output to the remote output interface 220 (remote image output device 221). Then, when the operator operates the work machine 40 in the virtual space via the remote input interface 210 (remote operation mechanism 211), the simulation image shows how the work machine 40 operates in accordance with the operation mode, and how the position and attitude of the tip attachment 445 and the like change. Therefore, the remote control device 200, the remote input interface 210 (remote operation mechanism 211), and the remote output interface 220 (remote image output device 221) constitute a "simulation device." Here, in the simulation image showing the operating status of the work machine 40, a depth perception image for showing the position or depth perception of one or more locations defined in the work mechanism 44 is displayed. 14 is a diagram showing a depth perception image in the operation system according to this embodiment. For example, as shown in FIG. 14 , a mark M is displayed on the simulation image. A mark M is displayed indicating the position of a second specified point p2 resulting from vertical projection of a first specified point p1, which is the left-right center point of the base end of the tip attachment 445. The first specified point p1 may be specified as a point other than the left-right center point of the base end of the tip attachment 445 (such as the tip of the tip attachment 445). The second specified point p2 may be displayed a predetermined distance away from the first specified point p1 in the virtual space, or may be displayed at any height in the virtual space.

[0085] Figure 15 is a diagram showing a terrain image in the operation system according to this embodiment. When terrain image display is set to ON, a terrain image showing the terrain around (forward of) the work machine 40 is superimposed on a simulation image showing the operating status of the work machine 40. This causes, for example, a mesh-like terrain image showing the terrain or unevenness of the ground surface to be displayed in the simulation image, as shown in Figure 15.

[0086] Fig. 16A is a diagram showing a first button function setting screen in the operation system according to this embodiment. When "Button assignment" is selected as a specification item on the menu screen, the first button function setting screen shown in Fig. 16A is output to the remote output interface 220 (remote image output device 221). As shown in Fig. 16A, the first button function setting screen includes an operation guide window Q71, a button layout window Q72, a function assignment status window Q73, and an OK button Q74. The operation guide window Q71 displays information guiding the user through the operation method for assigning functions to the multiple buttons that make up the remote operation mechanism 211. Functions that can be assigned to the buttons include a function to adjust the angle of the actual machine imaging device 412 left and right, a function to adjust the angle of the actual machine imaging device 412 up and down, a zoom-in / zoom-out function for the actual machine imaging device 412, a function to switch on / off an audio output device (for example, a horn that issues an alarm to the surrounding area) mounted on the work machine 40, a setting function for calibrating the operation lever that makes up the remote operation mechanism 211, a function to switch between increasing and decreasing the pressure of the hydraulic oil for the work mechanism 44, a function to switch between exciting and releasing the lifting magnet that serves as the tip attachment 445, and a function to release the work machine 40 from a temporary stop state. The button layout window Q72 shows the layout of multiple buttons to which functions can be assigned. For example, the button layout window Q72 shows the layout of a total of 10 buttons, with buttons L1, L2, and L3 arranged on the left travel lever 2110, buttons L4 and L5 arranged on the left operation lever 2111, buttons R1, R2, and R3 arranged on the right travel lever 2110, and buttons R4 and R5 arranged on the right operation lever 2112. The function assignment status window Q73 shows a chart showing the functions assigned to each of the multiple buttons. When the OK button Q74 is tapped, the functions assigned to each button shown in the function assignment status window Q73 are finally set.

[0087] The functions assigned to each button after the actual settings may be associated with an operator identifier and / or an actual device identifier and stored in a storage device constituting the remote control device 200 and / or an external database. The operator identifier used in the authentication process and / or the actual device identifier used in specifying the operation target may be used to search for or read out the functions assigned to each button after the actual settings from the storage device, and then displayed by default in the function assignment status window Q73. The functions assigned to each button after the actual settings may simply be stored in a storage device constituting the remote control device 200 and / or an external database, and then read out from the storage device, etc., and then displayed by default in the function assignment status window Q73.

[0088] FIG. 16B illustrates a second button function setting screen in the operation system according to this embodiment. When a button selected from the plurality of buttons displayed in the button layout window Q72 (or the function assignment status window Q73) is tapped, the second button function setting screen illustrated in FIG. 16B is output to the remote output interface 220 (remote image output device 221). As illustrated in FIG. 16B , the second button function setting screen includes a button window Q75, a button function window Q76, a button function selection window Q77, and an OK button Q78. The button window Q75 displays (highlighted as appropriate) the name and / or position of a button selected from the plurality of buttons. The button function window Q76 displays the function assigned to the selected button. The button function selection window Q77 displays a list of functions assignable to the selected button (e.g., in the form of a pull-down menu). When the OK button Q78 is tapped, the first button function setting screen is output to the remote output interface 220 (remote image output device 221) (see FIG. 16A).

[0089] In addition, if the work machine 40 is equipped with multiple actual imaging devices 412 with different imaging ranges (which may partially overlap), ``screen allocation'' may be selected as a specification item on the menu screen, and a correspondence may be set between each of the multiple image output devices (displays or monitors) that make up the remote image output device 221 and the captured images obtained through each of the multiple actual imaging devices 412 that are displayed on each image output device.

[0090] As described above, various designated item setting processes are executed. If it is determined that the operation preparation process that was being executed has been completed (YES in FIG. 4 / STEP 216), the remote control device 200 in the remote operation device 20 executes remote operation process for remotely operating the work machine 40 (FIG. 4 / STEP 218). If the designated item setting process for a certain designated item is in progress when the operation preparation process is completed, that designated item setting process may be executed continuously (FIG. 4 / STEP 214).

[0091] On the condition that communication for remotely operating the work machine 40 (operation target) by the remote operation device 20 has been established, the actual machine control device 400 in the work machine 40 transmits the captured image acquired via the actual machine imaging device 412 to the remote operation device 20. The transmitted captured image (or captured image data) may be the captured image itself, or image data representing a simulated environmental image generated based on the captured image.

[0092] For example, in the remote operation device 20, the remote control device 200 may establish communication for remotely operating the work machine 40 (operation target) in response to transmitting a designation signal to the work machine 40 via the remote wireless communication device 250. If multiple work machines 40 are multiple operation target candidates, a designation signal may be transmitted to each of the multiple work machines 40. The presence or absence of a designation operation via the remote input interface 210 may be determined by the operator, and if the determination result is positive, a designation signal may be transmitted. A "designation operation" is, for example, an operation such as tapping on the remote input interface 210 for the operator to designate the work machine 40 that he or she intends to remotely operate.

[0093] When the remote control device 20 receives a captured image via the remote wireless communication device 250, the captured image or the work environment image is output to the remote image output device 221 (display device) (FIG. 6 / STEP 280). As a result, for example, an image showing the ground spreading out in front of the cab 43, as well as the bucket corresponding to the tip attachment 445 that is part of the work mechanism 44, the ground, and the like, is output to the remote image output device 221.

[0094] When the "button assignment" designation is selected in the designation item setting process and the function assignment of each button is finally set, the button assignment is applied. As a result, the range and / or scale of the captured image can be adjusted by the operator operating the button to which the angle adjustment and / or zoom-in / zoom-out functions of the actual imaging device 412 are assigned (see FIG. 4 / STEP 214, FIGS. 16A and 16B).

[0095] When the designated item "guidance" is selected in the designated item setting process and the display of the depth perception image is set to "ON," a mark M as a depth perception image may be superimposed on the captured image (see FIG. 4 / STEP 214, FIGS. 13 and 14). For example, the remote control device 200 calculates the position coordinates of a first designated point p1, which is the left-right center point of the base end of the tip attachment 445, from the hoisting angle of the boom 441, the rotation angle of the arm 443, and the rotation angle of the tip attachment 445, and displays a mark M superimposed on the position of a second designated point p2 obtained by projecting the position coordinates vertically downward. When the designated item "guidance" is selected in the designated item setting process and the display of the terrain image is set to "ON," a mesh as a terrain image may be superimposed on the captured image (see FIG. 4 / STEP 214, FIGS. 13 and 15). The mesh to be superimposed is generated, for example, based on the results of the distance from the work machine 40 to the surrounding ground surface obtained by a sensing device mounted on the work machine 40 that measures the distance to surrounding objects.

[0096] Next, the remote control device 200 recognizes the operation mode of the remote operation mechanism 211 (FIG. 6 / STEP 281). Specifically, when the designated item "lever pattern" is selected in the designated item setting process and the lever pattern is finally set, the operation mode of the remote operation mechanism 211 or the operating levers 2110, 2111, and 2112 that constitute it is recognized in accordance with the lever pattern (see FIG. 4 / STEP 214, FIGS. 9A, and 9B). When the designated item "lever sensitivity" is selected in the designated item setting process and the lever sensitivity is finally set, the operation mode of the remote operation mechanism 211 or the operating levers 2110, 2111, and 2112 that constitute it is recognized in accordance with the lever sensitivity (see FIG. 4 / STEP 214 and FIG. 10).

[0097] Next, the remote control device 200 determines whether or not the authentication flag f (see FIG. 5 / STEP 261 and STEP 262) is 1 (FIG. 6 / STEP 282). If the authentication flag f is set to "1" in response to the success of the operator authentication processing (YES in FIG. 6 / STEP 282 (YES in FIG. 5 / STEP 260 → see STEP 261)), a remote operation command corresponding to the operation mode is transmitted to the work machine 40 via the remote wireless communication device 250 (FIG. 6 / STEP 284). If the authentication flag f is set to "0" in response to the failure of the operator authentication processing (NO in FIG. 6 / STEP 282 (NO in FIG. 5 / STEP 254, NO in STEP 260 → see STEP 262)), transmission of the remote operation command corresponding to the operation mode to the work machine 40 via the remote wireless communication device 250 is suspended, or only a part of the remote operation command is transmitted (FIG. 6 / STEP 286).

[0098] Then, the time series of the manner in which the operator operated the remote operation mechanism 211 is sequentially acquired as the operation history of the work machine 40 in association with the operator identifier and / or the actual machine identifier, and is stored and held in a storage device constituting the remote control device 200 and / or an external database (FIG. 6 / STEP 288). If the execution of remote operation processing for remotely operating the work machine 40 by the remote control device 200 (FIG. 4 / STEP 218) continues, the processing of the operation processing function (FIG. 6 / STEP 280 → ... →FIG. 6 / STEP 288) is repeated.

[0099] In the work machine 40, when the actual machine control device 400 receives a remote operation command via the actual machine wireless communication device 450, the operation of the work mechanism 44, etc. is controlled in accordance with the remote operation command. For example, while holding an object such as earth and sand with a bucket serving as the tip attachment 445, a predetermined task is performed that involves the operation of the work mechanism 44, etc. in order to displace the position of the bucket in real space.

[0100] When the designated item "operation assist" is selected in the designated item setting process and operation assist is officially set to "enabled (ON)," specific operation content is assisted based on that setting. For example, when a horizontal pulling operation is assisted based on that setting, when the operator operates via remote input interface 210 (remote operation mechanism 211) to move the bucket from a first position that is away in the forward direction from upper revolving body 42 of work machine 40 (the object of operation) to a second position that is closer to upper revolving body 42 than the first position, the tip of the bucket of work machine 40 moves linearly from the first position to the second position, and a captured image or work environment image capturing this movement is output to remote image output device 221 (display device). Specifically, when the operator performs an operation through the remote input interface 210 (remote operation mechanism 211) to rotate the boom 441 upward while rotating the arm 443 downward and further rotate the bucket so that the tip of the bucket faces downward, the operation is corrected (the amount of deviation of the bucket tip from the target trajectory is reduced) so that the tip of the bucket does not deviate more than a predetermined distance from a target trajectory that linearly connects the first position to the second position, and this is reflected in the operation of the work mechanism 44. The first position, the second position, and the target trajectory may be superimposed on a captured image or an image of the work environment and output to the remote image output device 221 (display device), and the first position and the second position may be set to any position by the operator's operation (for example, the position can be set by moving the tip of the bucket to a position desired as the first position or the second position and pressing a confirm button).

[0101] In the work machine 40, the actual machine control device 400 transmits the actual machine operating status acquired through the actual machine sensor group 416 to the remote operation device 20. The actual machine operating status includes the actual machine attitude (the tilt angle of the upper rotating body 42 about the pitch axis and / or roll axis relative to the vertical direction), whether the prime mover is operating, and the attitude of the work mechanism 44. When the remote operation device 20 receives the actual machine operating status through the remote wireless communication device 250, the remote control device 200 controls the operation of the seat drive device 224 to adjust the attitude of the seat 24 so that the seat attitude is aligned with or coincides with the actual machine attitude. At this time, high-frequency components exceeding a threshold may be cut out of the frequency components representing the time-varying manner of the actual machine attitude, and the time-varying manner of the seat attitude may be adjusted in accordance with only low-frequency components below the threshold.

[0102] When remote control device 200 determines, based on the actual machine operating status, that the prime mover of work machine 40 is operating, seat drive device 224 may vibrate seat 24. By vibrating seat 24 around the pitch axis and / or roll axis with a predetermined minute amplitude and at a frequency corresponding to the frequency of the prime mover, the attitude of seat 24 changes slightly with respect to the designated attitude. Alternatively, by vibrating seat 24 in a direction parallel to at least one of the pitch axis, roll axis, and yaw axis with a predetermined minute amplitude and at a frequency corresponding to the frequency of the prime mover, the attitude of seat 24 changes slightly in at least one of the forward / backward, left / right, and up / down directions while being maintained in the designated attitude.

[0103] When the specified item "seat vibration" is selected in the specified item setting process and the vibration intensity of the seat 24 (or seat device) is set, the operating speed and amplitude of the seat drive device 224, and ultimately the speed and amount of change in the posture of the seat 24, are adjusted according to the intensity (see Figure 4 / STEP 214 and Figure 11).

[0104] In the remote operation device 20, the remote control device 200 determines whether or not the operation end condition for the work machine 40 has been satisfied (FIG. 4 / STEP 220). The operation end condition is a condition for determining that work is not being performed using the work machine 40 that is the object of operation.

[0105] The operation termination conditions include, for example, a condition that the operation of the prime mover of the work machine 40 has been stopped (or an operation to stop the operation of the prime mover of the work machine 40 has been performed via the remote input interface 210). Note that the operation termination conditions may also include a condition that the control lever, travel pedal, and / or option pedal constituting the remote input interface 210 have been returned to the neutral position or the initial position (and that state has been maintained for a certain period of time) to cause the work machine 40 to end a predetermined operation of an operating mechanism such as the work mechanism 44. In addition, the operation termination conditions may include conditions such as whether or not a shut-off lever has been operated to the lock position, whether or not an operation to stop the operation of the prime mover of the work machine 40 has been performed via the remote input interface 210, and whether or not a switch constituting the remote input interface 210 has been switched from the ON state to the OFF state to cause the work machine 40 to end a predetermined operation of an operating mechanism such as the work mechanism 44, and so on, and detection may be performed based on whether or not multiple conditions are satisfied.

[0106] If it is determined that the operation end condition is not satisfied (NO in FIG. 4 / STEP 220), remote operation of the work machine 40 by the remote control device 20 continues (FIG. 4 / STEP 218).

[0107] On the other hand, if it is determined that the operation termination condition is satisfied (YES in FIG. 4 / STEP 220), the remote control device 200 determines the operation content by the operator via the remote input interface 210 (FIG. 4 / STEP 222).

[0108] Fig. 17 is a diagram showing an operation selection screen in the operation system according to this embodiment. Specifically, the operation selection screen shown in Fig. 17 is output to the remote output interface 220. The operation selection screen shows an operation resume button Q81, an operation switch button Q82, and an operation end button Q83 that constitute the remote input interface 210. The operation resume button Q81 constitutes an input interface for resuming remote operation, via the remote operation device 20, of a work machine 40 that is an operation target and for which communication has been established with the remote operation device 20. The operation switch button Q82 constitutes an input interface for starting remote operation, via the remote operation device 20, of a work machine 40 (second work machine) that is an operation target that is different from the work machine 40 (first work machine) that is an operation target and for which communication has been established with the remote operation device 20. The operation end button Q83 constitutes an input interface for ending remote operation, via the remote operation device 20, of a work machine 40 that is an operation target and for which communication has been established with the remote operation device 20.

[0109] When the operator taps the operation resume button Q81 (when the operation content is an operation to resume remote operation of the work machine 40, which is the operation target and has established communication with the remote control device 20, via the remote control device 20) (FIG. 4 / STEP 222-1), remote operation of the work machine 40 by the remote control device 20 continues (FIG. 4 / STEP 218).

[0110] When the operation switch button Q82 is tapped or the like (when the operation content is an operation to start remote operation, via the remote operation device 20, of a work machine 40 (second work machine) that is an operation target other than the work machine 40 (first work machine) that is an operation target and for which communication has been established with the remote operation device 20) (FIG. 4 / STEP 222-2), the authentication process (FIG. 4 / STEP 200) and subsequent processes are repeated. In this case, because there is a work machine 40 (first work machine) that is an operation target and for which communication has been established with the remote operation device 20, communication disconnection processing with the first work machine and communication establishment processing with the new operation target work machine 40 (second work machine) are executed as "operation preparation processing" (FIG. 4 / STEP 208: YES → STEP 212).

[0111] If the authentication flag f is set to "0" during the authentication process (see FIG. 5 / STEP 254: NO, STEP 260: NO → STEP 262), the operation resume button Q81 and / or the operation switch button Q82 may not be displayed, or operations such as tapping may be prohibited.

[0112] When the operation end button Q83 is tapped or the like (when the operation content is an operation to end remote operation of the work machine 40, which is the operation target and in a state of established communication with the remote control device 20, via the remote control device 20) (FIG. 4 / STEP 222-3), processing to disconnect communication between the remote control device 20 and the work machine 40, which is the operation target, is executed (FIG. 4 / STEP 224), and remote operation of the work machine 40 by the remote control device 20 ends.

[0113] (Action and Effect) With the remote control device 20 that performs the above functions and the operation system configured thereby, it is possible to set the content of the designated items to assist in the operation of the work machine 40 that is the object of operation by the remote control device 20 after the communication establishment process is completed, during the execution of the communication establishment process as the operation preparation process (FIG. 4 / STEP 208: NO → STEP 210 → STEP 214, see FIGS. 9A, 9B, 10, 11, 12, 13, 16A and 16B). Therefore, the waiting time during the execution of the communication establishment process between the remote control device 20 and the work machine 40 that is the object of operation can be utilized. As a result, it is also possible to reduce the stress of the operator during this waiting time. It is noted that the setting of the content of the designated items (designated item setting process) only needs to be executed during at least the execution of the communication establishment process.

[0114] When the operation target is switched from one work machine 40 (first operation target / first work machine) to another work machine 40 (second operation target / second work machine) through the communication device, communication disconnection processing and communication establishment processing are executed as operation preparation processing (see FIG. 4 / STEP 208: YES → STEP 212). When the communication disconnection processing and the communication establishment processing are executed, it becomes possible to set the content of the specified items through the remote input interface 210 (see FIG. 4 / STEP 214). Therefore, the waiting time during communication disconnection processing and communication establishment processing for switching the operation target can be utilized.

[0115] It is possible to virtually experience, through the simulation device, the assistance modes and / or operation modes of the work machine 40 that is the object of operation, which are predicted to occur in accordance with the settings of the specified items through the remote input interface 210 (see FIGS. 14 and 15). Based on the experience, it is possible to reset or change the settings of the specified items through the remote input interface 210. As described above, the simulation device is made up of the remote control device 200, the remote input interface 210 (remote operation mechanism 211), and the remote output interface 220 (remote image output device 221).

[0116] The settings of the specified items are stored in a storage device or the like constituting the remote control device 200, and the settings of the specified items stored in the storage device or the like are output to the remote output interface 220 (remote image output device 221) (see FIGS. 9A / Q22, 10 / Q32, 11 / Q42, 12 / Q52, 13 / Q62, Q63, and 16A / Q73). The current settings of the specified items output to the remote output interface 220 can be confirmed by an operator or the like. Based on the output contents, the contents of the specified items can be reset or changed via the remote input interface 210 (see FIGS. 9B / Q26-Q29, 10 / Q32, 11 / Q42, 12 / Q52, 13 / Q62, Q63, and 16B / Q75-Q78).

[0117] According to the remote control device 20 that performs the above functions and the operation system configured thereby, when the operation target is switched from a work machine 40 (first work machine) as one operation target to a work machine 40 (second work machine) as another operation target, communication disconnection processing and communication establishment processing are executed in parallel (see FIG. 4 / STEP 208: YES → STEP 210). The "communication disconnection processing" is processing for disconnecting communication that had been established between the remote control device 20 and the work machine 40 as one operation target. The "communication establishment processing" is processing for establishing communication between the remote control device 20 and the work machine 40 as the other operation target. This allows for smooth switching of the operation target.

[0118] The operator can, in accordance with his or her will, designate a work machine 40 as a new operation target through the remote input interface 210. More specifically, the operator can, in accordance with his or her will, cancel the designation of the work machine 40 that had been the operation target through the remote input interface 210, and then designate a new work machine 40 as the operation target (see Figure 4 / STEP 222-2 →STEP 204 and Figure 17 / Q82 → ... → Figure 8B). This can prevent the operation target from being erroneously switched when the operator does not intend to switch the operation target.

[0119] When a work machine 40 that is an operation target with established communication is designated as a new operation target via the remote input interface 210, the execution of the communication disconnection processing and communication establishment processing is put on hold (FIG. 4 / STEP 222-1 → STEP 218 and FIG. 17 / Q81). If the work machine 40 designated as the new operation target is the same as the work machine 40 that was previously designated, the communication that was already established with that work machine 40 continues to be used. This eliminates the need to execute the communication disconnection processing and communication establishment processing again when designating (re-designating) a work machine 40 that is a new operation target, and operation of the operation target can be resumed smoothly.

[0120] When the operator issues a command to end operation of the work machine 40 that is the object of operation via the remote input interface 210 in accordance with his or her will, communication disconnection processing is executed (see FIG. 4 / STEP 222-3 →STEP 224 and FIG. 17 / Q83). This can prevent a situation in which communication established between the remote control device 20 and the work machine 40 that is the object of operation is accidentally disconnected.

[0121] With a remote control device 20 that performs the above functions and an operation system configured thereby, the operation history of the work machine 40 that is the operation target can be acquired separately for each operator identifier, and ultimately for each operator identified by that operator identifier, and can be stored and retained in a database or the like (see FIG. 6 / STEP 288). When the operation target of the remote control device 20 is switched from one work machine 40 (first work machine) to another work machine 40 (second work machine) (see FIG. 4 / STEP 222-2 → ... → STEP 204 and FIG. 8B / Q14, Q15), the operation history of that other work machine 40 is acquired together with the operator identifier (see FIG. 6 / STEP 288). This allows the operation history of each different operation target (work machine 40) by the operator to be acquired separately.

[0122] Furthermore, the presence or absence of an instruction to switch the operation object is determined on the condition that no work is being performed using the work machine 40 that is the operation object (see FIG. 4 / STEP 220: YES -> STEP 222). Therefore, when work is being performed using a work machine 40 that is one operation object, it is possible to avoid a situation in which the one operation object is switched to another operation object and the work is stopped against the operator's intention.

[0123] When an operator is replaced by another operator, the operation history of the work machine 40 that is the operation target is acquired together with an operator identifier for identifying the new operator (see FIG. 5 / STEP 202 → ... → YES in STEP 250, FIG. 6 / STEP 288). This makes it possible to distinguish and acquire the operation history of the work machine 40 that is the operation target for each different operator.

[0124] Furthermore, when a notification of a change of operator from one operator to another is received through the remote input interface 210 (and further when the operator is actually changed), the operation history of the operation target is acquired together with the operator identifier of the other operator (see FIG. 5 / STEP 202 → ... → YES in STEP 240, and FIG. 6 / STEP 288). This makes it possible to distinguish and acquire the operation history of the work machine 40 that is the operation target for each different operator. Furthermore, when a command to switch from one work machine 40 (first work machine) to another work machine 40 (second work machine) is received through the remote input interface 210 (and further when the operation target is actually switched), the operation history of the other work machine 40 is acquired together with the operator identifier (see FIG. 5 / STEP 202 → ... → YES in STEP 240, and FIG. 6 / STEP 288). This makes it possible to distinguish and acquire the operation history of each work machine 40 that is the operation target by each operator.

[0125] When one operator is replaced by another operator, if authentication of the other operator fails (see FIG. 5 / STEP 250: YES → STEP 251 → ... → STEP 260 ... NO → STEP 262), the other operator is prohibited or restricted from operating the work machine 40 as the operation target via the remote operation mechanism 211 (see FIG. 6 / STEP 282: NO → STEP 286). As a result, the prohibited or restricted operation history of the other operator and the work machine 40 as the operation target is acquired and stored and held in a database or the like.

[0126] When one operator is replaced by another operator, if authentication of the other operator fails (see FIG. 5 / YES in STEP 250 → STEP 251 → ... → NO in STEP 260 → STEP 262), switching of the work machine 40 as the operation target in response to a switching instruction from the other operator is prohibited. On the other hand, if authentication of the other operator is successful, switching of the work machine 40 as the operation target in response to a switching instruction from the other operator is permitted.

[0127] (Another embodiment of the present invention) When the operation target is switched from one work machine 40 that is an operation target to another work machine 40 that is an operation target with specifications different from the one operation target, the setting of a designated item corresponding to the difference in specifications may be accepted by the remote control device 200 via the remote input interface 210. For example, if the type of tip attachment 445 is different, it may be possible to set the assignment mode for each button so that a function specific to the operation of the tip attachment 445 (e.g., a lifting magnet) of the work machine 40 after the switch is assigned to the button. Specifically, if a function specific to the operation of the tip attachment 445, which is a difference in the specifications of the work machine 40 after the switch, is not assigned to a button, it may be determined that the operation preparation processing is not complete (NO in FIG. 4 / STEP 216), and the designated item setting processing may continue to be executed ( FIG. 4 / STEP 214). This reliably assigns a function specific to the operation corresponding to the difference in specifications to the button, enabling smooth operation of the other work machine 40 after the switch of the operation target.

[0128] In the above embodiment, the "simulation device" presents a simulation image showing the operating status of the work machine 40 by displaying operation assist, depth perception images, and terrain images specified on the menu screen, but the "simulation device" may also be configured to allow the user to experience functions of other designations that can be specified on the menu screen. For example, when the designation item "lever pattern" is selected in the designation item setting process and the lever pattern is finally set, the operation mode of the remote control mechanism 211 or the operation levers 2110, 2111, and 2112 that make up this mechanism is recognized in accordance with the lever pattern, and the operation of the work mechanism 44 and the like of the work machine 40 in the virtual space is controlled in accordance with the recognized operation mode. When the designation item "lever sensitivity" is selected in the designation item setting process and the lever sensitivity is finally set, the operation of the work mechanism 44 and the like of the work machine 40 in the virtual space is controlled in accordance with the operation mode recognized in accordance with the lever sensitivity. When the designated item "seat vibration" is selected in the designated item setting process and the intensity of the seat vibration is finally set, the operating speed and amplitude of the seat drive device 224, and ultimately the rate and amount of change in the attitude of the seat 24, are adjusted according to the operating status of the work machine 40 in the virtual space in accordance with the intensity. The operating status of the work machine 40 in the virtual space includes the tilt angle of the upper rotating body 42 of the work machine 40 about the pitch axis and / or roll axis relative to the vertical direction in the virtual space, whether or not the prime mover of the work machine 40 is operating in the virtual space, and the attitude of the work mechanism 44 of the work machine 40 in the virtual space. When the designated item "button assignment" is selected in the designated item setting process and the button assignment level is finally set, for example, by the operator operating buttons assigned with angle adjustment and / or zoom-in and zoom-out functions of the actual machine imaging device 412 of the work machine 40 in the virtual space, a virtual captured image or work environment image with the range and / or scale of the captured image adjusted is output to the remote image output device 221 (display device).

[0129] Of the functions of the remote operation device 20, excluding functions unique to the remote operation device 20, such as communication establishment processing (see FIG. 4 / STEP 210 and STEP 212), communication disconnection processing (see FIG. 4 / STEP 212), and remote operation processing (see FIG. 4 / STEP 218), authentication processing (see FIG. 4 / STEP 200), operation target and work content setting processing (see FIG. 4 / STEP 204), and designated item setting processing (see FIG. 4 / STEP 214), may be executed by the information processing terminal 10. The functions of the remote operation device 20, such as communication establishment processing (see FIG. 4 / STEP 210 and STEP 212), communication disconnection processing (see FIG. 4 / STEP 212), and remote operation processing (see FIG. 4 / STEP 218), may be executed by a server (management device) capable of intercommunicating with the remote operation device 20 and the work machine 40 via the second communications network.

[0130] Even after communication disconnection processing has been executed between the remote control device 20 and one operation target work machine 40, the operating status (such as the powered state) of the actual control device 400 and actual wireless communication device 450 of that work machine 40 may be maintained as it was before the communication disconnection processing. This eliminates the need for processing to restore the operating status of the actual control device 400, actual wireless communication device 450, etc. to the status before the communication disconnection processing (such as processing to power the actual wireless communication device 450 and restore it to a state where the communication establishment processing can be executed) when communication is re-established between the remote control device 20 and that operation target work machine 40, and therefore enables the communication establishment processing to be executed smoothly and quickly. For example, when the communication disconnection process is executed, the power supply to the actual machine control device 400 mounted on the work machine 40 is maintained (processing to cut off the power supply to the actual machine control device 400 is not performed), and the power supply to the actual machine imaging device 412 and the actual machine sensor group 416 is maintained (processing to cut off the power supply to the actual machine imaging device 412 and the actual machine sensor group 416 is not performed).The actual machine control device 400 then transmits signals corresponding to the actual machine operating status relating to the images acquired by the actual machine imaging device 412 and the status of the work machine 40 acquired by the actual machine sensor group 416 to a destination identified and indicated by the operation device identifier, and continues processing to receive the signals corresponding to the operation mode of the remote control lever for controlling the operation of the operating mechanism via the actual machine wireless communication equipment 450. Furthermore, in this case, a signal according to the actual machine operating status is transmitted to the destination identified and indicated by the operation device identifier, and the remote operation device 20, which is the destination, can determine whether the information related to the actual machine identifier included in the received signal corresponds to that of the work machine 40 currently being operated by the remote operation device 20, and thereby determine that the received signal is not that of the work machine 40 currently being operated by the remote operation device 20. Then, since the received signal is not that of the work machine 40 currently being operated by the remote operation device 20, processing is performed to discard the received signal. Note that, if the control device in the present invention includes an actual machine control device 400 for the work machine 40, the actual machine control device 400 may perform control so that the operating status (such as the power-on state) of the actual machine control device 400 and the actual machine wireless communication device 450, etc., is maintained as it was before the communication disconnection processing.

[0131] Execution of the communication disconnection process may disconnect some of the communications established between the remote control device 20 and one of the work machines 40 that is the target of operation. For example, when the communication disconnection process is executed, processing is executed to turn off the power to the actual machine imaging device 412 and the actual machine sensor group 416 that have been activated, but power is maintained to the actual machine control device 400 mounted on the work machine 40 (processing is not executed to cut off the power supply to the actual machine control device 400). The actual machine control device 400 then transmits signals corresponding to the actual machine operating status relating to the images acquired by the actual machine imaging device 412 and the status of the work machine 40 acquired by the actual machine sensor group 416 to a destination identified and indicated by the operation device identifier, and continues processing to receive the signals corresponding to the operation mode of the remote control lever for controlling the operation of the operating mechanism via the actual machine wireless communication equipment 450. In this case, the signal corresponding to the actual machine operating status regarding the state of the work machine 40 acquired by the actual machine sensor group 416 will be an empty signal containing no information, and the signal corresponding to the operation mode of the remote control lever will not be received until the work machine 40 becomes the target of operation again. The empty signal may be replaced with a signal indicating information indicating that the signal is empty. The empty signal may be transmitted at a transmission cycle shorter than the transmission cycle at which signals corresponding to the actual machine operating status acquired by the activated actual machine imaging device 412 and the actual machine sensor group 416 are transmitted to the transmission destination. In this case, the signal corresponding to the actual machine operating status is transmitted to the transmission destination identified and indicated by the operation device identifier. The remote operation device 20, which is the transmission destination, can determine whether the information related to the actual machine identifier included in the received signal corresponds to that of the work machine 40 currently operated by the remote operation device 20, thereby determining that the received signal is not the target of operation by the remote operation device 20. Since the received signal is not the target of operation by the remote operation device 20, the received signal is discarded.As a result, the communication volume between the remote control device 20 and the work machine 40 is maintained, albeit at a lower level compared to the communication established state, so that when communication is re-established between the remote control device 20 and the work machine 40 as the single operation target, the process of restoring the operating status of the actual machine control device 400, the actual machine wireless communication device 450, etc. to the status before the communication disconnection process is unnecessary or can be simplified. For example, when the actual machine control device 400 receives a communication establishment request signal from the remote control device 200, the actual machine control device 400 can send a signal including an imaged image and the actual machine operating status to the remote control device 200 simply by activating the actual machine imaging device 412 and the actual machine sensor group 416, thereby restoring the status before the communication disconnection process. Also, for example, the actual machine control device 400 may keep the power supply to the actual machine imaging device 412 and the actual machine sensor group 416 active by not performing the process of turning off the power to the activated actual machine imaging device 412 and the actual machine sensor group 416 when executing the communication disconnection process. In this case, the remote control device 200 receives a signal including the captured image and the actual machine operating status before transmitting a communication establishment request signal to the actual machine control device 400, and when executing the communication establishment process, outputs the captured image and the actual machine operating status to the remote output interface 220. As described above, the communication establishment process can be executed smoothly and quickly.

[0132] On the condition that there has been no operator change notification via the remote input interface 210 (the first flag f1 is set to "0") (see FIG. 5 / STEP 240 -> NO -> STEP 242), the presence or absence of an instruction to switch the work machine 40 as the operation target may be determined via the remote input interface 210 (see FIG. 4 / STEP 222), while maintaining the communication established with the work machine 40 as one operation target. As a result, if the work machine 40 designated as the new operation target is the same as the previous operation target (see FIG. 4 / STEP 222-1), the communication already established between the remote operation device 20 and that work machine 40 will continue to be used, eliminating the need to execute communication disconnection processing and communication establishment processing again, and operation of the work machine 40 can be smoothly resumed.

[0133] The presence or absence of an instruction to switch the work machine 40 as the operation target via the remote input interface 210 may be determined (see FIG. 4 / STEP 222) based on the requirements that an operator change has been reported via the remote input interface 210 (the first flag f1 is set to "1") and that the acquired operator identifier has changed from one operator identifier to another operator identifier (the second flag f2 is set to "1") (see FIG. 5 / STEP 240 → YES → STEP 241 and STEP 250 → YES → STEP 251). In this way, when an operator is switched from one to another, the one operator is prevented from designating a work machine 40 as the operation target to be remotely operated by the other operator, and it is possible for the other operator to designate a work machine 40 as the operation target to be remotely operated by himself or herself in accordance with his or her wishes.

[0134] When an operator change notification has been made via the remote input interface 210 (the first flag f1 is set to "1") (see FIG. 5 / STEP 240 → STEP 241), communication disconnection processing may be executed to disconnect the communication established between the remote operation device 20 and the work machine 40 as the operation target. This prevents a situation in which established communication remains between the remote operation device 20 and the work machine 40 as the operation target designated by the previous operator, causing the new operator to feel uncomfortable. For example, a captured image or work environment image received by the remote operation device 20 from the work machine 40 as the operation target designated by the previous operator is output to the remote image output device 221. Because the captured image or work environment image is unnecessary information unrelated to the remote operation by the new operator, continuing to output the captured image or work environment image to the remote image output device 221 would cause the new operator to feel uncomfortable. Here, when a changeover request is made, the communication established between the remote control device 20 and the work machine 40 to be operated is cut off, and the captured image or work environment image is no longer output to the remote image output device 221, thereby preventing the other operator from feeling uncomfortable.

[0135] In the above embodiment, when it is determined that an existing operation target exists (YES in FIG. 4 / STEP 208) on the condition that an instruction to switch the work machine 40 as the operation target has been given via the remote input interface 210 (see FIG. 4 / STEP 222-2 and FIG. 17 / Q82), the remote control device 200 starts to execute a communication disconnection process as an "operation preparation process" (FIG. 4 / STEP 212) in addition to the communication establishment process, but this is not limited to this. For example, after communication established with one work machine 40 (first work machine) is disconnected in response to an operation by the operator via the remote input interface 210, on the condition that an instruction to switch the work machine 40 as the operation target has been given via the remote input interface 210 by the operator (see FIG. 4 / STEP 222-2 and FIG. 17 / Q82), the communication establishment process may be executed to establish communication with another work machine 40 (second work machine). This makes it possible to execute processing for establishing communication with the work machine 40 (first work machine) and processing for disconnecting communication with the work machine 40 (second work machine) in accordance with the operator's wishes.

[0136] When work is being carried out using the work machine 40 as the operation target (FIG. 4 / STEP 218), provided that a change of operator notification has been made from one operator to another through the remote input interface 210 (see FIG. 5 / STEP 240: YES → STEP 241), the operation history of the work machine 40 as the operation target may be acquired together with the operator identifier that identifies the other operator (see FIG. 6 / STEP 288).

[0137] A first aspect of the present invention provides an operation device for operating an operation object, the operation device including: a communication device capable of communicating with the operation object; a control device capable of executing a communication establishment process for establishing the communication with the operation object via the communication device; and a designated item setting process for receiving, at least during the communication establishment process, through an input unit, the setting of designated items for assisting the operation of the operation object by the operation device after the communication establishment process has been executed.

[0138] With the operation device having this configuration, it is possible to set, during the execution of the communication establishment process, the content of the designation items to assist the operation of the operation target by the operation device after the completion of the communication establishment process, thereby making use of the waiting time during the execution of the communication establishment process.

[0139] An operating device according to a second aspect of the present invention is the operating device according to the first aspect, wherein the control device, when the operating object is switched from one operating object to another operating object, executes a communication disconnection process to disconnect the communication established between the communication device and the one operating object, and a communication establishment process to establish communication between the operating device and the other operating object, and it is preferable that the control device accepts setting of the content of the specified item through the input unit when at least one of the communication disconnection process and the communication establishment process is executed in conjunction with switching of the operating object through the communication device.

[0140] According to the operation device having this configuration, when the operation target is switched from one operation target to another operation target, a communication disconnection process and a communication establishment process are executed. The "communication disconnection process" is a process for disconnecting communication established between the operation device and the one operation target. The "communication establishment process" here is a process for establishing communication between the operation device and the other operation target. When at least one of the "communication disconnection process" and "the communication establishment process" is executed, it becomes possible to set the content of the specified items via the input unit. Therefore, the waiting time during the execution of the process for switching the operation target is utilized.

[0141] In the operating device according to a third aspect of the present invention, in the operating device according to the first or second aspect, it is preferable that when the operating object is switched from one operating object to another operating object having specifications different from those of the one operating object, the control device accepts, through the input unit, the setting of the content of the specified item corresponding to the differences in the specifications.

[0142] According to the operation device having the above configuration, when an operation target is switched from one operation target to another operation target having specifications different from the first operation target, it is possible to set, via the input unit, the specifications corresponding to the differences in the specifications, thereby enabling smooth operation of the other operation target after the operation target is switched.

[0143] An operating device according to a fourth aspect of the present invention is an operating device according to any one of the first to third aspects, further comprising a simulation device for operating a virtual operating object corresponding to the operating object in a virtual space, and it is preferable that the control device assists in operating the virtual operating object in the virtual space based on the setting contents of the specified items received through the input unit.

[0144] The operation device having this configuration allows a user to virtually experience, through the simulation device, the support mode and / or operation mode of the operation target that is predicted to occur depending on the setting contents of the specified items through the input unit. Based on the experience, the user can reset or change the setting contents of the specified items through the input unit.

[0145] It is preferable that the operating device according to the fifth aspect of the present invention is an operating device according to any one of the first to fourth aspects, further comprising a storage device that stores the setting contents of the specified items, and an output unit that outputs the setting contents of the specified items stored in the storage device.

[0146] The operating device having this configuration allows an operator to check the current settings of the specified items output to the output unit. Based on the output contents, the specified items can be reset or changed via the input unit.

[0147] An operating device according to a sixth aspect of the present invention is the operating device according to the fifth aspect, and preferably has a function of authenticating an operator, and the output unit outputs the setting contents of the specified items stored in the storage device in association with the authenticated operator.

[0148] According to the operation device having such a configuration, the contents of the specified items last set by each operator (or the contents of the specified items appropriate for each operator) are stored and output to the output unit. The output contents may be defined as the default contents of the specified items.

[0149] An operating device according to a seventh aspect of the present invention is the operating device according to the fifth aspect, which has a function of authenticating an operator, and the storage device preferably stores the setting contents of the specified items received through the input unit in association with the authenticated operator.

[0150] According to the operation device having this configuration, the contents of the specifications last set by each operator can be stored in the storage device.

[0151] An operating device according to an eighth aspect of the present invention is an operating device according to any one of the first to seventh aspects, wherein the control device preferably accepts settings via the input unit for the content of a specified item selected from among the plurality of specified items.

[0152] According to the operation device having the above configuration, only the content of one or more designated items selected (through the input unit) from among the plurality of designated items is set through the input unit.

[0153] What is provided by the present invention is an operation system, which is configured by an operation device according to any one of the first to eighth aspects and the operation target.

Claims

1. An operating device for operating an operation target, comprising: a communication device capable of communicating with the operation target; and a control device capable of executing a communication establishment process for establishing communication with the operation target through the communication device; and a designated item setting process for receiving, at least during the execution of the communication establishment process, through an input unit, the setting of the contents of designated items for assisting the operation of the operation target by the operating device after the execution of the communication establishment process.

2. An operating device as described in claim 1, wherein the control device, when the operation target is switched from one operation target to another operation target, executes a communication disconnection process for disconnecting communication established between the communication device and the one operation target, and a communication establishment process for establishing communication between the communication device and the other operation target, and accepts the setting of the content of the specified item through the input unit when at least one of the communication disconnection process and the communication establishment process is executed in conjunction with the switching of the operation target through the communication device.

3. An operating device as described in claim 2, wherein, when the operation target is switched from the one operation target to the other operation target having specifications different from those of the one operation target, the control device accepts, via the input unit, the setting of the content of the specified item corresponding to the difference in the specifications.

4. An operating device as described in claim 1, further comprising a simulation device for operating a virtual operation target corresponding to the operation target in a virtual space, wherein the control device assists in operating the virtual operation target in the virtual space based on the setting contents of the specified items received through the input unit.

5. An operating device according to claim 1, comprising: a storage device that stores the setting contents of the specified items; and an output section that outputs the setting contents of the specified items stored in the storage device.

6. An operating device according to claim 5, further comprising a function for authenticating an operator, and wherein the output unit outputs the setting contents of the specified items stored in the storage device in association with the authenticated operator.

7. An operating device according to claim 5, further comprising a function for authenticating an operator, wherein the storage device stores the setting contents of the specified items received through the input unit in association with the authenticated operator.

8. An operating device according to claim 1, wherein the control device accepts settings, via the input section, for the content of a specified item selected from among a plurality of specified items.

9. An operation system comprising: an operation device according to any one of claims 1 to 8; and the operation target.

Images