Remote control system and communication connection method

Abstract

The system stabilizes the state of the work machine when switching between connecting and disconnecting communication between the work machine and the control device. [Solution] The management server comprises a management server communication unit, a work machine management database that records the work machine status based on information received from the work machine via the management server communication unit, and a status management unit that performs a status transition process when it determines that the status transition conditions are met. The status management unit refers to the work machine status in the work machine management database and determines that the status transition conditions are met when the work machine is stopped, and performs a status transition process that involves connecting communication between the work machine and the operating device and a status transition process that involves disconnecting communication between the work machine and the operating device. When the work machine is in operation, it determines that the status transition conditions are not met, and does not perform either a status transition process that involves connecting communication between the work machine and the operating device or a status transition process that involves disconnecting communication between the work machine and the operating device.

Description

Technical Field

[0005]

[0001] The present invention relates to a remote operation system for managing a remotely operable work machine and a communication connection method thereof.

Background Art

[0002] With the development of wireless communication systems, it has become possible to provide high-quality wireless networks such as high capacity and low latency. As a result, remote applications that require high communication quality, such as remote medical services and remote robot control, which have been difficult to implement until now, are being considered. In the industries of construction machinery and mining machinery, remote operation systems for work machines such as hydraulic excavators and bulldozers are also being considered for the purpose of reducing the movement burden of operators.

[0003] Conventionally, remote operation systems have been designed on the premise that the work machine to be operated by the operation device is not changed (1-to-1 connection). This is because stable video transmission is difficult in a low-quality wireless network, and thus a use case of operating the work machine while directly visualizing it has been assumed. On the other hand, in recent years, since stable video transmission is possible with a high-quality network, it is possible to remotely operate from a location where the work machine cannot be directly visualized. Therefore, efficient operation such as having one operator in charge of a plurality of work machines existing at a distant location and quickly switching and operating the work machine to be operated by the operator is being considered.

[0004] In a 1-to-1 connection remote operation system, the work machine and the operation device are connected according to a predetermined setting. When switching the connection between the work machine and the operation device, restarting is required for setting change and setting reading, and it has taken time for the switching. Therefore, it has been difficult to cope with use cases such as having one operator in charge of a plurality of work machines. In view of such a situation, a remote operation system has been considered on the premise that the combination of the work machine and the operation device can be flexibly changed (N-to-M connection) in order to improve the flexibility and efficiency of the system.

[0005] For example, Patent Document 1 discloses an invention that detects instability in the operation of a work machine and the operator, and switches operators when instability is detected. In the invention described in Patent Document 1, instability such as "the work machine is not operating smoothly" or "the operator is unwell" is detected based on information from sensors and input devices such as buttons. In the invention described in Patent Document 1, if instability is detected, the work is handed over to a standby operator. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Patent No. 7287047 [Overview of the project] [Problems that the invention aims to solve]

[0007] However, in the invention described in Patent Document 1 above, the state of the work machine may become unstable when switching between connecting and disconnecting the work machine and the operating device. When the state of the work machine becomes unstable when switching between connecting and disconnecting the work machine and the operating device, it becomes difficult for the operator to quickly switch and operate the work machine in charge, resulting in a decrease in the work efficiency of the work machine.

[0008] The present invention has been made in view of the above problems, and aims to provide a remote control system and a communication connection method that stabilizes the state of a work machine when switching between connecting and disconnecting communication between the work machine and the operating device. [Means for solving the problem]

[0009] A representative example of the invention disclosed in this application is as follows: a remote control system comprising a work machine remotely controlled by communication over a network, an operating device for an operator to remotely control the work machine via communication over the network, and a management server that manages the connection and disconnection of communication between the work machine and the operating device over the network. The work machine comprises a work machine communication unit connected to the network, a sensor for obtaining information about the work machine, a camera for photographing the area around the work machine, a machine control unit that controls the operation of the work machine in response to operation commands received from the operating device via the work machine communication unit, an image transmission unit that processes and encodes images acquired from the camera and transmits them to the operating device via the work machine communication unit, and a work machine information collection unit that transmits information collected from the sensor and the machine control unit to the management server via the work machine communication unit. The operating device comprises an operating device communication unit connected to the network, an image receiving unit that decodes images received from the work machine via the operating device communication unit, a display that displays the images decoded by the image receiving unit, and an operating unit that transmits operation commands to the work machine. The management server comprises a management server communication unit connected to a network, a work machine management database that records work machine status based on information received from the work machine information collection unit of the work machine via the management server communication unit, and a status management unit that determines whether the work machine status satisfies predetermined status transition conditions, and performs status transition processing of the work machine status in the work machine management database when it is determined that the status transition conditions are met. The work machine status recorded in the work machine management database includes information regarding the operation or stopping of the work machine, and information regarding the connection or disconnection of communication between the work machine and the operating device via the network.The status management unit refers to the status of the work machine recorded in the work machine management database, determines that the status transition conditions are met when the work machine is stopped, and performs a status transition process that involves connecting or disconnecting communication via the network between the work machine and the operating device, based on information regarding the connection or disconnection of communication. When the work machine is in operation, it determines that the status transition conditions are not met, and does not perform either a status transition process that involves connecting or disconnecting communication via the network between the work machine and the operating device. [Effects of the Invention]

[0010] According to one aspect of the present invention, the state of the work machine can be made more stable when switching between connecting and disconnecting communication between the work machine and the operating device. Other issues, configurations, and effects not mentioned above will be clarified by the following description of the embodiments. [Brief explanation of the drawing]

[0011] [Figure 1] A block diagram showing the system configuration of the remote control system of the embodiment. [Figure 2] A block diagram showing the sensors of the work machine in the embodiment. [Figure 3] A schematic diagram showing the hardware configuration of the work machine of the embodiment. [Figure 4] A schematic diagram showing the hardware configuration of the operating device of the embodiment. [Figure 5] A block diagram showing the hardware configuration of the management server in the embodiment. [Figure 6] (A) is a diagram representing the work machine status table of the work machine management database of the embodiment, and (B) is a diagram representing the work machine configuration device management table of the work machine management database of the embodiment. [Figure 7] (A) is a diagram representing the control device management table of the control device management database of the embodiment, and (B) is a diagram representing the control device configuration device management table of the control device management database of the embodiment. [Figure 8] Figure showing the operator management table of the operator management database in the embodiment. [Figure 9] State transition diagram showing the state transition model of the working machine for the state management unit in the embodiment to perform state transition processing. [Figure 10] Flowchart showing the procedure of the state transition processing of the working machine in the embodiment. [Figure 11] Sequence diagram showing the procedure to start remote operation work on the working machine in the standby state in the embodiment. [Figure 12] Sequence diagram showing the procedure to end remote operation work on the working machine in the working state in the embodiment. [Figure 13] Sequence diagram showing the procedure to start up the working machine in the stopped state in the embodiment. [Figure 14] Sequence diagram showing the procedure to stop the working machine in the standby state in the embodiment.

MODE FOR CARRYING OUT THE INVENTION

[0012] Embodiments of the present disclosure will be described in detail with reference to the drawings. In the embodiment, as an example of the working machine constituting the remote operation system of the working machine, a hydraulic excavator will be described as an example. Note that the present disclosure is not limited to the embodiments shown below. The description in this specification is merely a typical example, and the present disclosure can be implemented in various modified and improved forms based on the knowledge of those skilled in the art.

[0013] Hereinafter, examples will be described with reference to the accompanying drawings. In the accompanying drawings, functionally identical elements are denoted by the same reference numerals. Note that the accompanying drawings show examples in accordance with the principles of the present disclosure, but these are for the purpose of understanding the present disclosure and are not used to limit the interpretation of the present disclosure in any way limitedly. The description in this specification is merely a typical example and does not limit the scope of the claims or application examples of the present disclosure in any sense. Components having the same reference numerals in this specification and the drawings indicate the same components.

[0014] FIG. 1 is a block diagram showing the system configuration of the remote operation system in the present embodiment. FIG. 1 is a block diagram created by focusing on the functions of the system, and the hardware configuration for realizing this will be described later in FIGS. 3 to 5. The operation outline of the system and the functions of each component will be described using FIG. 1. The remote operation system 100 includes at least one or more working machines 101, at least one or more operation devices 102, and a management server 103. In the present embodiment, the remote operation system 100 includes either at least two or more working machines 101 and at least two or more operation devices 102.

[0015] The working machine 101 is remotely operated by communication via the network NW. In the operation device 102, the operator OP remotely operates the working machine 101 by communication via the network NW. The management server 103 manages the connection and disconnection of communication via the network between the working machine 101 and the operation device 102.

[0016] The working machine 101, the operation device 102, and the management server 103 are connected by a network and can communicate with each other using IP (Internet Protocol) or the like. The remote operation is realized by the working machine 101 transmitting a video signal for checking the on-site environment to the operation device 102, and the operation device 102 transmitting an operation signal for control to the working machine 101. The management server 103 enables remote operation between an arbitrary working machine 101 and the operation device 102 in response to an operator's request.

[0017] Specifically, as will be described later, it manages the state regarding the usage status of the working machine 101 such as the working state and the resting state, and performs state transition processing (described later in S200 of FIGS. 9 and 10) of the working machine 101 that may include connecting or disconnecting the communication between the working machine 101 and the operation device 102 in response to a request from the operator. In the present embodiment, by managing the state of the working machine 101, it is possible to prevent the system from falling into an unstable state such as simultaneous operation by multiple operators, and a system that can flexibly connect and switch between the working machine 101 and the operation device 102 is realized.

[0018] Furthermore, in the state transition process (S200), state transition condition confirmation processes (described later in S205 of Figure 10) such as communication quality confirmation and parking attitude confirmation are performed to improve the stability of the work site before connecting and disconnecting the work machine 101 and the operating device 102. In addition, in the work machine connection process (described later in S101 of Figure 9) and work machine disconnection process (described later in S102 of Figure 9), attention is paid to the order of connection and disconnection of communication between the video signal and the operating signal to improve stability during communication connection and disconnection. Specific procedures will be described later in Figures 9 to 14.

[0019] As shown in Figure 1, the work machine 101 includes one or more sensors 104, one or more cameras 105, a machine control unit 106, a work machine information collection unit 107, an image transmission unit 108, and a work machine communication unit 109A. The sensors 104 are for obtaining information about the work machine 101. As shown in Figure 2, the types of sensors 104 include a positioning sensor 601, an acceleration sensor 602, a gyroscope sensor 603, a contact sensor 604, a three-dimensional sensor 605, a distance sensor 606, and a temperature sensor 607. For example, a GPS (Global Positioning System) can be used as the positioning sensor 601.

[0020] Camera 105 is for photographing the area around the work machine 101. Camera 105 is a device for acquiring images of the work environment.

[0021] The machine control unit 106 controls the operation of the work machine 101 in response to operation commands received from the operating device 102 via the work machine communication unit 109A. The machine control unit 106 interprets the operation signals received from the operating device 102 and controls the prime mover, such as the internal combustion engine, and actuators of the work machine 101. The machine control unit 106 also acquires machine control information related to the control of the work machine 101 (such as the status of the prime mover, the status of the actuators, the speed of the work machine 101, information on operation signals, the status of the manned operation interface, and the status of the storage stairs).

[0022] The machine information collection unit 107 transmits information collected from the machine communication unit 109A, the sensor 104, and the machine control unit 106 to the management server via the machine communication unit 109A. The machine information collection unit 107 has the function of collecting and storing information measured and acquired by the sensor 104, the machine control unit 106, and the machine communication unit 109A (information from the sensor 104, machine control information acquired by the machine control unit 106, and communication quality information acquired by the machine communication unit 109A). The information stored by the machine information collection unit 107 is provided to the state management unit 116 of the management server 103 for the state transition condition confirmation process described later.

[0023] The image transmission unit 108 processes and encodes the image acquired from the camera 105 and transmits it to the operating device 102 via the machine communication unit 109A. The image transmission unit 108 has the function of encoding the image acquired by the camera 105, reducing the amount of information, and transmitting it to the operating device 102.

[0024] The machine communication unit 109A connects to the network NW. The machine communication unit 109A functions as a relay between the internal network and the external network of the machine 101. The internal network refers to wired communication using Ethernet®, etc. The external network refers to wireless communication using Wi-Fi®, LTE®, 4G, and 5G, etc. Furthermore, the machine communication unit 109A acquires communication quality information through active measurement between the machine communication unit 109A and any other communication device using measurement traffic, and through passive measurement within the machine communication unit 109A.

[0025] The communication quality information acquired by the machine communication unit 109A includes communication delay, packet loss rate, throughput, input / output traffic volume of the machine communication unit 109A, number of lost packets, RSSI (Received Signal Strength Indicator), RSRP (Reference Signal Received Power), RSRQ (Reference Signal Received Quality), SINR (Signal to Interface plus Noise Ratio), and SNR (Signal to Noise Ratio).

[0026] The operating device 102 comprises an operating device communication unit 109B, an operating unit 110, an image receiving unit 111, and a display 112. The operating device communication unit 109B is connected to a network NW. The functions of the operating device communication unit 109B are the same as those of the machine communication unit 109A described above, so their explanation is omitted. However, unlike the machine communication unit 109A, the operating device communication unit 109B usually communicates via wired connections to both the internal and external networks.

[0027] The operation unit 110 transmits operation commands to the work machine 101. The operation unit 110 has an interface that physically receives input from the operator OP and has the function of transmitting operation signals to the work machine 101 according to the input from the operator OP. The image receiving unit 111 decodes the image received from the work machine 101 via the operation device communication unit 109B. The image receiving unit 111 decodes the encoded image received from the work machine 101 and transmits it to the display 112. The display 112 displays the image decoded by the image receiving unit 111. The display 112 is a device that displays the image received from the image receiving unit 111.

[0028] The management server 103 comprises a management server communication unit 109C, a work machine management database 113, an operating device management database 114, an operator management database 115, and a status management unit 116. The management server communication unit 109C is connected to the network NW. The management server communication unit 109C is equivalent to the work machine communication unit 109A and the operating device communication unit 109B described above, so its description is omitted. Also, like the operating device communication unit 109B, the management server communication unit 109C usually communicates via wired connections to both the internal and external networks.

[0029] The work machine management database 113 records the work machine status related to the state of the work machine 101 based on information received from the work machine information collection unit 107 of the work machine 101 via the management server communication unit 109C. The work machine management database 113 is a function that manages the work machine status related to the usage status of the work machine 101, such as stopped state and operating state, as well as network information of the devices that constitute the work machine 101. The work machine status recorded in the work machine management database 113 includes information on when the work machine 101 is operating and stopped, and information on when communication between the work machine 101 and the operating device 102 via the network NW is connected and disconnected.

[0030] The working machine status is used for mutual exclusion control when connecting the working machine 101 and the operating device 102. For example, by managing the status of whether the working machine 101 is already connected to any of the operating devices 102, it is possible to prevent other operating devices 102 from connecting to that working machine 101 simultaneously. In addition, it is also possible to manage statuses such as "operating with a human" and "malfunction adjustment in progress," and to control remote operation according to the status. The network information of the devices that make up the working machine 101 is used in the state transition processing described later to actually access the devices that make up the working machine 101 and perform processes such as "starting or stopping," "acquiring information," and "changing settings." Details of the working machine management DB 113 will be described later in Figure 7.

[0031] The control device management database 114 is a function that manages the status of the control device 102, such as whether it is in use or not, and network information of the devices that make up the control device 102. The control device status is used to check the usage status of the control device 102. The network information of the devices that make up the control device 102 is used to access the devices and change settings when connecting and disconnecting communication between the work machine 101 and the control device 102. Details of the control device management DB 114 will be described later in Figure 8.

[0032] The operator management database 115 records state transition conditions added for each operator OP, as will be described later. The operator management database 115 has the function of managing the operator OP's authorization information and authentication information for the remote control system 100. The authorization information is information for controlling the operations that each operator OP can execute. The authentication information is information for authenticating whether the request is legitimate when the operator OP executes a state transition process. Details of the operator management database 115 will be described later in Figure 9.

[0033] The state management unit 116 determines whether the state of the work machine satisfies predetermined state transition conditions, and when it determines that the state transition conditions are met, it performs state transition processing of the work machine state in the work machine management database 113. The state management unit 116 has the function of performing state transition processing of the work machine 101, assigning an operator OP to the operating device 102, and providing an interface for performing state transition processing and operator assignment. State transition processing is executed when a request regarding state transition is received from the operator. After receiving the request, the state management unit 116 accesses the work machine information collection unit 107 of the work machine 101 and collects information regarding the work machine 101. Then, it performs state transition condition confirmation processing for the work machine 101 using the collected information. If it is determined that the state transition conditions are met, it performs processing (state transition processing) which may involve connecting or disconnecting the video signals and operating signals of the work machine 101 and the operating device 102.

[0034] State transition processing is performed using network information and other data of the devices constituting the work machine 101 and the operating device 102, which are managed in the work machine management database 113 and the operating device management database 114. Operator assignment is performed in response to a request from operator OP. After authentication by the state management unit 116, operator OP sends a request to use the operating device 102. Upon receiving the request, the state management unit 116 changes the status of the operating device 102 requested by operator OP to "in use".

[0035] State transition processing and operator assignment are performed through an interface provided by the state management unit 116. The interface is implemented, for example, by a web system. Operators can access the operating device 102 using HTTP (Hyper Text Transfer Protocol) from a web browser on a mobile terminal 500 such as a smartphone or tablet, and can authenticate and send arbitrary requests via a GUI (Graphical User Interface). Note that the device to which requests can be sent is not limited to the mobile terminal 500, but may also be, for example, the operating unit 110 or other systems.

[0036] In addition to the configuration shown in Figure 1, the work machine 101, operating device 102, and management server 103 may be connected by multiple different networks NW. The work machine communication unit 109A, the operating device communication unit 109B, and the management server communication unit 109C are equipped with the function of blocking specific traffic using a firewall and connecting to a VPN (Virtual Private Network), which is an encrypted network NW.

[0037] The work machine 101 may further include a microphone, and the image transmission unit 108 may encode the audio signal acquired from the microphone in addition to the video signal and transmit it to the operating device 102, and the operating device 102 may further include a speaker, and the image receiving unit 111 may decode the audio signal received from the work machine 101 and play it back through the speaker. The state management unit 116 may store the information acquired from the work machine information collection unit 107 and use that information to perform state transition condition confirmation processing. Multiple management servers 103 may be installed and the work machine 101 and the operating device 102 may be managed separately.

[0038] Figure 3 is a schematic diagram showing the hardware configuration of the work machine 101 in this embodiment. The work machine 101, which is a hydraulic excavator, is equipped with crawlers 201, an upper rotating body 202 supported and transported by the crawlers 201, a boom 203 mounted on the upper rotating body 202, an arm 204, a bucket 205 for excavating the ground G, and a manned operation room 206. The work machine 101 is also equipped with a prime mover 220, such as an internal combustion engine, which is the power source for the operation of the work machine 101. The work machine 101 is also equipped with stairs 230 that can be retracted and extended. The stairs 230 are used for the operator OP to board the work machine 101 when it is in a manned operation state, which will be described later.

[0039] The upper rotating body 202 is equipped with a sensor 104, a camera 105, a communication device 207, and an antenna 208. The manned control room 206 is equipped with a vehicle controller 209, an information gathering device 210, and an image transmission device 211. The work machine 101 also includes, as hardware not shown, a hydraulic actuator and hydraulic pump for driving the work machine 101, hydraulic electric auxiliary equipment to assist in the control of the hydraulic system, a generator driven by the prime mover 220 to generate electricity, a main power supply for storing electricity, a backup power supply for storing electricity, and cables for connecting various hardware.

[0040] The crawler 201 is a device that allows the work machine 101 to move forward, backward, left, and right. The work machine 101 can move by driving the crawler 201 with a hydraulic motor, which is a hydraulic actuator. The boom 203, arm 204, and bucket 205 are devices for excavating the ground G. The boom 203, arm 204, and bucket 205 are driven by a hydraulic cylinder, which is a hydraulic actuator, and perform lifting, extending, and excavating operations.

[0041] The upper rotating body 202 is configured to rotate relative to the crawler 201. The upper rotating body 202 can face any direction by driving a slewing hydraulic motor, which is a hydraulic actuator, enabling efficient excavation work. Each hydraulic actuator is powered by the prime mover 220 and receives power via a hydraulic pump.

[0042] The manned control room 206 is a space for manned operation. In addition to an operator's seat and a manned operation interface, the manned control room 206 is equipped with a vehicle controller 209, an information collection device 210, and an image transmission device 211. The vehicle controller 209, the information collection device 210, and the image transmission device 211 are computers that implement the machine control unit 106, the work machine information collection unit 107, and the image transmission unit 108 shown in Figure 1, respectively. Detailed hardware will be described later in Figure 5.

[0043] The vehicle controller 209 controls the operation of the work machine 101 by operating the hydraulic pump, the control valve for controlling the pressurized oil supplied from the hydraulic pump, the prime mover 220, etc., in response to the operation signals input by the operator. The vehicle controller 209 also acquires machine control information related to the work machine 101 (details are shown above in Figure 1) and provides it to the information collection device 210. Specific methods for acquiring machine control information include "estimating the angle, speed, position, and direction of the hydraulic actuator from the state of the hydraulic pump and control valve," "understanding the operating status of the prime mover 220 from the transmission history of operation signals to the prime mover 220," and "recording the input history of the manned operation interface."

[0044] The vehicle controller 209 is a device that collects and stores information measured and acquired by the ECUs (Electronic Control Units) of the sensor 104, communication device 207, and work machine 101. The vehicle controller 209 and the sensor 104 are connected by wire using shielded cables and coaxial cables, etc., and information is collected using communication protocols such as SPI (Serial Peripheral Interface), I2C (registered trademark), RS-232C (Recommended Standard-232C), Modbus (registered trademark), and UART (Universal Asynchronous Receiver Transmitter).

[0045] The vehicle controller 209 and the communication device 207 are connected by wire using UTP (Unshielded Twisted Pair) cables and fiber optic cables, and information is collected using communication protocols such as SNMP (Simple Network Management Protocol). The vehicle controller 209 and the ECUs of each part of the work machine 101 are connected by wire using shielded cables and UTP cables, and information is collected using communication protocols such as CAN (Controller Area Network).

[0046] The image transmission device 211 is a device that encodes and transmits images acquired from the camera 105. It is connected to the camera 105 by a wired connection using a coaxial cable or UTP cable, and acquires images using communication protocols such as RS-232C, SDI (Serial Digital Interface), and GigE Vision (registered trademark). Encoding is performed using H.264, H.265, Motion JPEG, etc.

[0047] The communication device 207 installed on the upper rotating body 202 is a computer that implements the work machine communication unit 109A shown in Figure 1. The communication device 207 relays communication between the internal network of the work machine 101 and the external network. In addition, the communication device 207 of the work machine 101 also converts between wireless and wired signals via the antenna 208, which is a device that transmits and receives radio waves. It is connected to the vehicle controller 209 and the information collection device 210 by wire using UTP cables and optical fiber cables, etc., and communicates using communication protocols such as IP.

[0048] The above-mentioned devices receive power from the main power supply and the backup power supply. When the prime mover 220 is started, all devices receive power, but when the prime mover 220 is stopped, in order to conserve power, only some devices necessary for remotely starting the prime mover 220 receive power. Devices that receive power when the prime mover 220 is stopped include, for example, the sensor 104, the communication device 207, the antenna 208, the vehicle controller 209, and the information gathering device 210. When the prime mover 220 is stopped, the devices may be operated in power-saving mode and supplied with less power than when the prime mover 220 is started.

[0049] In addition to the configuration shown in Figure 3, multiple sensors 104, cameras 105, antennas 208, and image transmission devices 211 may be installed. The image transmission device 211 may be integrated with the camera 105. The vehicle controller 209 and the information collection device 210 may be integrated. The communication device 207 may be integrated with the antenna 208.

[0050] Figure 4 is a schematic diagram showing the hardware configuration of the operating device 102 in this embodiment. The operating device 102 consists of a display 112, a communication device 307, an operator's seat 301, an operating stick 302, a control room controller 303, and an image receiving device 304. The operator's seat 301 is a chair in which the operator OP sits. The operating stick 302 and the control room controller 303 are provided on its side. The operating stick 302 is an input interface device for operating the work machine 101. The control room controller 303 is a computer that implements the operation unit 110. The communication device 307 is a computer that implements the operating device communication unit 109B shown in Figure 1. The communication device 307 relays the internal network of the operating device 102 and the external network. Detailed hardware will be described later in Figure 5.

[0051] The control stick 302 and the control room controller 303 are connected by a wire. The control room controller 303 generates an operation signal in response to the input from the control stick 302 and transmits it to the work machine 101 via the communication device 307. This makes it possible to operate the work machine 101. The image receiving device 304 is a computer that realizes the image receiving unit 111 shown in Figure 1. Detailed hardware will be described later in Figure 5. The image receiving device 304 decodes the image received from the work machine 101 via the communication device 307 and transmits it to the display 112 for display. The image receiving device 304 and the display 112 are connected by a coaxial cable and a shielded cable, etc., and communicate using communication protocols such as SDI and HDMI (High-Definition Multimedia Interface®). This makes it possible for the operator OP to visually grasp the site environment and remotely operate the work machine 101.

[0052] In addition to the configuration shown in Figure 4, multiple image receiving devices 304 may be installed. Only one display 112 may be installed. In addition to images, sounds related to the on-site environment may be output from the display 112 or a separately installed speaker. The operating device 102 may include various levers, switches, pedals, touch panels, game controllers, etc., in addition to the operating rod 302, and the control room controller 303 may generate and transmit operating signals in response to their inputs. The control room controller 303 may access the management server 103 using input / output interface devices such as touch panels, displays, keyboards, and mice.

[0053] Figure 5 is a block diagram showing the hardware configuration of the management server 103 in this embodiment. The management server 103 is a computer for implementing the work machine management database 113, the operation device management database 114, the operator management database 115, the status management unit 116, and the management server communication unit 109C. In Figure 5, the management server communication unit 109C is implemented by a communication device 119. Furthermore, the term "server" here refers to a processing device that can send and receive information with a communication device via a communication line, etc., and does not refer to hardware forms such as so-called personal computers or embedded information devices.

[0054] The computer consists of a CPU (Central Processor Unit) 401, memory 402, auxiliary storage device 403, communication interface 404, and input / output interface 405. This configuration is the same for the communication device 207, vehicle body controller 209, information acquisition device 210, image transmission device 211, control room controller 303, and image receiving device 304 described above.

[0055] The CPU 401 is an arithmetic unit that executes programs stored in the memory 402. The CPU 401 executes various programs, thereby realizing the functions of the management server 103, the work machine 101, and the operating device 102. Note that some of the processing performed by the CPU 401 through program execution may be performed by other arithmetic units (such as a GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), or FPGA (Field Programmable Gate Array)).

[0056] Memory 402 includes non-volatile memory elements such as ROM (Read Only Memory) and volatile memory elements such as RAM (Random Access Memory). ROM stores immutable programs (such as BIOS: Basic Input Output System). RAM is a high-speed, volatile memory element like DRAM (Dynamic Random Access Memory) and temporarily stores programs executed by CPU 401 and data used during program execution.

[0057] The auxiliary storage device 403 is a high-capacity, non-volatile storage device such as a magnetic storage device (HDD: Hard Disk Drive) or flash memory (SSD: Solid State Drive). The auxiliary storage device 403 also stores data used by the CPU 401 when executing programs, and the programs executed by the CPU 401. In other words, programs are read from the auxiliary storage device 403, loaded into memory 402, and executed by the CPU 401, thereby realizing the functions of the management server 103, the work machine 101, and the operating device 102.

[0058] The communication interface 404 is a network interface device that controls communication with other devices according to a predetermined protocol. The input / output interface 405 is an interface device for connecting devices that perform input and output to the computer. Sensors 104, hydraulic and electric auxiliary equipment, a keyboard, a mouse, and a display 112 are connected to the input / output interface 405, and by reading and writing specific addresses in memory 402 using the CPU 401, it is possible to read input signals and output video, etc.

[0059] Figures 6(A) and 6(B) show the work machine management database 113 in this embodiment. The work machine management table 1101 shown in Figure 6(A) is a table that manages information about the work machine 101. As shown in Figure 6(A), the work machine management table 1101 consists of work machine ID, static work machine information (work machine name, work machine type, work machine location), and dynamic work machine information (work machine status).

[0060] The work machine ID is the identifier for work machine 101. The work machine ID is used as an index when the operator OP sends a state transition request and when the management server 103 refers to information about work machine 101 during state transition processing.

[0061] Static information about the work machine is information that the state management unit 116 of the management server 103 reads only. Static information about the work machine includes the work machine name, work machine type, and work machine location. The work machine name is the name of the work machine 101. The work machine type is information about the type of work machine 101. The work machine location is information about the location of the work machine 101. Static information about the work machine is basically used to facilitate the identification of the work machine 101, but it may also be used in determining whether a state transition is possible in state transition processing, such as restricting the operation devices 102 that can be connected according to the work machine location and work machine type.

[0062] The work machine dynamic information is information that the status management unit 116 of the management server 103 reads and writes. The work machine dynamic information includes the work machine status. The work machine status includes information regarding when the work machine 101 is operating and stopped, and information regarding when communication between the work machine 101 and the operating device 102 via the network NW is connected and disconnected.

[0063] The working machine status recorded in the working machine management database 113 includes an "in operation" status, which indicates that the prime mover 220 of working machine 101 is running, working machine 101 is in operation, and communication between working machine 101 and the control device 102 via the network NW is established. In the example in Figure 6(A), working machines 101 with working machine IDs C2 and C3 are in the "in operation" status. Note that "working machine 101 is in operation" means that the state of working machine 101 has changed as seen from outside the working machine 101. In other words, "working machine 101 is in operation" means that any of the crawler 201, upper rotating body 202, boom 203, arm 204, bucket 205, and stairs 230 of working machine 101 are operating.

[0064] The working machine status recorded in the working machine management database 113 includes a dormant state, which indicates that the prime mover 220 of working machine 101 is operating, working machine 101 is stopped, and communication between working machine 101 and the control device 102 via the network NW is disconnected. In the example in Figure 6(A), working machine 101 with working machine ID C1 is in the dormant state. Note that "working machine 101 is operating" means that the state of working machine 101 has not changed as viewed from outside the working machine 101. In other words, "working machine 101 is stopped" means that the crawler 201, upper rotating body 202, boom 203, arm 204, bucket 205, and stairs 230 of working machine 101 are not operating.

[0065] The machine status recorded in the machine management database 113 includes a stopped state, which indicates that the prime mover 220 of machine 101 is stopped, machine 101 is stopped, and communication between machine 101 and the operating device 102 via the network NW is disconnected. In the example in Figure 6(A), machine 101 with machine ID C4 is in the stopped state.

[0066] The work machine status recorded in the work machine management database 113 includes a manned operation status, which indicates that an operator OP is on board work machine 101 and operating work machine 101, and that communication between work machine 101 and the control device 102 via the network NW is currently disconnected. In the example in Figure 6(A), work machines 101 with work machine IDs C5 and C6 are in the manned operation status.

[0067] The working machine status recorded in the working machine management database 113 includes information on whether the working machine 101 is stopped with its bucket 205 in contact with the ground G on a horizontal surface G. As will be described later, the state in which the working machine 101 is stopped with its bucket 205 in contact with the ground G on a horizontal surface G is called the "parked position". In the example in Figure 6(A), working machines 101 with working machine IDs C1, C3, C4, and C6 are in the parked position.

[0068] The work machine status recorded in the work machine management database 113 includes information about the received radio wave strength received by the work machine 101 via communication from the operating device 102 over the network NW. In the example in Figure 6(A), the received radio wave strength of work machine 101 with work machine ID C2 is -50 dBm, and the received radio wave strength of work machine 101 with work machine ID C3 is -48 dBm.

[0069] The work machine status recorded in the work machine management database 113 includes information regarding the communication delay of communication between the work machine 101 and the operating device 102 via the network NW. In the example in Figure 6(A), the communication delay of work machine 101 with work machine ID C2 is 43ms, and the communication delay of work machine 101 with work machine ID C3 is 39ms.

[0070] The work machine status recorded in the work machine management database 113 includes information on whether or not work machine 101 is located in a pause area, as described later. In the example in Figure 6(A), work machine 101 with work machine ID C1 is located in a pause area. The work machine status recorded in the work machine management database 113 includes information on whether or not work machine 101 is located in a stop area, as described later. In the example in Figure 6(A), work machines 101 with work machine IDs C4 and C6 are located in a pause area.

[0071] The work machine status recorded in the work machine management database 113 includes information on whether or not the stairs 230 are stored. In the example in Figure 6(A), the stairs 230 are stored for work machines 101 with work machine IDs C1 to C6. The work machine status recorded in the work machine management database 113 also includes information on the combination of work machine 101 and operating device 102 that are connected via network NW. In the example in Figure 6(A), work machine 101 with work machine ID C2 is connected to operating device 102 with operating device ID R1, which will be described later. Also in the example in Figure 6(A), work machine 101 with work machine ID C3 is connected to operating device 102 with operating device ID R3.

[0072] The work machine status recorded in the work machine management database 113 includes state transition conditions that are added for each work machine 101. In the example in Figure 6(A), work machine 101 with work machine ID C1 is subject to additional state transition conditions that the N value of the rest area must exceed 20 and the N value of the stop area must exceed 30. In addition to the N values ​​of the rest area and stop area, the state transition conditions added for each work machine 101 may also include the speed, load capacity, and fuel amount for each work machine 101.

[0073] The work machine configuration device management table 1102 shown in Figure 6(B) is a table that manages information about the devices that make up the work machine 101. As shown in Figure 6(B), the work machine configuration device management table 1102 consists of a device ID, a device name, and device control information (device type, device software version, related work machine ID, device IP address). The device ID is an identifier for the device. The device name is a name for the device to facilitate identification of the device. The device control information is information used to execute state transition processing.

[0074] The device control information includes the device type, device software version, associated work machine ID, and device IP address. The device type is information about the type of device. The device software version is information such as the version number indicating the version of the software that constitutes the device. The associated work machine ID is the work machine ID of work machine 101 that is equipped with the device. The device IP address is access information for the device.

[0075] In the state transition process, the state management unit 116 of the management server 103 identifies the devices that constitute the work machine 101 using the related work machine ID. It identifies the devices that implement each function, such as the work machine information collection unit 107, using the device type. It accesses the various devices using the device IP address and performs information collection from the work machine information collection unit 107 and state change processing in a manner appropriate to the device software version.

[0076] Figures 7(A) and 7(B) show the table configuration of the control device management database 114 in this embodiment. The control device management table 1201 shown in Figure 7(A) is a table that manages information about the control device 102. As shown in Figure 7(A), the control device management table 1201 consists of a control device ID, static control device information (control device name, control device type, control device location), and dynamic control device information (control device status, operator ID in use). The control device ID is the identifier of the control device 102. It is used as an index when an operator sends a control device assignment request and when the management server 103 refers to information about the control device 102 during state transition processing.

[0077] Static information about the operating device is information that the state management unit 116 of the management server 103 reads only. Static information about the operating device includes the operating device name, operating device type, and operating device location. The operating device name is the name of the operating device 102. The operating device type is information about the type of operating device 102. The operating device location is information about the location of the operating device 102. Static information about the operating device is basically used to facilitate the identification of the operating device 102, but it may also be used in the determination of whether a state transition is possible in the state transition process, such as restricting the work machine 101 that can be connected according to the operating device location and operating device type.

[0078] The operating device dynamic information is information that the status management unit 116 of the management server 103 reads and writes. The operating device dynamic information includes the operating device status and the operator ID in use. The operating device status is information regarding the usage status of the operating device 102, such as whether it is in use or not. The operator ID in use is the identifier of the operator OP who is using the operating device 102. The operating device status is used to confirm and record the current operating device status in the operating device status transition process. The operator ID in use is used to confirm and record the operator OP who is using the operating device 102. It is also used to refer to information about the operator OP in state transition feasibility determination, work machine connection process, work machine disconnection process, etc.

[0079] The control device configuration management table 1202 shown in Figure 7(B) is a table that manages information about the devices that make up the control device 102. As shown in Figure 7(B), the control device configuration management table 1202 consists of a device ID, a device name, and device control information (device type, device software version, associated control device ID, and device IP address). The device ID is an identifier for the device. The device name is a name for the device to facilitate identification of the device.

[0080] Device control information is information used to execute state transition processing. Device control information includes device type, device software version, associated operating device ID, and device IP address. Device type is information about the type of device. Device software version is information such as the version number indicating the version of the software that constitutes the device. Associated operating device ID is the operating device ID of the operating device 102 that houses the device. Device IP address is access information for the device. In state transition processing, the state management unit 116 of the management server 103 identifies the devices that constitute the work machine 101 using the associated operating device ID. It identifies the devices that realize each function, such as the operating unit 110, using the device type. It accesses various devices using the device IP address and performs state change processing in a manner corresponding to the device software version.

[0081] Figure 8 is a diagram showing the table structure of the operator management database 115 in this embodiment. As shown in Figure 8, the operator management table 1301 is a table that manages information about operators OP. The operator management table 1301 consists of operator ID, operator name, operator operation proficiency, operator authority, and authentication password.

[0082] The Operator ID is an identifier for the Operator OP. It is used as an index when the Operator OP sends a request for assignment of an operating device and when the management server 103 refers to information about the Operator OP during state transition processing. The Operator Name is the name of the Operator OP to facilitate identification of the Operator OP. The Operator Proficiency level is the degree of the Operator OP's proficiency in remote operation. The Operator Proficiency level is used to restrict the operating devices 102 that can be connected in the state transition feasibility determination during state transition processing, to change the conditions for determining whether the state transition conditions are met in the state transition condition confirmation processing, and to change the setting items of the image transmission unit 108 during the work machine connection processing.

[0083] Operator privileges refer to information about the scope of operations that an operator (OP) can perform in a remote control system. They are used to restrict the patterns of allowed state transitions in the state transition approval / failure determination process. The authentication password is information used to authenticate the operator.

[0084] In this embodiment, the management server 103 includes an operator management database 115 that records additional state transition conditions for each operator OP. For example, in the example shown in Figure 8, operators OP with operator IDs P2 and P3 are subject to additional state transition conditions: they cannot transition to the stopped state, the communication delay must be less than 70ms, and the received signal strength must be greater than -60dBm.

[0085] The details of the state transition process (S200) of the work machine 101 in this embodiment will be explained below with reference to Figures 9 and 10. The state transition process (S200) of the work machine 101 is a process that transitions the state of a work machine 101 from one state to another.

[0086] First, the working machine status will be explained. The working machine status refers to the state of use of the working machine 101, which is managed to prevent inappropriate remote operation such as simultaneous operation by multiple operators. Working machine statuses include the working state M101, which indicates that the working machine 101 is connected to the control device 102, and the idle state M102, which indicates that the prime mover 220 of the working machine 101 is running and is not connected to any control device 102. For example, by controlling the system so that the control device 102 is not connected to the working machine 101 in the working state M101, it is possible to prevent simultaneous operation by multiple operators. Details of the working machine status will be explained later in Figure 9.

[0087] Next, with reference to Figure 10, an overview of the procedure for the state transition processing (S200) of the work machine 101 will be described. The state transition processing (S200) of the work machine 101 is executed when a state transition request is sent. Upon receiving the state transition request, the state management unit 116 performs state transition condition confirmation processing (S205) according to the state transition pattern, and state transition processing (S207) for the work machine 101 and the operating device 102, according to a pre-programmed state transition model.

[0088] The state transition condition confirmation process (S205) is a process to confirm the stability of the work machine 101. For example, it checks whether the work machine 101 is in an operable position, and whether the communication delay between the work machine 101 and the operating device 102 is sufficiently small.

[0089] The state transition process (S207) is a modification process for the work machine 101 and the operating device 102 necessary for the transition of the work machine state. For example, this includes changing the image transmission destination of the image transmission unit 108 in the connection process between the work machine 101 and the operating device 102 (work machine connection process: S101). The details of the state transition model of the work machine 101 are shown in Figure 9, and the detailed procedure of the state transition process (S200) is explained in Figure 10.

[0090] Figure 9 is a diagram showing the machine state transition model for the state management unit 116 of Embodiment 1 to perform state transition processing (S200). The circled area represents the machine state, and the arrows indicate the direction in which state transitions are possible. Near each arrow, the state transition conditions (in small font) used in the state transition condition confirmation process (S205) and the state transition processing (S207) (in bold font) for the machine 101 and the operating device 102 are indicated. In this embodiment, the machine state can take on four states: working state M101, idle state M102, stopped state M103, and manned operation state M104. Each state will be described below.

[0091] As described above, the working state M101 is a state in which the prime mover 220 of the working machine 101 is operating, the working machine 101 is in operation, and communication between the working machine 101 and the operating device 102 via the network NW is established. The working state M101 indicates that the working machine 101 is performing work via remote control. In the working state M101, the working machine 101 and the operating device 102 are connected, and video signals and operation signals are transmitted, respectively. It is possible to directly transition from the working state M101 to the idle state M102. On the other hand, it is not possible to directly transition from the working state M101 to the stopped state M103 or the manned operation state M104.

[0092] The conditions for transitioning from the working state M101 to the idle state M102 are: "the prime mover 220 of the work machine 101 is operating," "the work machine 101 is in a parked position," and "the work machine 101 is located within the idle area." The fact that the prime mover 220 is running can be determined, for example, by "the state of the prime mover 220 acquired by the machine control unit 106 being a value that represents the operating state."

[0093] The parked position is the posture for stably parking the work machine 101, and refers to the state where the bucket 205 is in contact with the ground on a level surface and stopped. For example, the parked position can be determined by methods such as "the Z-axis of the gyro sensor horizontally mounted on the upper rotating body 202 is 0 degrees", "the height of the bucket 205 calculated from the angles of the hydraulic actuators of the boom 203, arm 204, and bucket 205 acquired by the machine control unit 106 is 0 m", "the speed of the work machine 101 acquired by the machine control unit 106 is 0 km / h", and "the state of the Pilot-Shutoff Lever acquired by the machine control unit 106 is OFF". Note that "level" is not strictly limited to the inclination angle of the ground G being 0 degrees; for example, the ground G may have an inclination angle of about ±5 degrees.

[0094] The idle area refers to the range in which the work machine 101 can enter the idle state M102. Specifically, the idle area is a location near the mining site in a mine where the work machine 101 can be stably stopped while the prime mover 220 is still running. For example, it is possible to determine whether the work machine 101 is within the idle area by checking whether the latitude and longitude obtained from the positioning sensor 601 attached to the upper rotating body 202 fall within a specific range. The state transition process (S207) when transitioning to the idle state M102 is the work machine disconnection process (S102). The work machine disconnection process (S102) stops the transmission of video signals from the work machine 101 to the control device 102, and stops the transmission of operation signals from the control device 102 to the work machine 101. Details of the work machine disconnection process (S102) will be described later in Figure 12.

[0095] As described above, the idle state M102 is a state in which the prime mover 220 of the work machine 101 is running, the work machine 101 is stopped, and communication between the work machine 101 and the operating device 102 via the network NW is disconnected. The idle state M102 indicates that the work machine 101 is not in use and that work can be started by remote control. In the idle state M102, the work machine 101 is not connected to any operating device 102, and the prime mover 220 is started. From the idle state M102, it is possible to directly transition to the working state M101, the stopped state M103, and the manned operation state M104.

[0096] The conditions for transitioning from the idle state M102 to the working state M101 are: "the communication delay between the work machine 101 and the operating device 102 is less than a specific communication delay threshold," "the received radio wave strength of the work machine 101 exceeds a specific radio wave strength threshold," and "the work machine 101 is located within the workable area."

[0097] Communication delay refers to the delay between the work machine 101 and the operating device 102 attempting to connect. For example, it can be determined whether the communication delay is below a specific communication delay threshold by checking if the communication delay measured by the work machine communication unit 109A is less than 100 ms. Received radio wave strength refers to the strength of the radio signal received by the antenna 208 of the work machine 101. For example, it can be determined whether the received radio wave strength exceeds a specific radio wave strength threshold by checking if the received radio wave strength measured by the work machine communication unit 109A exceeds -80 dBm.

[0098] The workable area refers to the range in which the work machine 101 can be in the working state M101. For example, it is possible to determine whether the work machine 101 is within the workable area by methods such as "the latitude and longitude obtained from the positioning sensor 601 attached to the upper rotating body 202 fall within a specific range."

[0099] The state transition process (S207) when transitioning from the idle state M102 to the working state M101 is the work machine connection process (S101). In the work machine connection process (S101), the transmission of a video signal from the work machine 101 to the control device 102 is started, and the transmission of an operation signal from the control device 102 to the work machine 101 is started. Details of the work machine connection process (S101) will be described later in Figure 11.

[0100] As described above, the stopped state M103 is a state in which the prime mover 220 of the work machine 101 is stopped, the work machine 101 is stopped, and communication between the work machine 101 and the operating device 102 via the network NW is disconnected. The stopped state M103 indicates that the work machine 101 is not in use and that it is not possible to start work by remote operation. In the stopped state M103, the work machine 101 is not connected to any of the operating devices 102, and the prime mover 220 is stopped. From the stopped state M103, it is possible to directly transition to the idle state M102 and the manned operation state M104. On the other hand, it is not possible to directly transition from the stopped state M103 to the operation state M101.

[0101] The conditions for transitioning to the stopped state M103 are: "the prime mover of the work machine 101 is stopped," "the work machine 101 is in a parked position," "the stairs 230 of the work machine 101 are retracted," and "the work machine 101 is located within the stopping area."

[0102] The stairs 230 refer to the retractable stairs 230 used when boarding the work machine 101. For example, it is possible to determine whether the retractable stairs are stored by checking if the state of the stairs 230 obtained by the machine control unit 106 is a value that represents the stored state.

[0103] The stopping area refers to the range in which the work machine 101 can be in a stopped state M103. Specifically, the stopping area is a parking area in close proximity to the mining site in a mine. For example, it is possible to determine whether the work machine 101 is within the stopping area by checking whether the latitude and longitude obtained from the positioning sensor 601 attached to the upper rotating body 202 fall within a specific range.

[0104] The state transition process (S207) when transitioning from the idle state M102 to the stopped state M103 is the work machine stop process (S104). In the work machine stop process (S104), the prime mover of the work machine 101 and various devices are stopped. Details of the work machine stop process (S104) will be described later in Figure 14.

[0105] As described above, the manned operation state M104 is a state in which an operator OP is on board the work machine 101 and operating the work machine 101, and communication between the work machine 101 and the control device 102 via the network NW is disconnected. The manned operation state M104 indicates that the work machine 101 is being used for manned operation and that work cannot be started by remote operation. In the manned operation state M104, the work machine 101 is not connected to any of the control devices 102. From the manned operation state M104, it is possible to directly transition to the idle state M102 and the stopped state M103. On the other hand, it is not possible to directly transition from the manned operation state M104 to the working state M101.

[0106] There are no specific state transition conditions when transitioning from idle state M102 and stopped state M103 to manned operation state M104. Furthermore, there are no specific state transition processes (S207) performed by the management server 103 when transitioning to manned operation state M104. The operator OP, who is on board the work machine 101, performs the manned operation processes (S105, S107). Therefore, in the state transition process (S200) from idle state M102 and stopped state M103 to manned operation state M104, it is sufficient to simply change the work machine state managed by the state management unit 116 to manned operation state M104.

[0107] The conditions for transitioning from the manned operation state M104 to the idle state M102 are: "the prime mover 220 of the work machine 101 is operating," "the work machine 101 is in a parked position," and "the work machine 101 is located within the idle area." These conditions overlap with those described above in the work machine cutting process (S102), which is the state transition process (S207) from the working state M101 to the idle state M102, so the explanation is omitted.

[0108] The state transition process (S207) when transitioning from the manned operation state M104 to the idle state M102 is the same as the machine cutting process (S102) in the state transition process (S207) when transitioning from the working state M101 to the idle state M102 described above, and is the machine idle process (S106).

[0109] The conditions for transitioning from the manned operation state M104 to the stopped state M103 are: "the prime mover 220 of the work machine 101 is stopped," "the work machine 101 is in a parked position," "the stairs 230 of the work machine 101 are retracted," and "the work machine 101 is located within the stopping area." These conditions overlap with the content described above in the work machine stopping process (S104), which is the state transition process (S207) from the idle state M102 to the stopped state M103, so the explanation is omitted.

[0110] The state transition process (S207) when transitioning from the manned operation state M104 to the stopped state M103 is the same as the machine stop process (S104) in the state transition process (S207) when transitioning from the idle state M102 to the stopped state M103 described above, and is the machine stop process (S108).

[0111] In addition to the configuration shown in Figure 9, the state transition model may also include working machine states such as "under maintenance" that indicate communication interruption or equipment malfunction. Multiple state transition condition confirmation processes (S205) and state transition processes (S207) may be managed and switched according to information regarding time, operator OP, working machine 101, and operating device 102.

[0112] To summarize the operations in the above state transition process, the state management unit 116 refers to the working state M101, idle state M102, stopped state M103, and manned operation state M104, which are included in the working machine state recorded in the working machine management database 113.

[0113] The state management unit 116 determines that the state transition conditions are met when the work machine 101 is stopped, and based on information regarding the connection and disconnection of communication, it performs state transition processing that involves the connection of communication between the work machine 101 and the operating device 102 via the network NW, and state transition processing that involves the disconnection of communication between the work machine 101 and the operating device 102 via the network NW.

[0114] On the other hand, the state management unit 116 determines that the state transition conditions are not met when the work machine 101 is in operation, and therefore does not perform either the state transition process that involves establishing communication between the work machine 101 and the operating device 102 via the network NW, or the state transition process that involves disconnecting communication between the work machine 101 and the operating device 102 via the network NW. In other words, in this embodiment, establishing and disconnecting communication between the work machine 101 and the operating device 102 is performed only when the work machine 101 is stopped.

[0115] The state management unit 116 performs state transition processing that includes a direct state transition between the working state M101 and the idle state M102, state transition processing that includes a direct state transition between the idle state M102 and the stopped state M103, state transition processing that includes a direct state transition between the idle state M102 and the manned operation state M104, and state transition processing that includes a direct state transition between the stopped state M103 and the manned operation state M104.

[0116] On the other hand, the state management unit 116 does not perform state transition processing that involves a direct state transition between the working state M101 and the stopped state M103, nor does it perform state transition processing that involves a direct state transition between the working state M101 and the manned operation state M104. In other words, in this embodiment, it is not possible to directly perform state transition processing to the working state M101 without first going through the idle state M102.

[0117] The state management unit 116 refers to information regarding the combination of work machine 101 and operating device 102 that are connected to the network NW, as recorded in the work machine management database 113. The state management unit 116 performs a state transition process that involves changing the combination of work machine 101 and operating device 102 that are connected to the network NW, through a direct state transition from the working state to the idle state, followed by a direct state transition from the idle state M102 to the working state M101. In other words, in this embodiment, the change in the combination of work machine 101 and operating device 102 that are connected to the network NW is only performed during the state transition process between the working state M101 and the idle state M102.

[0118] The status management unit 116 refers to information recorded in the work machine management database 113 regarding whether the work machine 101, which is included in the work machine status, is stopped on a horizontal ground G with its bucket 205 in contact with the ground G.

[0119] The state management unit 116 determines that the state transition conditions are met when the work machine 101 is stopped on a horizontal ground G with the bucket 205 in contact with the ground G, and performs state transition processing that involves a direct state transition from the working state M101 to the idle state M102, a state transition processing that involves a direct state transition from the stopped state M103 to the idle state M102, a state transition processing that involves a direct state transition from the manned operation state M104 to the idle state M102, and a state transition processing that involves a direct state transition from the manned operation state M104 to the stopped state M103.

[0120] On the other hand, the state management unit 116 determines that the state transition conditions are not met when the work machine 101 is not stopped with its bucket in contact with the ground G on a horizontal surface G, and does not perform state transition processing that involves a direct state transition from the working state M101 to the idle state M102, state transition processing that involves a direct state transition from the stopped state M103 to the idle state M102, state transition processing that involves a direct state transition from the manned operation state M104 to the idle state M102, and state transition processing that involves a direct state transition from the manned operation state M104 to the stopped state M103.

[0121] In other words, in this embodiment, unless the work machine 101 is in a parked position with its bucket in contact with the ground G on a horizontal surface, the state transition process involving state transitions to the idle state M102 and the stopped state M103 will not be performed.

[0122] The state management unit 116 refers to the information regarding the received radio wave strength included in the work machine state recorded in the work machine management database 113. When the received radio wave strength exceeds the radio wave strength threshold, the state management unit 116 determines that the state transition condition is met and performs a state transition process that involves a direct state transition from the idle state M102 to the working state M101. On the other hand, when the received radio wave strength does not exceed the radio wave strength threshold, the state management unit 116 determines that the state transition condition is not met and does not perform a state transition process that involves a direct state transition from the idle state M102 to the working state M101. In other words, in this embodiment, when the received radio wave strength is low, the state transition process that involves a direct state transition from the idle state M102 to the working state M101 is not performed.

[0123] The state management unit 116 refers to information regarding communication delays included in the work machine status recorded in the work machine management database 113. When the communication delay is smaller than the communication delay threshold, the state management unit 116 determines that the state transition condition is met and performs a state transition process that involves a direct state transition from the idle state M102 to the working state M101. On the other hand, when the communication delay is not smaller than the communication delay threshold, the state management unit 116 determines that the state transition condition is not met and does not perform a state transition process that involves a direct state transition from the idle state M102 to the working state M101. In other words, in this embodiment, when the communication delay is large, the state transition process that involves a direct state transition from the idle state M102 to the working state M101 is not performed.

[0124] The state management unit 116 refers to information recorded in the work machine management database 113 regarding whether or not the work machine 101, which is included in the work machine status, is located in the idle area. When the work machine 101 is located in the idle area, the state management unit 116 determines that the state transition conditions are met and performs state transition processing involving a direct state transition from the working state M101 to the idle state M102, a state transition processing involving a direct state transition from the stopped state M103 to the idle state M102, and a state transition processing involving a direct state transition from the manned operation state M104 to the idle state M102.

[0125] On the other hand, the state management unit 116 determines that the state transition conditions are not met when the work machine is not located in the area where it can be paused, and does not perform state transition processing that involves a direct state transition from the working state to the paused state, state transition processing that involves a direct state transition from the stopped state to the paused state, and state transition processing that involves a direct state transition from the manned operation state to the paused state. In other words, in this embodiment, when the work machine 101 is not located in the area where it can be paused, state transition processing that involves a direct state transition to the paused state M102 is not performed.

[0126] The state management unit 116 refers to information recorded in the work machine management database 113 regarding whether or not the work machine 101, which is included in the work machine status, is located in the stoppable area. When the work machine 101 is located in the stoppable area, the state management unit 116 determines that the state transition conditions are met and performs state transition processing that involves a direct state transition from idle state M102 to stopped state M103 and state transition processing that involves a direct state transition from manned operation state M104 to stopped state M103.

[0127] On the other hand, the state management unit 116 determines that the state transition conditions are not met when the work machine 101 is located in a stoppable area, and does not perform state transition processing that involves a direct state transition from idle state M102 to stopped state M103, nor state transition processing that involves a direct state transition from manned operation state M104 to stopped state M103. In other words, in this embodiment, when the work machine 101 is not located in a stoppable area, state transition processing that involves a direct state transition to stopped state M103 is not performed.

[0128] The state management unit 116 refers to information in the work machine management database 113 regarding whether or not the stairs 230, which are included in the work machine status recorded therein, are stored. When the stairs 230 are stored therein, the state management unit 116 determines that the state transition conditions are met and performs state transition processing that involves a direct state transition from idle state M102 to stopped state M103 and state transition processing that involves a direct state transition from manned operation state M104 to stopped state M103.

[0129] On the other hand, the state management unit 116 determines that the state transition conditions are not met when the stairs 230 are not stored, and does not perform state transition processing that involves a direct state transition from the idle state M102 to the stopped state M103, nor does it perform state transition processing that involves a direct state transition from the manned operation state M104 to the stopped state M103. In other words, in this embodiment, when the stairs 230 are not stored, state transition processing that involves a direct state transition to the stopped state M103 is not performed.

[0130] In addition, in this embodiment, other information is also referenced in the state transition process described above. The state management unit 116 refers to the state transition conditions added for each operator OP recorded in the operator management database 115. The state management unit 116 performs the state transition process when it determines that the state transition conditions added for each operator OP are met, and does not perform the state transition process when it determines that the state transition conditions added for each operator OP are not met. In other words, in this embodiment, the state transition conditions are changed for each operator OP.

[0131] The state management unit 116 refers to the additional state transition conditions for the work machine 101 recorded in the work machine management database 113. The state management unit 116 performs a state transition process when it determines that the additional state transition conditions for each work machine 101 are met, and does not perform a state transition process when it determines that the additional state transition conditions for each work machine 101 are not met. In other words, in this embodiment, the state transition conditions are changed for each work machine 101.

[0132] Figure 10 is a flowchart showing the procedure for the state transition processing (S200) of the work machine 101 in this embodiment. The specific steps are described below. In S201, the state management unit 116 starts the state transition processing (S200). In S202, the state management unit 116 receives a state transition request transmitted from a mobile terminal 500 operated by the operator OP. The state transition request includes an identifier that uniquely identifies the target work machine 101 (work machine ID), the desired state of the work machine to transition to, and an identifier that uniquely identifies the operator OP who sent the state transition request (operator ID). Note that the transmission of the state transition request is not limited to the mobile terminal 500, but can also be transmitted from the operation unit 110, the machine control unit 106, another system, etc., and from any location.

[0133] In S203, the state management unit 116 determines whether a state transition is possible based on the state transition request. In this step, it checks whether a transition path exists and whether the operator's authority is appropriate. The transition path is the arrow in the state transition model shown in Figure 9. In the state transition model, a state transition that is not connected by an arrow is determined to be impossible. For example, if the state management unit 116 receives a state transition request from the stopped state M103 of the work machine 101 to the working state M101, it determines that the state transition is impossible.

[0134] Operator privileges are the privilege information of operator OPs, which are associated with the operator ID shown in Figure 8. Depending on the privileges, the ability to transition between states is set. Specifically, "operator OPs who manage the entire system are allowed to transition between all work machine states," and "general operator OPs are only allowed to transition between states from idle state M102 to working state M101, and from working state M101 to idle state M102," etc. This step makes it possible to prevent inappropriate system operation, such as work machine 101 being stopped by an operator OP without the proper privileges, which would reduce work efficiency.

[0135] In S204, if it is determined in S203 that a state transition is possible, the state management unit 116 acquires sensor information, machine control information, and operating device information from the work machine information collection unit 107 to be used in the state transition condition confirmation process S205. Note that only some of the information may be acquired based on the state of the work machine at the destination and source of the transition, the work machine ID, the operator ID, etc.

[0136] In S205, based on the state transition model shown in Figure 9, the state management unit 116 performs state transition condition confirmation processing using the information acquired in S204. The state transition processing confirmation processing, as described above with reference to Figure 9, includes checking the parking posture, checking the position of the work machine 101, checking the communication delay, checking the received radio wave strength, checking the state of the prime mover 220, and checking the stairs 230. Performing this step before the state change processing (S607) makes it possible to improve the stability of the work site before connecting and disconnecting the work machine 101 and the operating device 102. This step may be omitted if necessary.

[0137] In S206, the state management unit 116 checks whether it was determined in S205 that the state transition conditions were met. If S205 is omitted, it is assumed that the state transition conditions were met. In S207, the state management unit 116 performs state transition processing on the work machine 101 and the operating device 102 based on the state transition model. The state transition processing includes work machine connection processing (S101), work machine cutting processing (S102), work machine startup processing (S103), and work machine stop processing (S104), as described above with reference to Figure 9. Details of each process will be described later in Figures 11 to 14. This step may be omitted if necessary.

[0138] In S208, the state management unit 116 determines whether the state transition process in S207 was successful. If step S207 is omitted, the state transition process is deemed to have been successful. In S209, the state management unit 116 transitions the state of the work machine in the work machine management DB 113. In S210, the state management unit 116 notifies the request sender of the result, including whether the state transition process (S200) was successful or not. If the state transition process failed, the state management unit 116 also notifies the cause of the failure. In S211, the state management unit 116 terminates the state transition process (S200).

[0139] Figures 11 to 14 will be used to explain the state transition process in more detail. Figure 11 is a sequence diagram showing the procedure for starting remote operation on the work machine 101 in the idle state M102 in this embodiment. The operator OP assigns the control device 102 and connects the work machine 101 and the control device 102 through the management server 103. Before connecting, the management server 103 checks the state transition conditions for the work machine 101, and if it is confirmed that the state transition conditions are met, it performs the work machine connection process (S101). The specific procedure will be explained below. Note that the contents described above in Figures 9 and 10 will not be explained again.

[0140] In S301, the operator OP accesses the management server 103 from the mobile terminal 500 and logs into the system. The operator OP sends an authentication request to the management server 103, including the operator ID and authentication password. The management server 103 authenticates the operator based on the authentication request and sends an authentication response to the operator, including the authentication result. Note that authentication may use methods such as TLS (Transport Layer Security).

[0141] In S302, the operator OP selects the control device 102 for which they request an assignment. The operator OP sends a control device assignment request to the management server 103, which includes their operator ID and an identifier (control device ID) that uniquely identifies the control device 102 for which they request an assignment. In S303, the management server 103 assigns the requested control device 102 to the operator OP if its status is "unused," and does not assign it if its status is "in use." If the management server 103 decides to assign it, it changes the status of the control device it manages to "in use." After that, the management server 103 sends a control device assignment response to the operator OP, which includes information regarding the success or failure of the assignment.

[0142] In S304, the operator OP assigned to the control device 102 connects the work machine 101 to the control device 102 in order to start work. The operator OP sends a state transition request to the management server 103, which includes their operator ID, the work machine ID for which connection is requested, and the state of the work machine to which the transition is requested. Upon receiving the state transition request, the management server 103 performs a state transition feasibility determination (S203), obtains information on the work machine 101 (S204), and performs a state transition condition confirmation process (S205). If it is determined that a state transition is possible and the state transition conditions are met, the management server 103 performs the work machine connection process (S101).

[0143] In the machine connection process (S101), the video signal connection process is performed first (S401).

[0144] In the video signal connection process, the settings of the image transmission unit 108 of the work machine 101 are changed (S401α). The management server 103 sends a setting change request to the image transmission unit 108, including the setting items and setting content, and the work machine 101 changes the image transmission destination and whether or not image transmission is required. As a result, the image transmission unit 108 can transmit a video signal to the operating device 102.

[0145] In addition to the above, the resolution of the transmitted image, the coding rate of the transmitted image, the color information of the transmitted image, the refresh rate and error correction coding rate of the transmitted image may be further changed in the image transmission unit 108. Also, if it is not possible to set whether or not to transmit an image in the work machine 101, it is not necessary to set it. Furthermore, the work machine 101 may further change the image reception buffer amount of the image reception unit 111 of the operating device 102. The above setting changes may be made according to the operator OP, the work machine 101 and the operating device 102, and the setting values ​​may be changed accordingly.

[0146] If it is difficult to provide an API (Application Programming Interface) for receiving setting change requests in the image transmission unit 108, the settings of the work machine 101 may be changed by remote operation using SSH (Secure Shell) and a headless browser, etc.

[0147] In the work machine connection process (S101), the next step is the operation signal connection process (S402). In the operation signal connection process, the settings of the operation unit 110 of the operation device 102 are changed (S402α). The management server 103 sends a setting change request to the operation unit 110, including the setting items and setting content, and the operation device 102 changes the destination of the operation signal and whether or not to send the operation signal. As a result, the operation unit 110 can send an operation signal to the work machine 101.

[0148] If the operating device 102 cannot set whether or not to send an operation signal, it is not necessary to set it. If it is difficult to provide an API for receiving setting change requests in the operating unit 110, the operating unit 110 may have its settings changed by remote operation using SSH and a headless browser, etc.

[0149] In other words, in this embodiment, when the state management unit 116 of the management server 103 performs state transition processing that involves communication between the work machine 101 and the operating device 102 via the network NW, it establishes a connection between the machine control unit 106 of the work machine 101 and the operating unit 110 of the operating device 102 after the connection between the image transmission unit 108 of the work machine 101 and the image receiving unit 111 of the operating device 102 has been established.

[0150] At least one of the work machine 101 and the operating device 102 is further equipped with a firewall to block unauthorized communications. When the state management unit 116 of the management server 103 performs state transition processing that involves establishing communication between the work machine 101 and the operating device 102 via the network NW, it establishes a connection between the image transmission unit 108 of the work machine 101 and the image receiving unit 111 of the operating device 102, and establishes a connection between the machine control unit 106 of the work machine 101 and the operation unit 110 of the operating device 102, and then configures the firewall to open communication between the work machine 101 and the operating device 102 via the network NW (S403).

[0151] The management server 103 sends a firewall configuration request to the work machine 101 and the operating device 102. After the work machine 101 and the operating device 102 configure the firewall, they return the firewall configuration request to the management server 103.

[0152] Furthermore, in the firewall settings, by configuring it to block communication between the work machine 101 and the operating device 102 before the setting change, and then configuring it to open communication between the work machine 101 and the operating device 102 after the setting change, it is possible to connect the video signal and the operating signal simultaneously. In other words, regardless of the order of the video signal connection process (S401) and the operating signal connection process (S402), it is possible to improve the stability of the connection between the work machine 101 and the operating device 102.

[0153] After configuring the firewall, the management server 103 transitions the working machine status to the working state M101 (S209) and notifies the operator OP of the result of the status transition process. Since the connection between the working machine 101 and the operating device 102 is complete, the operator OP starts working (S305).

[0154] According to the procedure shown in Figure 11, in the state transition process (S200), the work machine connection process (S101) is performed after the state transition condition confirmation process (S205). This improves the stability before the connection between the work machine 101 and the operating device 102. Furthermore, in the work machine connection process (S101), the operation signal connection process (S402) is performed after the video signal connection process (S401). This prevents the operator from transmitting an operation signal before they can grasp the work environment via video, thereby improving the stability during the connection between the work machine 101 and the operating device 102.

[0155] Figure 12 is a sequence diagram showing the procedure for terminating remote operation on the work machine 101 in working state M101 in this embodiment. The operator OP disconnects communication between the work machine 101 and the control device 102 via the management server 103. Before disconnecting communication, the management server 103 performs a state transition condition confirmation process for the work machine 101, and if it is confirmed that the state transition condition is met, it performs the work machine disconnection process (S102). The specific procedure will be described below. Note that the contents described above in Figures 9 and 10 will not be explained again.

[0156] As shown in Figure 12, after operator OP authentication (S501), operator OP disconnects communication between the work machine 101 and the control device 102 in order to complete the work. Operator OP sends a state transition request to the management server 103, including their operator ID and the state of the work machine to which they wish to transition (S502). Upon receiving the state transition request, the management server 103 performs a state transition feasibility determination (S203), acquires information on the work machine 101 (S204), and performs a state transition condition confirmation process (S205). If the management server 103 determines that a state transition is possible and that the state transition conditions are met, the management server 103 performs a work machine disconnection process (S102).

[0157] When the state management unit 116 of the management server 103 performs a state transition process that involves disconnecting communication between the work machine 101 and the operating device 102 via the network NW, it sets up a firewall to block communication between the work machine 101 and the operating device 102 via the network NW before establishing a disconnection between the machine control unit 106 of the work machine 101 and the operating unit 110 of the operating device 102, and before establishing a disconnection between the image transmission unit 108 of the work machine 101 and the image receiving unit 111 of the operating device 102 (S601). The management server 103 sends a firewall configuration request to the work machine 101 and the operating device 102. After the work machine 101 and the operating device 102 configure the firewall, they return the firewall configuration request to the management server 103.

[0158] Furthermore, by configuring the firewall settings to block communication between the work machine 101 and the operating device 102, it is possible to disconnect both the video signal and the operating signal simultaneously. In other words, regardless of the order of the operating signal disconnection process (S602) and the video signal disconnection process (S603), it is possible to improve the stability during disconnection between the work machine 101 and the operating device 102.

[0159] In the machine cutting process (S102), the operation signal cutting process is performed first (S602). In the operation signal cutting process, the settings of the operation unit 110 of the operation device 102 are changed (S602α). The management server 103 sends a setting change request to the operation unit 110, including the setting items and setting content, and the operation device 102 changes whether or not operation signal transmission is required. As a result, the operation unit 110 can stop transmitting operation signals to the machine 101. When the state management unit 116 of the management server 103 performs a state transition process that involves disconnecting communication between the machine 101 and the operation device 102 via the network NW, it sets the destination address of the operation unit 110 of the operation device 102 to the address of the management server 103.

[0160] In addition to the above, if the operating device 102 cannot set whether or not to send an operation signal, the operating device 102 may set the destination of the operation signal to itself. If it is difficult to provide an API for receiving setting change requests in the operation unit 110, the operation unit 110 may change its settings by remote operation using SSH and a headless browser, etc.

[0161] In the machine cutting process (S102), the next step is to perform a video signal cutting process (S603). In the video signal cutting process, the settings of the image transmission unit 108 of the machine 101 are changed (S603α). The management server 103 sends a setting change request to the image transmission unit 108, including the setting items and setting content, and the machine 101 changes whether or not it needs to transmit images. As a result, the image transmission unit 108 can stop transmitting video signals to the operating device 102. When the state management unit 116 of the management server 103 performs a state transition process that involves disconnecting communication between the machine 101 and the operating device 102 via the network NW, it sets the destination address of the image transmission unit 108 of the machine 101 to the address of the management server 103.

[0162] In addition to the above, if the work machine 101 cannot set whether or not to transmit images, it may set itself as the destination for transmitting the image signal. If it is difficult to provide an API for receiving setting change requests in the image transmission unit 108, the image transmission unit 108 may have its settings changed by remote operation using SSH and a headless browser, etc.

[0163] In other words, in this embodiment, when performing a state transition process that involves disconnecting communication between the work machine 101 and the operating device 102 via the network NW, the disconnection between the machine control unit 106 of the work machine 101 and the operating unit 110 of the operating device 102 is established first, and then the disconnection between the image transmission unit 108 of the work machine 101 and the image receiving unit 111 of the operating device 102 is established.

[0164] After the video signal disconnection process is complete, the management server 103 transitions the working machine state to the idle state M102 (S209) and notifies the operator OP of the state transition process. The operator OP releases the operating device 102 that is currently in use (S503). The operator OP sends an operating device release request to the management server 103, which includes their operator ID. Based on the operator ID, the management server 103 identifies the operating device 102 that the operator OP is using and changes the operating device state to "unused" (S504). Subsequently, the management server 103 sends an operating device assignment response to the operator OP, which includes information regarding the success or failure of the release. Since the communication between the working machine 101 and the operating device 102 has been disconnected and the operating device 102 has been released, the operator OP terminates their work.

[0165] According to the procedure shown in Figure 12, in the state transition process, the work machine disconnection process (S102) is performed after the state transition condition confirmation process (S205). This improves the stability of communication between the work machine 101 and the operating device 102 before disconnection. Furthermore, in the work machine disconnection process (S102), the video signal disconnection process (S603) is performed after the operation signal disconnection process (S602). This prevents the operator OP from transmitting an operation signal after they can no longer perceive the work environment via video, thereby improving the stability of communication between the work machine 101 and the operating device 102 during disconnection.

[0166] Figure 13 is a sequence diagram showing the procedure for starting up the work machine 101 in this embodiment. The operator OP starts up the work machine 101 via the management server 103. That is, this procedure makes it possible to establish a communication connection between the work machine 101 and the operating device 102. Before starting up, the management server 103 checks the state transition conditions for the work machine 101, and if it is confirmed that the state transition conditions are met, it performs the work machine start-up process (S103). The specific procedure will be described below. Note that the contents described above in Figures 9 and 10 will not be explained again.

[0167] After operator OP authentication (S701), operator OP starts up the work machine 101. The operator sends a state transition request to the management server 103, including their operator ID and the state of the work machine to which they wish to transition (S702). Upon receiving the state transition request, the management server 103 performs a state transition feasibility determination (S203), obtains information on the work machine 101 (S204), and performs a state transition condition confirmation process (S205). If it is determined that a state transition is possible and the state transition conditions are met, the work machine start-up process (S103) is performed.

[0168] In the machine startup process (S503), the prime mover start-up process is performed first (S801). In the prime mover start-up process, the machine control unit 106 of the machine 101 starts the prime mover 220. The management server 103 sends a prime mover start-up request to the machine control unit 106, and the machine control unit 106 starts the prime mover 220 (S801α). Subsequently, the machine control unit 106 sends a prime mover start-up response to the management server 103, including whether the prime mover 220 was started successfully. After the prime mover 220 has started up normally, power is supplied to all computers.

[0169] In the machine startup process (S103), the next step is to perform the computer startup process (S802). In the computer startup process, each computer is started using, for example, WoL (Wake-on-LAN). WoL is a mechanism that allows computers to be started by sending a predetermined signal to a network interface device. The management server 103 sends a computer startup request to computers such as the communication device 207, the vehicle controller 209, and the information gathering device 210, which are always running. Subsequently, the computers send a predetermined signal to the stopped computers. After that, the computers perform their own startup process (S802α) and send a computer startup response to the management server 103, including whether the computer startup was successful or not. This makes it possible to start any computer equipped on the machine 101.

[0170] In addition to the above, the power-saving mode of a computer operating in power-saving mode may be deactivated in response to a computer startup request. The computer may also be started when power is supplied after the prime mover 220 has started up normally. Instead of waiting for a computer startup response, the management server 103 may use a mechanism such as ICMP (Internet Control Message Protocol) echo to confirm the success or failure of the computer startup. After the computer startup process (S802α) is completed, the management server 103 transitions the machine status to the idle state M102 (S209) and notifies the operator OP of the result of the state transition process.

[0171] According to the procedure shown in Figure 13, in the state transition process (S200), the work machine startup process (S103) is performed after the state transition condition confirmation process (S205). This makes it possible to improve the stability of the work machine 101 before startup.

[0172] Figure 14 is a sequence diagram showing the procedure for stopping the idle work machine 101 in this embodiment. The operator stops the work machine 101 through the management server 103. That is, the communication connection between the work machine 101 and the operating device 102 is made impossible. Before stopping, the management server 103 checks the state transition conditions for the work machine 101, and if it is confirmed that the state transition conditions are met, it performs the work machine stop process (S104). The specific procedure will be described below. Note that the contents described above in Figures 9 and 10 will not be explained again.

[0173] After operator OP authentication (S901), operator OP starts up the work machine 101. Operator OP sends a state transition request to the management server 103, including their operator ID and the state of the work machine to which they wish to transition (S902). Upon receiving the state transition request, the management server 103 performs a state transition feasibility determination (S203), obtains information on the work machine 101 (S204), and performs a state transition condition confirmation process (S205). If it is determined that a state transition is possible and the state transition conditions are met, the management server performs a work machine stop process (S104).

[0174] In the machine shutdown process (S104), the computer shutdown process is performed first (S1001). In the computer shutdown process, the computer is stopped using, for example, a dedicated API provided by the computer. The management server 103 sends a computer shutdown request to computers such as the image transmission device 211 that do not need to operate in the shutdown state M103. A computer that receives a computer shutdown request performs its own shutdown process (S1001α) and sends a computer shutdown response to the management server 103, including whether the computer shutdown was successful or not. This makes it possible to stop any computer of the machine 101.

[0175] In addition to the above, a computer that receives a computer shutdown request may switch to power-saving mode. Instead of waiting for a computer shutdown response, the management server 103 may use a mechanism such as ICMP echo to confirm the success or failure of the computer shutdown. If it is difficult for a computer to have an API to accept computer shutdown requests, the computer shutdown process may be performed remotely using SSH and a headless browser.

[0176] In the machine stop process (S104), the next step is to stop the prime mover (S1002). In the prime mover stop process, the machine control unit 106 of the machine 101 stops the prime mover 220. The management server 103 sends a prime mover stop request to the machine control unit 106, and the machine control unit 106 stops the prime mover 220 (S1002α). Subsequently, the machine control unit 106 sends a prime mover stop response to the management server 103, including whether the prime mover stop was successful or not. After the prime mover 220 is stopped, the power supply to some computers and hydraulic electric auxiliary equipment is stopped. After the completion of the prime mover stop process (S1002), the management server 103 transitions the machine state to the stopped state M103 (S209) and notifies the operator OP of the result of the state transition process.

[0177] According to the procedure shown in Figure 14, in the state transition process (S200), the work machine stop process (S104) is performed after the state transition condition confirmation process (S205). This makes it possible to improve the stability of the work machine 101 before it is stopped.

[0178] A remote control system with an N-to-M connection features an interface that allows for configuration changes and reloading without restarting the work machine and control device, and enables connection switching with a simple mechanism in response to requests from the operator. In an N-to-M remote control system, efficient remote control requires switching the connection between the work machine and the control device while considering the status of the operator and the work machine. If the status of the operator and the work machine is not sufficiently considered, efficiency at the work site will decrease due to issues such as "switching the connection to a work machine that is in operation" or "an operator who is not working smoothly continuing to work."

[0179] To address the above challenges, one possible approach is to detect instability in the operation of the work machine and the operator, and switch operators when instability is detected. Based on information from sensors and input devices such as buttons, instability such as "the work machine is not operating smoothly" or "the operator is unwell" can be detected. If instability is detected, the work will be handed over to a standby operator.

[0180] However, while such technology can improve the stability of a work machine during operation through remote control, it is difficult to improve the stability of communication between the work machine and the control device during connection and disconnection. For example, regarding the stability of communication during connection after a disconnection, there is the challenge of taking over a work machine in an unstable state. If the operator before the handover stops the work machine in an unstable state and disconnects the communication, the operator who reconnects the communication and takes over the work may not be able to determine that the situation is unstable.

[0181] Furthermore, regarding the stability of communication after a disconnection, there is a problem of erroneous operation due to delays in the connection of the video signal. In remote control systems, both video signals and control signals need to be transmitted, but in the connection process of communication between the work machine and the control device, the connection process of the video signal and the control signal is not necessarily completed simultaneously. Therefore, for example, if the connection of the video signal is delayed, a situation may occur where the video signal is not connected but the control signal is connected. That is, nothing is displayed on the display, but the operator is in a position to transmit control signals.

[0182] In this situation, the operator may rely on the display to determine the connection status between the work machine and the control device, potentially leading to incorrect operation. Specifically, the operator might assume that both the video signal and the control signal are disconnected, and then input commands into the control device to check its feel, potentially causing unintended operation.

[0183] On the other hand, in this embodiment, the connection and disconnection of communication between the work machine 101 and the operating device 102 is performed only when the work machine 101 is stopped. Therefore, since the connection and disconnection of communication are performed only when the work machine 101 is stopped and in a stable state, the state of the work machine 101 can be made more stable when switching between connecting and disconnecting communication between the work machine 101 and the operating device 102.

[0184] Furthermore, in this embodiment, it is not possible to directly transition to the operating state M101 without first going through the idle state M102. The idle state M102 is when the work machine 101 is stopped and the prime mover 220 of the work machine 101 is running. Therefore, when switching from disconnection to connection of communication between the work machine and the operating device, the work machine 101 can be stabilized and the state transition process to the operating state M101 can be performed quickly.

[0185] Furthermore, in this embodiment, changes in the combination of the work machine 101 and the operating device 102 while communication is connected are only performed during the state transition process between the working state M101 and the idle state M102. As described above, the state transition process between the working state M101 and the idle state M102 is performed quickly and while the work machine 101 is in a stable state. Therefore, changes in the combination of the work machine 101 and the operating device 102 while communication is connected can also be performed quickly and while the work machine 101 is in a stable state. This improves work efficiency.

[0186] Furthermore, in this embodiment, the state transition process involving state transitions to the idle state M102 and the stopped state M103 is not performed unless the work machine 101 is in a parked position with its bucket in contact with the ground G on a horizontal surface G. Therefore, the state transition process involving state transitions to the idle state M102 and the stopped state M103 can be performed when the work machine 101 is in a stable parked position, thereby improving the stability of the work machine 101.

[0187] Furthermore, in this embodiment, when the received radio wave strength is low, a state transition process involving a direct state transition from the idle state M102 to the working state M101 is not performed. Therefore, when the received radio wave strength is low and the communication condition is poor, it is possible to prevent the working machine 101 from undergoing a state transition process to the working state M101.

[0188] Furthermore, in this embodiment, when the communication delay is large, the state transition process involving a direct state transition from the idle state M102 to the working state M101 is not performed. Therefore, when the communication delay is large and the communication condition is poor, it is possible to prevent the state transition process to the working state M101 from being performed on the work machine 101.

[0189] Furthermore, in this embodiment, when the work machine 101 is not located in a rest area, a state transition process involving a direct state transition to the rest state M102 is not performed. Therefore, when the work machine 101 is located in a place unsuitable for transitioning to the rest state M102, it is possible to prevent the state transition process to the rest state M102 from being performed on the work machine 101.

[0190] Furthermore, in this embodiment, when the work machine 101 is not located in a stoppable area, a state transition process involving a direct state transition to the stopped state M103 is not performed. Therefore, when the work machine 101 is located in a place unsuitable for transitioning to the stopped state M103, it is possible to prevent the state transition process to the stopped state M103 from being performed on the work machine 101.

[0191] Furthermore, in this embodiment, when the stairs 230 are not stored, no state transition processing involving a direct state transition to the stopped state M103 is performed. Therefore, it is possible to prevent the state transition processing to the stopped state M103 from being performed on the work machine 101 when the stairs 230 are not stored and the work machine 101 is in an inappropriate state for transitioning to the stopped state M103.

[0192] Furthermore, in this embodiment, the state transition conditions are changed for each operator OP. This allows the state transition conditions to be changed appropriately according to the operator OP's authority, experience, and skills, thereby enabling stable state transition processing.

[0193] Furthermore, in this embodiment, the state transition conditions are changed for each work machine 101. This allows the state transition conditions to be changed appropriately according to the performance, weight, and load capacity of the work machine 101, enabling stable state transition processing.

[0194] Furthermore, in this embodiment, the sensor 104 of the work machine 101 includes at least one of the following: a positioning sensor 601, an acceleration sensor 602, a gyro sensor 603, a contact sensor 604, a three-dimensional sensor 605, a distance sensor 606, and a temperature sensor 607, so that various information about the work machine 101 can be obtained.

[0195] Furthermore, in this embodiment, when performing state transition processing that involves communication connection, the connection between the image transmission unit 108 of the work machine 101 and the image receiving unit 111 of the operating device 102 is established first, followed by the connection between the machine control unit 106 of the work machine 101 and the operating unit 110 of the operating device 102. When performing state transition processing that involves communication disconnection, the disconnection between the machine control unit 106 of the work machine 101 and the operating unit 110 of the operating device 102 is established first, followed by the disconnection between the image transmission unit 108 of the work machine 101 and the image receiving unit 111 of the operating device 102. As a result, the operator OP is always given image information when switching between a state where the work machine 101 can be operated and a state where the work machine 101 cannot be operated. Therefore, the remote operation of the work machine 101 by the operator OP can be made more stable.

[0196] Furthermore, in this embodiment, when performing a state transition process that involves disconnecting communication, the destination addresses of the operating unit 110 of the operating device 102 and the image transmission unit 108 of the work machine 101 are set to the address of the management server 103. This prevents inappropriate operations by the operator OP when performing a state transition process that involves disconnecting communication, and ensures the security of information.

[0197] Furthermore, in this embodiment, the firewall is configured to open communication after the communication connection is completed, and to block communication before the communication disconnection is completed, thereby ensuring the security of information.

[0198] Although several embodiments of the present invention have been described above, the invention is not limited to the embodiments described above and can be realized in various configurations without departing from the spirit of the invention. For example, configurations that arbitrarily combine or omit the configurations of the above embodiments can easily be conceivable. These variations are included in the scope of the invention described in the claims and its equivalents. [Explanation of Symbols]

[0199] 100…Remote control system 101...Work Machinery 102...Operating device 103... Management Server 104...Sensor 105...Camera 106...Machine Control Unit 107...Work Machinery Information Collection Department 108...Image transmission section 109A...Work Machinery Communications Department 109B...Operation device communication section 109C...Management Server Communication Unit 110...Operation unit 111...Image receiving unit 112…Display 113...Work Machinery Management Database 114...Operating device management database 115…Operator Management Database 116...Status Management Department 119...Communication device 201... Crawler 202… Upper rotating body 203... Boom 204...arm 205... Bucket 206...Manned control room 207...Communication equipment 208… Antenna 209... Vehicle controller 210... Information gathering device 211...Image transmission device 220... Engine 230...Stairs 301...Operation seat 302...Control stick 303... Control Room Controller 304...Image receiving device 307...Communication equipment 401…CPU 402...Memory 403…Auxiliary storage device 404…Communication Interface 405… Input / Output Interface 500… Mobile devices 601... Positioning sensor 602...Accelerometer 603...Gyroscope 604... Contact sensor 605... Three-dimensional sensor 606... Distance sensor 1101...Work Machine Management Table 1102...Work Machinery Configuration and Equipment Management Table 1201...Operating device management table 1202...Operating device configuration device management table 1301... Operator Management Table M101...Work in progress M102...Currently inactive M103...Stopped state M104…Manned operation status NW...Network OP... Operator G...ground

Claims

1. A work machine that is remotely controlled by communication over a network, An operating device for an operator to remotely control the work machine via the aforementioned network, A management server that manages the connection and disconnection of the communication between the work machine and the operating device via the network, A remote control system comprising, The aforementioned work machine is A work machine communication unit connected to the aforementioned network, A sensor for obtaining information about the aforementioned work machine, A camera for photographing the surroundings of the aforementioned work machine, A machine control unit that controls the operation of the work machine in response to operation commands received from the operating device via the work machine communication unit, An image transmission unit that processes and encodes the image acquired from the camera and transmits it to the operating device via the work machine communication unit, A work machine information collection unit transmits information collected from the sensor and the machine control unit to the management server via the work machine communication unit. Equipped with, The aforementioned operating device is A communication unit of the operating device connected to the aforementioned network, An image receiving unit that decodes the image received from the aforementioned work machine via the operating device communication unit, The image receiving unit displays the image that has been decoded, An operating unit that transmits the operation command to the aforementioned work machine, Equipped with, The aforementioned management server A management server communication unit connected to the aforementioned network, A work machine management database records the work machine status related to the state of the work machine based on information received from the work machine information collection unit of the work machine via the management server communication unit, A state management unit determines whether the state of the work machine satisfies predetermined state transition conditions, and when it determines that the state transition conditions are met, it performs state transition processing of the work machine state in the work machine management database. Equipped with, The status of the work machine recorded in the work machine management database includes: Information relating to the operation or stopping of the aforementioned work machine, Information relating to the connection or disconnection of the communication between the work machine and the operating device via the network, Includes, The aforementioned status management unit, Referencing the status of the work machine recorded in the work machine management database, When the work machine is stopped, it is determined that the state transition condition is met, and based on the information regarding the connection or disconnection of the communication, the state transition process that involves the connection of the communication between the work machine and the operating device via the network, or the state transition process that involves the disconnection of the communication, is performed. When the work machine is in operation, it is determined that the state transition conditions are not met, and neither the state transition process involving the connection of the communication between the work machine and the operating device via the network nor the state transition process involving the disconnection of the communication between the work machine and the operating device via the network is performed. A remote control system characterized by the following features.

2. The aforementioned work machine is The prime mover, which is the power source for the operation of the aforementioned work machine, Equipped with, The status of the work machine recorded in the work machine management database includes: The "working" state indicates that the prime mover of the work machine is operating, the work machine is in operation, and the communication between the work machine and the operating device via the network is established. A dormant state which indicates that the prime mover of the work machine is operating, the work machine is stopped, and the communication between the work machine and the operating device via the network is disconnected, A stopped state indicates that the prime mover of the work machine is stopped, the work machine is stopped, and the communication between the work machine and the operating device via the network is disconnected. The manned operation state indicates that the operator is on board the work machine and operating the work machine, and that the communication between the work machine and the operating device via the network is currently disconnected. Includes, The aforementioned status management unit, Referencing the working state, idle state, stopped state, and manned operation state included in the working machine status recorded in the working machine management database, The system performs the state transition process which involves a direct state transition between the working state and the paused state, the state transition process which involves a direct state transition between the paused state and the stopped state, the state transition process which involves a direct state transition between the paused state and the manned operation state, and the state transition process which includes a direct state transition between the stopped state and the manned operation state. The state transition process that involves a direct state transition between the working state and the stopped state and the state transition process that involves a direct state transition between the working state and the manned operation state are not performed. The remote control system according to feature 1.

3. The system comprises at least two of the aforementioned work machines and at least two of the aforementioned operating devices, The working machine status recorded in the working machine management database includes information regarding the combination of the working machine and the operating device that are connected via the network, The aforementioned status management unit, Referencing information regarding the combination of the work machine and the operating device that is connected to the network for communication, as recorded in the work machine management database, The state transition process involves a change in the combination of the work machine and the operating device, which are connected to the network for communication, through a direct state transition from the working state to the idle state, followed by a direct state transition from the idle state to the working state. The remote control system according to feature 2.

4. The aforementioned work machine is A bucket used for excavating the ground, Equipped with, The working machine status recorded in the working machine management database includes information regarding whether the working machine is stopped with its bucket in contact with the ground on a level surface. The aforementioned status management unit, Referencing information recorded in the aforementioned work machine management database regarding the status of the work machine, which includes whether the work machine is stopped on level ground with its bucket in contact with the ground, When the work machine is stopped on a level surface with its bucket in contact with the ground, it is determined that the state transition condition is met, and the state transition process is performed, which involves a direct state transition from the working state to the resting state, a state transition process which involves a direct state transition from the stopped state to the resting state, a state transition process which involves a direct state transition from the manned operation state to the resting state, and a state transition process which involves a direct state transition from the manned operation state to the stopped state. When the work machine is not stopped with its bucket in contact with the ground on a level surface, it is determined that the state transition conditions are not met, and the state transition process involving a direct state transition from the working state to the idle state, the state transition process involving a direct state transition from the stopped state to the idle state, the state transition process involving a direct state transition from the manned operation state to the idle state, and the state transition process involving a direct state transition from the manned operation state to the stopped state are not performed. The remote control system according to claim 3.

5. The working machine status recorded in the working machine management database includes information regarding the received radio wave strength received by the working machine via the network from the operating device. The aforementioned status management unit, Referencing the information regarding the received radio wave intensity included in the working machine status recorded in the working machine management database, When the received radio wave intensity exceeds the radio wave intensity threshold, it is determined that the state transition condition is met, and the state transition process is performed, which involves a direct state transition from the idle state to the working state. When the received radio wave intensity does not exceed the radio wave intensity threshold, it is determined that the state transition condition is not met, and the state transition process involving a direct state transition from the idle state to the working state is not performed. The remote control system according to feature 4.

6. The working machine status recorded in the working machine management database includes information regarding the communication delay of the communication between the working machine and the operating device via the network. The aforementioned status management unit, Referencing the information regarding the communication delay included in the working machine status recorded in the working machine management database, When the communication delay is smaller than the communication delay threshold, it is determined that the state transition condition is met, and the state transition process is performed, which involves a direct state transition from the idle state to the working state. When the aforementioned communication delay is not smaller than the aforementioned communication delay threshold, it is determined that the state transition condition is not met, and the state transition process involving a direct state transition from the idle state to the working state is not performed. The remote control system according to feature 5.

7. The status of the work machine recorded in the work machine management database includes information on whether or not the work machine is located in a rest area. The aforementioned status management unit, Referencing the information in the aforementioned work machine management database regarding whether the work machine included in the work machine status is located in the area where it can be stopped, When the work machine is located in the area where it can be stopped, it is determined that the state transition condition is met, and the state transition process is performed, which involves a direct state transition from the working state to the stopped state, the state transition process which involves a direct state transition from the stopped state to the stopped state, and the state transition process which involves a direct state transition from the manned operation state to the stopped state. When the work machine is not located in the area where it can be stopped, it is determined that the state transition conditions are not met, and the state transition process involving a direct state transition from the working state to the stopped state, the state transition process involving a direct state transition from the stopped state to the stopped state, and the state transition process involving a direct state transition from the manned operation state to the stopped state are not performed. The remote control system according to claim 6.

8. The status of the work machine recorded in the work machine management database includes information on whether or not the work machine is located in a stoppable area. The aforementioned status management unit, Referencing the information in the aforementioned work machine management database regarding whether the work machine included in the work machine status is located in the stoppable area, When the work machine is located in the stoppable area, it is determined that the state transition condition is met, and the state transition process involving a direct state transition from the idle state to the stopped state and the state transition process involving a direct state transition from the manned operation state to the stopped state are performed. When the work machine is not located in the stoppable area, it is determined that the state transition conditions are not met, and the state transition process involving a direct state transition from the idle state to the stopped state and the state transition process involving a direct state transition from the manned operation state to the stopped state are not performed. The remote control system according to feature 7.

9. The aforementioned work machine is Stairs that can be retracted and extended, Equipped with, The status of the work machine recorded in the work machine management database includes information on whether or not the stairs are stored. The aforementioned status management unit, Referencing the information in the aforementioned work machine management database regarding whether or not the stairs are stored in the work machine status recorded in the aforementioned work machine management database, When the stairs are stored, it is determined that the state transition condition is met, and the state transition process is performed, which involves a direct state transition from the idle state to the stopped state and the state transition process that involves a direct state transition from the manned operation state to the stopped state. When the stairs are not stored, it is determined that the state transition conditions are not met, and the state transition process involving a direct state transition from the idle state to the stopped state and the state transition process involving a direct state transition from the manned operation state to the stopped state are not performed. The remote control system according to feature 8.

10. The aforementioned management server An operator management database that records the state transition conditions added for each operator, Equipped with, The aforementioned status management unit, Referencing the state transition conditions added for each operator recorded in the operator management database, When it is determined that the state transition conditions added for each operator are met, the state transition process is performed. When it is determined that the state transition conditions added for each operator are not met, the state transition process is not performed. The remote control system according to feature 9.

11. The state of the work machine recorded in the work machine management database includes the state transition conditions added for each work machine, The aforementioned status management unit, Referencing the additional state transition conditions of the work machine recorded in the work machine management database, When it is determined that the state transition conditions added for each of the aforementioned work machines are met, the state transition process is performed. When it is determined that the state transition conditions added for each of the aforementioned work machines are not met, the state transition process is not performed. The remote control system according to feature 10.

12. The sensors of the aforementioned work machine include at least one of a positioning sensor, an acceleration sensor, a gyroscope, a contact sensor, a three-dimensional sensor, a distance sensor, and a temperature sensor. The remote control system according to feature 11.

13. A communication connection method in a remote control system according to claim 12, The state management unit of the management server is: When performing the state transition processing which involves the connection of the communication between the work machine and the operating device via the network, after the connection between the image transmission unit of the work machine and the image receiving unit of the operating device is established, the connection between the machine control unit of the work machine and the operating unit of the operating device is established. When performing the state transition process which involves disconnecting the communication between the work machine and the operating device via the network, after the disconnection between the machine control unit of the work machine and the operating unit of the operating device is established, the disconnection between the image transmission unit of the work machine and the image receiving unit of the operating device is established. A communication connection method characterized by the following:

14. The state management unit of the management server is: When performing the state transition process which involves disconnecting the communication between the work machine and the operating device via the network, the destination addresses of the operating unit of the operating device and the image transmission unit of the work machine are set to the address of the management server. The communication connection method according to feature 13.

15. At least one of the aforementioned work machine and the aforementioned operating device further comprises a firewall for blocking the unauthorized communications. The state management unit of the management server is: When performing the state transition processing which involves establishing the communication between the work machine and the operating device via the network, after establishing the connection between the image transmission unit of the work machine and the image receiving unit of the operating device, and after establishing the connection between the machine control unit of the work machine and the operating unit of the operating device, the firewall is configured to open the communication between the work machine and the operating device via the network. When performing the state transition process which involves disconnecting the communication between the work machine and the operating device via the network, the firewall is configured to block the communication between the work machine and the operating device via the network before the disconnection between the machine control unit of the work machine and the operating unit of the operating device is established, and before the disconnection between the image transmission unit of the work machine and the image receiving unit of the operating device is established. The communication connection method according to feature 14.

Images