Remote operation system and communication connection method

Abstract

The purpose of the present invention is to stabilize the state of a work machine during switching between connection and disconnection of communication between the work machine and an operation device. For this purpose, a management server comprises a management server communication unit, a work machine management database that records a work machine state on the basis of information received from a work machine via the management server communication unit, and a state management unit that performs state transition processing when determining that a state transition condition is satisfied. The state management unit refers to the work machine state in the work machine management database, when the work machine is being stopped, determines that the state transition condition is satisfied and performs state transition processing involving communication connection between the work machine and the operation device and state transition processing involving communication disconnection between the work machine and the operation device, and when the work machine is operating, determines that the state transition condition is not satisfied and does not perform either the state transition processing involving communication connection between the work machine and the operation device or the state transition processing involving communication disconnection between the work machine and the operation device.

Description

Remote operation system and communication connection method

[0001] The present invention relates to a remote operation system for managing a work machine that can be remotely operated and a communication connection method therefor.

[0002] With the development of wireless communication systems, it has become possible to provide high-quality wireless networks with large capacity and low latency. As a result, remote medical services that have been difficult to implement until now and remote applications that require high communication quality such as remote robot control are being considered. In the construction machinery and mining machinery industries, remote operation systems for work machines such as hydraulic excavators and bulldozers are also being studied for the purpose of reducing the movement burden of operators.

[0003] Conventionally, remote operation systems have been designed on the premise that the work machine to be operated by the operation device is not changed (1:1 connection). This is because it was difficult to stably transmit video with a low-quality wireless network, so a use case of operating the work machine while directly visualizing it was assumed. On the other hand, in recent years, stable video transmission is possible with a high-quality network, so it is possible to remotely operate from a place where the work machine cannot be directly visualized. Therefore, efficient operation such as having one operator in charge of multiple work machines located in a distant place and quickly switching and operating the work machine to be operated by the operator is being considered.

[0004] In a 1:1 connection remote operation system, the work machine and the operation device are connected according to a predetermined setting. When switching the connection between the work machine and the operation device, restarting is required for setting change and setting reading, and it takes time for the switching. Therefore, it was difficult to respond to use cases such as having one operator in charge of multiple work machines. In view of such a situation, a remote operation system has been studied on the premise that the combination of the work machine and the operation device can be flexibly changed (N:M connection) in order to improve the flexibility and efficiency of the system.

[0005] For example, Patent Document 1 discloses an invention that detects instability in the operation of a work machine and the operator, and switches operators when instability is detected. In the invention described in Patent Document 1, instability such as "the work machine is not operating smoothly" or "the operator is unwell" is detected based on information from sensors and input devices such as buttons. In the invention described in Patent Document 1, when instability is detected, the work is handed over to a standby operator.

[0006] Patent No. 7287047

[0007] However, in the invention described in Patent Document 1 above, the state of the work machine may become unstable when switching between connecting and disconnecting the work machine and the operating device. When the state of the work machine becomes unstable when switching between connecting and disconnecting the work machine and the operating device, it becomes difficult for the operator to quickly switch and operate the work machine in charge, resulting in a decrease in the work efficiency of the work machine.

[0008] The present invention has been made in view of the above problems, and aims to provide a remote control system and a communication connection method that stabilizes the state of a work machine when switching between connecting and disconnecting communication between the work machine and the operating device.

[0009] A representative example of the invention disclosed in this application is as follows: a remote control system comprising a work machine remotely controlled by communication over a network, an operating device for an operator to remotely control the work machine via communication over the network, and a management server that manages the connection and disconnection of communication between the work machine and the operating device over the network. The work machine comprises a work machine communication unit connected to the network, a sensor for obtaining information about the work machine, a camera for photographing the area around the work machine, a machine control unit that controls the operation of the work machine in response to operation commands received from the operating device via the work machine communication unit, an image transmission unit that processes and encodes images acquired from the camera and transmits them to the operating device via the work machine communication unit, and a work machine information collection unit that transmits information collected from the sensor and the machine control unit to the management server via the work machine communication unit. The operating device comprises an operating device communication unit connected to the network, an image receiving unit that decodes images received from the work machine via the operating device communication unit, a display that displays the images decoded by the image receiving unit, and an operating unit that transmits operation commands to the work machine. The management server comprises a management server communication unit connected to a network, a work machine management database that records work machine status based on information received from the work machine information collection unit of the work machine via the management server communication unit, and a status management unit that determines whether the work machine status satisfies predetermined status transition conditions, and performs status transition processing of the work machine status in the work machine management database when it is determined that the status transition conditions are met. The work machine status recorded in the work machine management database includes information regarding the operation or stopping of the work machine, and information regarding the connection or disconnection of communication between the work machine and the operating device via the network.The status management unit refers to the status of the work machine recorded in the work machine management database, determines that the status transition conditions are met when the work machine is stopped, and performs a status transition process that involves connecting or disconnecting communication via the network between the work machine and the operating device, based on information regarding the connection or disconnection of communication. When the work machine is in operation, it determines that the status transition conditions are not met, and does not perform either a status transition process that involves connecting or disconnecting communication via the network between the work machine and the operating device.

[0010] According to one aspect of the present invention, the state of the work machine can be made more stable when switching between connecting and disconnecting communication between the work machine and the operating device. Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

[0011] A block diagram showing the system configuration of the remote control system of the embodiment. A block diagram showing the sensors of the work machine of the embodiment. A schematic diagram showing the hardware configuration of the work machine of the embodiment. A schematic diagram showing the hardware configuration of the operating device of the embodiment. A block diagram showing the hardware configuration of the management server of the embodiment. A diagram showing the work machine status table of the work machine management database of the embodiment. A diagram showing the work machine configuration device management table of the work machine management database of the embodiment. A diagram showing the operating device management table of the operating device management database of the embodiment. A diagram showing the operator management table of the operator management database of the embodiment. A state transition diagram showing the state transition model of the work machine for the state management unit of the embodiment to perform state transition processing. A flowchart showing the procedure for the state transition processing of the work machine of the embodiment. A sequence diagram showing the procedure for starting remote control work on a work machine in a idle state of the embodiment. A sequence diagram showing the procedure for ending remote control work on a work machine in an active state of the embodiment. A sequence diagram showing the procedure for starting a work machine in a stopped state of the embodiment. A sequence diagram showing the procedure for stopping a work machine in a idle state of the embodiment.

[0012] Embodiments of this disclosure will be described in detail with reference to the drawings. In the embodiments, a hydraulic excavator will be used as an example of a work machine that constitutes a remote control system for a work machine. However, this disclosure is not limited to the embodiments shown below. The description herein is merely a typical example, and this disclosure can be implemented in various modified and improved forms based on the knowledge of those skilled in the art.

[0013] Examples will be described below with reference to the attached drawings. In the attached drawings, functionally identical elements are indicated by the same numbers. The attached drawings show examples in accordance with the principles of this disclosure, but they are for the purpose of understanding this disclosure and are not to be used in any way to restrict the interpretation of this disclosure. The descriptions in this specification are merely typical examples and do not limit the claims or applications of this disclosure in any way. In this specification and in the drawings, components with the same reference numerals refer to the same component.

[0014] Figure 1 is a block diagram showing the system configuration of the remote control system in this embodiment. Figure 1 is a block diagram created focusing on the functions of the system, and the hardware configuration that realizes it will be described later in Figures 3 to 5. The system operation overview and the functions of each component will be explained using Figure 1. The remote control system 100 comprises at least one work machine 101, at least one operating device 102, and a management server 103. In this embodiment, the remote control system 100 comprises at least two work machines 101 and at least two operating devices 102.

[0015] The work machine 101 is remotely controlled via communication over a network NW. The operator OP remotely controls the work machine 101 via communication over the network NW using the control device 102. The management server 103 manages the connection and disconnection of the network communication between the work machine 101 and the control device 102.

[0016] The work machine 101, the operating device 102, and the management server 103 are connected via a network and can communicate with each other using IP (Internet Protocol) or the like. Remote operation is achieved when the work machine 101 transmits a video signal to the operating device 102 to check the site environment, and the operating device 102 transmits control signals to the work machine 101. The management server 103 enables remote operation between any work machine 101 and the operating device 102 at the request of the operator.

[0017] Specifically, as will be described later, the system manages the status of the work machine 101, such as the working state and the idle state, and performs state transition processing for the work machine 101 (described later in S200 of Figures 9 and 10), which may include connecting or disconnecting communication between the work machine 101 and the control device 102, in response to requests from the operator. In this embodiment, by managing the status of the work machine 101, it is possible to prevent unstable states such as simultaneous operation by multiple operators, and to realize a system that can flexibly connect and switch between the work machine 101 and the control device 102.

[0018] Furthermore, in the state transition process (S200), state transition condition confirmation processes (described later in S205 of Figure 10) such as communication quality confirmation and parking attitude confirmation are performed to improve the stability of the work site before connecting and disconnecting the work machine 101 and the operating device 102. In addition, in the work machine connection process (described later in S101 of Figure 9) and the work machine disconnection process (described later in S102 of Figure 9), attention is paid to the order of connection and disconnection of communication between the video signal and the operating signal to improve stability during communication connection and disconnection. Specific details of the procedure will be described later in Figures 9 to 14.

[0019] As shown in Figure 1, the work machine 101 includes one or more sensors 104, one or more cameras 105, a machine control unit 106, a work machine information collection unit 107, an image transmission unit 108, and a work machine communication unit 109A. The sensors 104 are for obtaining information about the work machine 101. As shown in Figure 2, the types of sensors 104 include a positioning sensor 601, an acceleration sensor 602, a gyro sensor 603, a contact sensor 604, a three-dimensional sensor 605, a distance sensor 606, and a temperature sensor 607. For example, a GPS (Global Positioning System) can be used as the positioning sensor 601.

[0020] Camera 105 is for photographing the area around the work machine 101. Camera 105 is a device for acquiring images of the work environment. In this embodiment, camera 105 is configured to capture images of the area around the work machine 101, and to generate, store, and output both still images and moving images (video) as captured images.

[0021] The machine control unit 106 controls the operation of the work machine 101 in response to operation commands received from the operating device 102 via the work machine communication unit 109A. The machine control unit 106 interprets the operation signals received from the operating device 102 and controls the prime mover, such as the internal combustion engine, and actuators of the work machine 101. The machine control unit 106 also acquires machine control information related to the control of the work machine 101 (such as the state of the prime mover, the state of the actuators, the speed of the work machine 101, information on operation signals, the state of the manned operation interface, and the state of the storage stairs).

[0022] The machine information collection unit 107 transmits information collected from the machine communication unit 109A, the sensor 104, and the machine control unit 106 to the management server via the machine communication unit 109A. The machine information collection unit 107 has the function of collecting and storing information measured and acquired by the sensor 104, the machine control unit 106, and the machine communication unit 109A (information from the sensor 104, machine control information acquired by the machine control unit 106, and communication quality information acquired by the machine communication unit 109A). The information stored by the machine information collection unit 107 is provided to the state management unit 116 of the management server 103 for the state transition condition confirmation process described later.

[0023] The image transmission unit 108 processes and encodes the image acquired from the camera 105 and transmits it to the operating device 102 via the machine communication unit 109A. The image transmission unit 108 has the function of encoding the image acquired by the camera 105, reducing the amount of information, and transmitting it to the operating device 102.

[0024] The machine communication unit 109A connects to the network NW. The machine communication unit 109A functions as a relay between the internal network and the external network of the machine 101. The internal network refers to wired communication using Ethernet®, etc. The external network refers to wireless communication using Wi-Fi®, LTE®, 4G, 5G, etc. Furthermore, the machine communication unit 109A acquires communication quality information through active measurement between the machine communication unit 109A and any other communication device using measurement traffic, and through passive measurement within the machine communication unit 109A.

[0025] The communication quality information acquired by the machine communication unit 109A includes communication delay, packet loss rate, throughput, input / output traffic volume of the machine communication unit 109A, number of lost packets, RSSI (Received Signal Strength Indicator), RSRP (Reference Signal Received Power), RSRQ (Reference Signal Received Quality), SINR (Signal to Interface plus Noise Ratio), and SNR (Signal to Noise Ratio).

[0026] The operating device 102 comprises an operating device communication unit 109B, an operating unit 110, an image receiving unit 111, and a display 112. The operating device communication unit 109B is connected to a network NW. The functions of the operating device communication unit 109B are the same as those of the machine communication unit 109A described above, so their explanation is omitted. However, unlike the machine communication unit 109A, the operating device communication unit 109B usually communicates via wired connections to both the internal and external networks.

[0027] The operation unit 110 transmits operation commands to the work machine 101. The operation unit 110 has an interface that physically receives input from the operator OP and has the function of transmitting operation signals to the work machine 101 according to the input from the operator OP. The image receiving unit 111 decodes the image received from the work machine 101 via the operation device communication unit 109B. The image receiving unit 111 decodes the encoded image received from the work machine 101 and transmits it to the display 112. The display 112 displays the image decoded by the image receiving unit 111. The display 112 is a device that displays the image received from the image receiving unit 111.

[0028] The management server 103 comprises a management server communication unit 109C, a work machine management database 113, an operating device management database 114, an operator management database 115, and a status management unit 116. The management server communication unit 109C is connected to the network NW. The management server communication unit 109C is equivalent to the work machine communication unit 109A and the operating device communication unit 109B described above, so its description is omitted. Also, like the operating device communication unit 109B, the management server communication unit 109C usually communicates via wired connections to both the internal and external networks.

[0029] The work machine management database 113 records the work machine status related to the state of the work machine 101 based on information received from the work machine information collection unit 107 of the work machine 101 via the management server communication unit 109C. The work machine management database 113 has the function of managing the work machine status related to the usage status of the work machine 101, such as stopped state and operating state, as well as network information of the devices that constitute the work machine 101. The work machine status recorded in the work machine management database 113 includes information on when the work machine 101 is operating and stopped, and information on when communication between the work machine 101 and the operating device 102 via the network NW is connected and disconnected.

[0030] The working machine status is used for mutual exclusion control when connecting the working machine 101 and the operating device 102. For example, by managing the status of whether the working machine 101 is already connected to any of the operating devices 102, it is possible to prevent other operating devices 102 from connecting to that working machine 101 at the same time. In addition to this, it is also possible to manage statuses such as "operated by a person" and "malfunction adjustment in progress" and to control remote operation according to the status. The network information of the devices that make up the working machine 101 is used in the state transition processing described later to actually access the devices that make up the working machine 101 and perform processes such as "starting or stopping," "acquiring information," and "changing settings." Details of the working machine management DB 113 will be described later in Figures 6A and 6B.

[0031] The control device management database 114 is a function that manages the status of the control device 102, such as whether it is in use or not, and network information of the devices that make up the control device 102. The control device status is used to check the usage status of the control device 102. The network information of the devices that make up the control device 102 is used to access the devices and change settings when connecting and disconnecting communication between the work machine 101 and the control device 102. Details of the control device management DB 114 will be described later in Figures 7A and 7B.

[0032] The operator management database 115 records state transition conditions added for each operator OP, as will be described later. The operator management database 115 has the function of managing the operator OP's authorization information and authentication information for the remote control system 100. The authorization information is information for controlling the operations that each operator OP can execute. The authentication information is information for authenticating whether the request is legitimate when the operator OP executes a state transition process. Details of the operator management database 115 will be described later in Figure 8.

[0033] The state management unit 116 determines whether the state of the work machine satisfies predetermined state transition conditions, and when it determines that the state transition conditions are met, it performs state transition processing of the work machine state in the work machine management database 113. The state management unit 116 has the function of performing state transition processing of the work machine 101, assigning operators OP to the operating device 102, and providing an interface for performing state transition processing and operator assignment. State transition processing is executed when a request regarding state transition is received from the operator. After receiving the request, the state management unit 116 accesses the work machine information collection unit 107 of the work machine 101 and collects information regarding the work machine 101. After that, it performs state transition condition confirmation processing for the work machine 101 using the collected information. If it is determined that the state transition conditions are met, it performs processing (state transition processing) which may involve connecting or disconnecting the video signals and operating signals of the work machine 101 and the operating device 102.

[0034] State transition processing is performed using network information and other data of the devices constituting the work machine 101 and the operating device 102, which are managed in the work machine management database 113 and the operating device management database 114. Operator assignment is performed in response to a request from the operator OP. After being authenticated by the state management unit 116, the operator OP sends a request to use the operating device 102. Upon receiving the request, the state management unit 116 changes the state of the operating device 102 that the operator OP has requested to use to "in use".

[0035] State transition processing and operator assignment are performed through an interface provided by the state management unit 116. The interface is implemented, for example, as a web system. Operators can access the operating device 102 using HTTP (Hyper Text Transfer Protocol) from a web browser on their mobile terminal 500, such as a smartphone or tablet, and can send authentication and arbitrary requests via a GUI (Graphical User Interface). Note that the device capable of sending requests is not limited to the mobile terminal 500, but may also be, for example, the operating unit 110 or other systems.

[0036] In addition to the configuration shown in Figure 1, the work machine 101, operating device 102, and management server 103 may be connected by multiple different networks. The work machine communication unit 109A, the operating device communication unit 109B, and the management server communication unit 109C are equipped with the function of blocking specific traffic using a firewall and connecting to a VPN (Virtual Private Network), which is an encrypted network.

[0037] The work machine 101 may further include a microphone, and the image transmission unit 108 may encode the audio signal acquired from the microphone in addition to the video signal and transmit it to the operating device 102, and the operating device 102 may further include a speaker, and the image receiving unit 111 may decode the audio signal received from the work machine 101 and play it back through the speaker. The state management unit 116 may store the information acquired from the work machine information collection unit 107 and use that information to perform state transition condition confirmation processing. Multiple management servers 103 may be installed and the work machine 101 and the operating device 102 may be managed separately.

[0038] Figure 3 is a schematic diagram showing the hardware configuration of the work machine 101 in this embodiment. The work machine 101, which is a hydraulic excavator, is equipped with crawlers 201, an upper rotating body 202 supported and transported by the crawlers 201, a boom 203 mounted on the upper rotating body 202, an arm 204, a bucket 205 for excavating the ground G, and a manned operation room 206. The work machine 101 is also equipped with a prime mover 220, such as an internal combustion engine, which is the power source for the operation of the work machine 101. The work machine 101 is also equipped with stairs 230 that can be stored and extended. The stairs 230 are used for the operator OP to board the work machine 101 when it is in a manned operation state, which will be described later.

[0039] The upper rotating body 202 is equipped with a sensor 104, a camera 105, a communication device 207, and an antenna 208. The manned control room 206 is equipped with a vehicle controller 209, an information gathering device 210, and an image transmission device 211. The work machine 101 also includes hardware (not shown) such as a hydraulic actuator and hydraulic pump for driving the work machine 101, hydraulic electric auxiliary equipment to assist in the control of the hydraulic system, a generator driven by the prime mover 220 to generate electricity, a main power supply for storing electricity, a backup power supply for storing electricity, and cables for connecting various hardware.

[0040] The crawler 201 is a device that allows the work machine 101 to move forward, backward, left, and right. The work machine 101 can move by driving the crawler 201 with a hydraulic motor, which is a hydraulic actuator. The boom 203, arm 204, and bucket 205 are devices for excavating the ground G. The boom 203, arm 204, and bucket 205 are driven by a hydraulic cylinder, which is a hydraulic actuator, and perform lifting, extending, and excavating operations.

[0041] The upper rotating body 202 is configured to rotate relative to the crawler 201. The upper rotating body 202 can face any direction by driving a slewing hydraulic motor, which is a hydraulic actuator, enabling efficient excavation work. Each hydraulic actuator is powered by the prime mover 220 and receives power via a hydraulic pump.

[0042] The manned control room 206 is a space for manned operation. In addition to an operator's seat and a manned operation interface, the manned control room 206 is equipped with a vehicle controller 209, an information collection device 210, and an image transmission device 211. The vehicle controller 209, the information collection device 210, and the image transmission device 211 are computers that implement the machine control unit 106, the work machine information collection unit 107, and the image transmission unit 108 shown in Figure 1, respectively. Detailed hardware will be described later in Figure 5.

[0043] The vehicle controller 209 controls the operation of the work machine 101 by operating the hydraulic pump, the control valve for controlling the pressurized oil supplied from the hydraulic pump, the prime mover 220, etc., in response to the operation signals input by the operator. The vehicle controller 209 also acquires machine control information related to the work machine 101 (details are shown above in Figure 1) and provides it to the information collection device 210. Specific methods for acquiring machine control information include "estimating the angle, speed, position, and direction of the hydraulic actuator from the state of the hydraulic pump and control valve," "understanding the operating status of the prime mover 220 from the transmission history of operation signals to the prime mover 220," and "recording the input history of the manned operation interface."

[0044] The vehicle body controller 209 is a device that collects and stores information measured and acquired by the sensor 104, the communication device 207, and the ECUs (Electronic Control Units) of each part of the work machine 101. The vehicle body controller 209 and the sensor 104 are connected by wire using a shielded cable, a coaxial cable, etc., and collect information using communication protocols such as SPI (Serial Peripheral Interface), I2C (registered trademark), RS-232C (Recommended Standard-232C), Modbus (registered trademark), and UART (Universal Asynchronous Receiver Transmitter).

[0045] The vehicle body controller 209 and the communication device 207 are connected by wire using an UTP (Unshielded Twisted Pair) cable, an optical fiber cable, etc., and collect information using a communication protocol such as SNMP (Simple Network Management Protocol). The vehicle body controller 209 and the ECUs of each part of the work machine 101 are connected by wire using a shielded cable, an UTP cable, etc., and collect information using a communication protocol such as CAN (Controller Area Network).

[0046] The image transmission device 211 is a device that encodes and transmits an image acquired from the camera 105. The camera 105 is connected by wire using a coaxial cable, an UTP cable, etc., and acquires an image using a communication protocol such as RS-232C, SDI (Serial Digital Interface), GigE Vision (registered trademark). Encoding is performed by H.264, H.265, Motion JPEG, etc.

[0047] The communication device 207 installed on the upper rotating body 202 is a computer that realizes the machine tool communication unit 109A shown in FIG. 1. The communication device 207 relays the internal network and the external network of the machine tool 101. In addition, in the communication device 207 of the machine tool 101, conversion between wireless signals and wired signals is also performed via an antenna 208 which is a device for transmitting and receiving radio waves. The body controller 209 and the information collection device 210 are connected by wire using a UTP cable, an optical fiber cable, etc., and communicate using a communication protocol such as IP.

[0048] Each of the above various devices receives power supply from the main power supply and the backup power supply. When the prime mover 220 is started, all devices receive and supply power. However, when the prime mover 220 is stopped, in order to save power, only some devices necessary for starting the prime mover 220 by remote operation receive and supply power. Devices that receive and supply power when the prime mover 220 is stopped are, for example, the sensor 104, the communication device 207, the antenna 208, the body controller 209, the information collection device 210, etc. Note that when the prime mover 220 is stopped, the devices may be operated in a power-saving mode and less power may be supplied compared to when the prime mover 220 is started.

[0049] In addition to the configuration shown in FIG. 3, a plurality of the sensor 104, the camera 105, the antenna 208, and the image transmission device 211 may be installed. The image transmission device 211 may be integrated with the camera 105. The body controller 209 and the information collection device 210 may be integrated. The communication device 207 may be integrated with the antenna 208.

[0050] Figure 4 is a schematic diagram showing the hardware configuration of the operating device 102 in this embodiment. The operating device 102 consists of a display 112, a communication device 307, an operator's seat 301, an operating stick 302, a control room controller 303, and an image receiving device 304. The operator's seat 301 is a chair in which the operator OP sits. The operating stick 302 and the control room controller 303 are provided on its side. The operating stick 302 is an input interface device for operating the work machine 101. The control room controller 303 is a computer that implements the operation unit 110. The communication device 307 is a computer that implements the operating device communication unit 109B shown in Figure 1. The communication device 307 relays the internal network of the operating device 102 and the external network. Detailed hardware will be described later in Figure 5.

[0051] The control stick 302 and the control room controller 303 are connected by a wire. The control room controller 303 generates an operation signal in response to the input from the control stick 302 and transmits it to the work machine 101 via the communication device 307. This makes it possible to operate the work machine 101. The image receiving device 304 is a computer that realizes the image receiving unit 111 shown in Figure 1. Detailed hardware will be described later in Figure 5. The image receiving device 304 decodes the image received from the work machine 101 via the communication device 307 and transmits it to the display 112 for display. The image receiving device 304 and the display 112 are connected by a coaxial cable and a shielded cable, etc., and communicate using communication protocols such as SDI and HDMI (High-Definition Multimedia Interface®). This makes it possible for the operator OP to visually grasp the site environment and remotely operate the work machine 101.

[0052] In addition to the configuration shown in Figure 4, multiple image receiving devices 304 may be installed. Only one display 112 may be installed. Sounds related to the on-site environment may be output from the display 112 or a separately installed speaker in addition to images. The operating device 102 may include various levers, switches, pedals, touch panels, game controllers, etc., in addition to the operating rod 302, and the control room controller 303 may generate and transmit operating signals in response to their inputs. The control room controller 303 may access the management server 103 using input / output interface devices such as touch panels, displays, keyboards, and mice.

[0053] Figure 5 is a block diagram showing the hardware configuration of the management server 103 in this embodiment. The management server 103 is a computer for implementing the work machine management database 113, the operation device management database 114, the operator management database 115, the status management unit 116, and the management server communication unit 109C. In Figure 5, the management server communication unit 109C is implemented by a communication device 119. Furthermore, the term "server" here refers to a processing device that can send and receive information with a communication device via a communication line, etc., and does not refer to hardware forms such as so-called personal computers or embedded information devices.

[0054] The computer consists of a CPU (Central Processor Unit) 401, memory 402, auxiliary storage device 403, communication interface 404, and input / output interface 405. This configuration is the same for the communication device 207, vehicle controller 209, information acquisition device 210, image transmission device 211, control room controller 303, and image receiving device 304 described above.

[0055] The CPU 401 is an arithmetic unit that executes programs stored in the memory 402. The CPU 401 executes various programs, thereby realizing the functions of the management server 103, the work machine 101, and the operating device 102. In addition, some of the processing performed by the CPU 401 when executing programs may be executed by other arithmetic units (such as a GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, etc.).

[0056] Memory 402 includes a non-volatile memory element called ROM (Read Only Memory) and a volatile memory element called RAM (Random Access Memory). ROM stores immutable programs (such as BIOS: Basic Input Output System). RAM is a high-speed, volatile memory element, such as DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by the CPU 401 and data used during program execution.

[0057] The auxiliary storage device 403 is a high-capacity, non-volatile storage device such as a magnetic storage device (HDD: Hard Disk Drive) or flash memory (SSD: Solid State Drive). The auxiliary storage device 403 also stores data used by the CPU 401 when executing programs, and the programs executed by the CPU 401. In other words, programs are read from the auxiliary storage device 403, loaded into memory 402, and executed by the CPU 401, thereby realizing the functions of the management server 103, the work machine 101, and the operating device 102.

[0058] The communication interface 404 is a network interface device that controls communication with other devices according to a predetermined protocol. The input / output interface 405 is an interface device for connecting devices that perform input and output to the computer. Sensors 104, hydraulic and electric auxiliary equipment, a keyboard, a mouse, and a display 112 are connected to the input / output interface 405, and by reading and writing specific addresses in the memory 402 using the CPU 401, it is possible to read input signals and output video, etc.

[0059] Figures 6A and 6B are diagrams representing the work machine management database 113 in this embodiment. The work machine management table 1101 shown in Figure 6A is a table that manages information about the work machine 101. As shown in Figure 6A, the work machine management table 1101 consists of a work machine ID, static work machine information (work machine name, work machine type, work machine location), and dynamic work machine information (work machine status).

[0060] The work machine ID is the identifier for work machine 101. The work machine ID is used as an index when the operator OP sends a state transition request and when the management server 103 refers to information about work machine 101 during state transition processing.

[0061] Static information about the work machine is information that the state management unit 116 of the management server 103 reads only. Static information about the work machine includes the work machine name, work machine type, and work machine location. The work machine name is the name of the work machine 101. The work machine type is information about the type of work machine 101. The work machine location is information about the location of the work machine 101. Static information about the work machine is basically used to facilitate the identification of the work machine 101, but it may also be used in the determination of whether a state transition is possible in the state transition process, such as restricting the operation devices 102 that can be connected according to the work machine location and work machine type.

[0062] The work machine dynamic information is information that the status management unit 116 of the management server 103 reads and writes. The work machine dynamic information includes the work machine status. The work machine status includes information regarding when the work machine 101 is operating and when it is stopped, and information regarding when communication between the work machine 101 and the operating device 102 via the network NW is connected and disconnected.

[0063] The working machine status recorded in the working machine management database 113 includes an "in operation" status, which indicates that the prime mover 220 of working machine 101 is running, working machine 101 is in operation, and communication between working machine 101 and the control device 102 via the network NW is established. In the example in Figure 6A, working machines 101 with working machine IDs C2 and C3 are in the "in operation" status. Note that "working machine 101 is in operation" means that the state of working machine 101 has changed as viewed from outside the working machine 101. In other words, "working machine 101 is in operation" means that any of the crawler 201, upper rotating body 202, boom 203, arm 204, bucket 205, and stairs 230 of working machine 101 are operating.

[0064] The working machine status recorded in the working machine management database 113 includes a paused state, which indicates that the prime mover 220 of working machine 101 is operating, working machine 101 is stopped, and communication between working machine 101 and the operating device 102 via the network NW is disconnected. In the example in Figure 6A, working machine 101 with working machine ID C1 is in the paused state. Note that working machine 101 being in operation means that the state of working machine 101 has not changed as viewed from outside working machine 101. In other words, working machine 101 being stopped means that the crawler 201, upper rotating body 202, boom 203, arm 204, bucket 205, and stairs 230 of working machine 101 are not operating.

[0065] The machine status recorded in the machine management database 113 includes a stopped state, which indicates that the prime mover 220 of machine 101 is stopped, machine 101 is stopped, and communication between machine 101 and the operating device 102 via the network NW is disconnected. In the example in Figure 6A, machine 101 with machine ID C4 is in the stopped state.

[0066] The work machine status recorded in the work machine management database 113 includes a manned operation status, which indicates that an operator OP is on board work machine 101 and operating work machine 101, and that communication between work machine 101 and the control device 102 via the network NW is currently disconnected. In the example in Figure 6A, work machines 101 with work machine IDs C5 and C6 are in the manned operation status.

[0067] The working machine status recorded in the working machine management database 113 includes information on whether the working machine 101 is stopped with its bucket 205 in contact with the ground G on a level surface G. As will be described later, the state in which the working machine 101 is stopped with its bucket 205 in contact with the ground G on a level surface G is called the "parked position". In the example in Figure 6A, working machines 101 with working machine IDs C1, C3, C4, and C6 are in the parked position.

[0068] The work machine status recorded in the work machine management database 113 includes information regarding the received radio wave strength received by the work machine 101 via communication from the operating device 102 over the network NW. In the example in Figure 6A, the received radio wave strength of work machine 101 with work machine ID C2 is -50 dBm, and the received radio wave strength of work machine 101 with work machine ID C3 is -48 dBm.

[0069] The work machine status recorded in the work machine management database 113 includes information regarding the communication delay of communication between the work machine 101 and the operating device 102 via the network NW. In the example in Figure 6A, the communication delay of work machine 101 with work machine ID C2 is 43 ms, and the communication delay of work machine 101 with work machine ID C3 is 39 ms.

[0070] The work machine status recorded in the work machine management database 113 includes information on whether or not work machine 101 is located in a pause area, which will be described later. In the example in Figure 6A, work machine 101 with work machine ID C1 is located in a pause area. The work machine status recorded in the work machine management database 113 includes information on whether or not work machine 101 is located in a stop area, which will be described later. In the example in Figure 6A, work machines 101 with work machine IDs C4 and C6 are located in a pause area.

[0071] The work machine status recorded in the work machine management database 113 includes information on whether or not the stairs 230 are stored. In the example in Figure 6A, the stairs 230 are stored for work machines 101 with work machine IDs C1 to C6. The work machine status recorded in the work machine management database 113 also includes information on the combination of work machine 101 and operating device 102 that are connected via network NW. In the example in Figure 6A, work machine 101 with work machine ID C2 is connected to operating device 102 with operating device ID R1, which will be described later. Also in the example in Figure 6A, work machine 101 with work machine ID C3 is connected to operating device 102 with operating device ID R3.

[0072] The work machine status recorded in the work machine management database 113 includes state transition conditions added for each work machine 101. In the example in Figure 6A, work machine 101 with work machine ID C1 is subject to additional state transition conditions: the N value of the rest area must exceed 20, and the N value of the stop area must exceed 30. In addition to the N values ​​of the rest area and stop area, the state transition conditions added for each work machine 101 may also include the speed, load capacity, and fuel amount for each work machine 101.

[0073] The work machine configuration device management table 1102 shown in Figure 6B is a table that manages information about the devices that make up the work machine 101. As shown in Figure 6B, the work machine configuration device management table 1102 consists of a device ID, a device name, and device control information (device type, device software version, related work machine ID, device IP address). The device ID is an identifier for the device. The device name is a name for the device to facilitate identification of the device. The device control information is information used to execute state transition processing.

[0074] The device control information includes the device type, device software version, associated work machine ID, and device IP address. The device type is information about the type of device. The device software version is information such as the version number indicating the version of the software that constitutes the device. The associated work machine ID is the work machine ID of the work machine 101 equipped with the device. The device IP address is access information for the device.

[0075] In the state transition process, the state management unit 116 of the management server 103 identifies the devices that constitute the work machine 101 using the related work machine ID. It identifies the devices that implement each function, such as the work machine information collection unit 107, using the device type. It accesses various devices using the device IP address and performs information collection from the work machine information collection unit 107 and state change processing in a manner appropriate to the device software version.

[0076] Figures 7A and 7B show the table configuration of the control device management database 114 in this embodiment. The control device management table 1201 shown in Figure 7A is a table that manages information about the control device 102. As shown in Figure 7A, the control device management table 1201 consists of a control device ID, static control device information (control device name, control device type, control device location), and dynamic control device information (control device status, operator ID in use). The control device ID is an identifier for the control device 102. It is used as an index when an operator sends a control device assignment request and when the management server 103 refers to information about the control device 102 during state transition processing.

[0077] Static information about the operating device is information that the state management unit 116 of the management server 103 reads only. Static information about the operating device includes the operating device name, operating device type, and operating device location. The operating device name is the name of the operating device 102. The operating device type is information about the type of operating device 102. The operating device location is information about the location of the operating device 102. Static information about the operating device is basically used to facilitate the identification of the operating device 102, but it may also be used in the determination of whether a state transition is possible in the state transition process, such as restricting the work machine 101 that can be connected according to the operating device location and operating device type.

[0078] The operating device dynamic information is information that the status management unit 116 of the management server 103 reads and writes. The operating device dynamic information includes the operating device status and the operator ID in use. The operating device status is information regarding the usage status of the operating device 102, such as whether it is in use or not. The operator ID in use is the identifier of the operator OP who is using the operating device 102. The operating device status is used to confirm and record the current operating device status in the operating device status transition process. The operator ID in use is used to confirm and record the operator OP who is using the operating device 102. It is also used to refer to information about the operator OP in status transition feasibility determination, work machine connection process, work machine cutting process, etc.

[0079] The control device configuration management table 1202 shown in Figure 7B is a table that manages information about the devices that make up the control device 102. As shown in Figure 7B, the control device configuration management table 1202 consists of a device ID, a device name, and device control information (device type, device software version, associated control device ID, and device IP address). The device ID is an identifier for the device. The device name is a name for the device to facilitate identification of the device.

[0080] Device control information is information used to execute state transition processing. Device control information includes device type, device software version, associated operating device ID, and device IP address. Device type is information about the type of device. Device software version is information such as the version number indicating the version of the software that constitutes the device. Associated operating device ID is the operating device ID of the operating device 102 that houses the device. Device IP address is access information for the device. In state transition processing, the state management unit 116 of the management server 103 identifies the devices that constitute the work machine 101 using the associated operating device ID. It identifies the devices that realize each function such as the operating unit 110 using the device type. It accesses various devices using the device IP address and performs state change processing in a manner corresponding to the device software version.

[0081] Figure 8 is a diagram showing the table structure of the operator management database 115 in this embodiment. As shown in Figure 8, the operator management table 1301 is a table that manages information about operators OP. The operator management table 1301 consists of operator ID, operator name, operator operation proficiency level, operator authority, and authentication password.

[0082] The Operator ID is an identifier for the Operator OP. It is used as an index when the Operator OP sends a request for assignment of an operating device and when the management server 103 refers to information about the Operator OP during state transition processing. The Operator Name is the name of the Operator OP to facilitate identification of the Operator OP. The Operator Proficiency level is the degree of the Operator OP's proficiency in remote operation. The Operator Proficiency level is used to restrict the operating devices 102 that can be connected in the state transition feasibility determination during state transition processing, to change the conditions for determining whether the state transition conditions are met in the state transition condition confirmation processing, and to change the setting items of the image transmission unit 108 during the work machine connection processing.

[0083] Operator privileges refer to information about the scope of operations that an operator (OP) can perform in a remote control system. They are used to restrict the patterns that are permitted for state transitions in the state transition approval / failure determination process. The authentication password is information used for operator authentication.

[0084] In this embodiment, the management server 103 includes an operator management database 115 that records state transition conditions added for each operator OP. For example, in the example in Figure 8, operators OP with operator IDs P2 and P3 are subject to additional state transition conditions: they cannot transition to the stopped state, the communication delay must be less than 70 ms, and the received signal strength must be greater than -60 dBm.

[0085] The details of the state transition process (S200) of the work machine 101 in this embodiment will be explained below with reference to Figures 9 and 10. The state transition process (S200) of the work machine 101 is a process that transitions the state of a work machine 101 from one state to another.

[0086] First, the working machine status will be explained. The working machine status refers to the state of use of the working machine 101, which is managed to prevent inappropriate remote operation such as simultaneous operation by multiple operators. Working machine statuses include the working state M101, which indicates that the working machine 101 is connected to the operating device 102, and the idle state M102, which indicates that the prime mover 220 of the working machine 101 is running and is not connected to any operating device 102. For example, by controlling the system so that the operating device 102 is not connected to the working machine 101 in the working state M101, it is possible to prevent simultaneous operation by multiple operators. Details of the working machine status will be explained later in Figure 9.

[0087] Next, with reference to Figure 10, an overview of the procedure for the state transition processing (S200) of the work machine 101 will be described. The state transition processing (S200) of the work machine 101 is executed when a state transition request is transmitted. Upon receiving the state transition request, the state management unit 116 performs state transition condition confirmation processing (S205) according to the state transition pattern, and state transition processing (S207) for the work machine 101 and the operating device 102, in accordance with a pre-programmed state transition model.

[0088] The state transition condition confirmation process (S205) is a process to confirm the stability of the work machine 101. For example, it confirms whether the work machine 101 is in an operable position, whether the communication delay between the work machine 101 and the operating device 102 is sufficiently small, etc.

[0089] The state transition process (S207) is a modification process for the work machine 101 and the operating device 102 necessary for the transition of the work machine state. For example, this includes changing the image transmission destination of the image transmission unit 108 in the connection process between the work machine 101 and the operating device 102 (work machine connection process: S101). The details of the state transition model of the work machine 101 are shown in Figure 9, and the detailed procedure of the state transition process (S200) is explained in Figure 10.

[0090] Figure 9 is a diagram showing the machine state transition model for the state management unit 116 of Embodiment 1 to perform state transition processing (S200). The circled area represents the machine state, and the arrows indicate the direction in which state transitions are possible. Near each arrow, the state transition conditions (in small font) used in the state transition condition confirmation process (S205) and the state transition processing (S207) (in bold font) for the machine 101 and the operating device 102 are indicated. In this embodiment, the machine state can take on four states: working state M101, idle state M102, stopped state M103, and manned operation state M104. Each state will be described below.

[0091] As described above, the working state M101 is a state in which the prime mover 220 of the working machine 101 is operating, the working machine 101 is in operation, and communication between the working machine 101 and the operating device 102 via the network NW is established. The working state M101 indicates that the working machine 101 is performing work via remote control. In the working state M101, the working machine 101 and the operating device 102 are connected, and video signals and operation signals are transmitted, respectively. It is possible to directly transition from the working state M101 to the idle state M102. On the other hand, it is not possible to directly transition from the working state M101 to the stopped state M103 or the manned operation state M104.

[0092] The conditions for transitioning from the working state M101 to the idle state M102 are: "the prime mover 220 of the work machine 101 is operating," "the work machine 101 is in a parked position," and "the work machine 101 is located within the idle area." The fact that the prime mover 220 is running can be determined, for example, by "the state of the prime mover 220 acquired by the machine control unit 106 being a value that represents the operating state."

[0093] The parked position is the posture for stably parking the work machine 101, and refers to the state where the bucket 205 is in contact with the ground on a level surface and stopped. For example, the parked position can be determined by methods such as "the Z-axis of the gyro sensor horizontally mounted on the upper rotating body 202 is 0 degrees", "the height of the bucket 205 calculated from the angles of the hydraulic actuators of the boom 203, arm 204, and bucket 205 acquired by the machine control unit 106 is 0 m", "the speed of the work machine 101 acquired by the machine control unit 106 is 0 km / h", and "the state of the Pilot-Shutoff Lever acquired by the machine control unit 106 is OFF". Note that "level" is not strictly limited to the inclination angle of the ground G being 0 degrees; for example, the ground G may have an inclination angle of about ±5 degrees.

[0094] The idle area refers to the range in which the work machine 101 can enter the idle state M102. Specifically, the idle area is a location near the mining site in a mine where the work machine 101 can be stably stopped while the prime mover 220 is still running. For example, it is possible to determine whether the work machine 101 is within the idle area by methods such as "the latitude and longitude obtained from the positioning sensor 601 attached to the upper rotating body 202 being within a specific range." The state transition process (S207) when transitioning to the idle state M102 is the work machine disconnection process (S102). In the work machine disconnection process (S102), the transmission of video signals from the work machine 101 to the operating device 102 is stopped, and the transmission of operation signals from the operating device 102 to the work machine 101 is stopped. Details of the work machine disconnection process (S102) will be described later in Figure 12.

[0095] As described above, the idle state M102 is a state in which the prime mover 220 of the work machine 101 is running, the work machine 101 is stopped, and communication between the work machine 101 and the operating device 102 via the network NW is disconnected. The idle state M102 indicates that the work machine 101 is not in use and that work can be started by remote control. In the idle state M102, the work machine 101 is not connected to any operating device 102, and the prime mover 220 is started. From the idle state M102, it is possible to directly transition to the working state M101, the stopped state M103, and the manned operation state M104.

[0096] The conditions for transitioning from the idle state M102 to the working state M101 are: "the communication delay between the work machine 101 and the operating device 102 is less than a specific communication delay threshold," "the received radio wave strength of the work machine 101 exceeds a specific radio wave strength threshold," and "the work machine 101 is located within the workable area."

[0097] Communication delay refers to the delay between the work machine 101 and the operating device 102 that are attempting to connect. For example, it is possible to determine whether the communication delay is below a specific communication delay threshold by checking, for instance, that "the communication delay measured by the work machine communication unit 109A is less than 100 ms." Received radio wave strength refers to the strength of the radio signal received by the antenna 208 of the work machine 101. For example, it is possible to determine whether the received radio wave strength exceeds a specific radio wave strength threshold by checking, for instance, that "the received radio wave strength measured by the work machine communication unit 109A exceeds -80 dBm."

[0098] The working area refers to the range in which the work machine 101 can be in the working state M101. For example, it is possible to determine whether the work machine 101 is within the working area by methods such as "the latitude and longitude obtained from the positioning sensor 601 attached to the upper rotating body 202 are within a specific range."

[0099] The state transition process (S207) when transitioning from the idle state M102 to the working state M101 is the work machine connection process (S101). In the work machine connection process (S101), the transmission of a video signal from the work machine 101 to the control device 102 is started, and the transmission of an operation signal from the control device 102 to the work machine 101 is started. Details of the work machine connection process (S101) will be described later in Figure 11.

[0100] As described above, the stopped state M103 is a state in which the prime mover 220 of the work machine 101 is stopped, the work machine 101 is stopped, and communication between the work machine 101 and the operating device 102 via the network NW is disconnected. The stopped state M103 indicates that the work machine 101 is not in use and that it is not possible to start work by remote operation. In the stopped state M103, the work machine 101 is not connected to any of the operating devices 102, and the prime mover 220 is stopped. From the stopped state M103, it is possible to directly transition to the idle state M102 and the manned operation state M104. On the other hand, it is not possible to directly transition from the stopped state M103 to the operation state M101.

[0101] The conditions for transitioning to the stopped state M103 are: "the prime mover of the work machine 101 is stopped," "the work machine 101 is in a parked position," "the stairs 230 of the work machine 101 are retracted," and "the work machine 101 is located within the stopping area."

[0102] The stairs 230 refer to the retractable stairs 230 used when boarding the work machine 101. For example, it is possible to determine whether the retractable stairs are stored by checking if the state of the stairs 230 obtained by the machine control unit 106 is a value that represents the stored state.

[0103] The stopping area refers to the range in which the work machine 101 can be in a stopped state M103. Specifically, the stopping area is a parking area in close proximity to the mining site in a mine. For example, it is possible to determine whether the work machine 101 is within the stopping area by checking whether the latitude and longitude obtained from the positioning sensor 601 attached to the upper rotating body 202 fall within a specific range.

[0104] The state transition process (S207) when transitioning from the idle state M102 to the stopped state M103 is the work machine stop process (S104). In the work machine stop process (S104), the prime mover of the work machine 101 and various devices are stopped. Details of the work machine stop process (S104) will be described later in Figure 14.

[0105] As described above, the manned operation state M104 is a state in which an operator OP is on board the work machine 101 and operating the work machine 101, and communication between the work machine 101 and the control device 102 via the network NW is disconnected. The manned operation state M104 indicates that the work machine 101 is being used for manned operation and that work cannot be started by remote operation. In the manned operation state M104, the work machine 101 is not connected to any of the control devices 102. From the manned operation state M104, it is possible to directly transition to the idle state M102 and the stopped state M103. On the other hand, it is not possible to directly transition from the manned operation state M104 to the working state M101.

[0106] There are no specific state transition conditions when transitioning from idle state M102 and stopped state M103 to manned operation state M104. Furthermore, there are no specific state transition processes (S207) performed by the management server 103 when transitioning to manned operation state M104. The operator OP, who is on board the work machine 101, performs the manned operation processes (S105, S107). Therefore, in the state transition process (S200) from idle state M102 and stopped state M103 to manned operation state M104, it is sufficient to simply change the work machine state managed by the state management unit 116 to manned operation state M104.

[0107] The conditions for transitioning from the manned operation state M104 to the idle state M102 are: "the prime mover 220 of the work machine 101 is operating," "the work machine 101 is in a parked position," and "the work machine 101 is located within the idle area." These conditions overlap with the content described above in the work machine cutting process (S102), which is the state transition process (S207) from the working state M101 to the idle state M102, so the explanation is omitted.

[0108] The state transition process (S207) when transitioning from the manned operation state M104 to the idle state M102 is the same as the machine cutting process (S102) in the state transition process (S207) when transitioning from the working state M101 to the idle state M102 described above, and is the machine idle process (S106).

[0109] The conditions for transitioning from the manned operation state M104 to the stopped state M103 are: "the prime mover 220 of the work machine 101 is stopped," "the work machine 101 is in a parked position," "the stairs 230 of the work machine 101 are retracted," and "the work machine 101 is located within the stopping area." These conditions overlap with the content described above in the work machine stopping process (S104), which is the state transition process (S207) from the idle state M102 to the stopped state M103, so the explanation is omitted.

[0110] The state transition process (S207) when transitioning from the manned operation state M104 to the stopped state M103 is the same as the machine stop process (S104) in the state transition process (S207) when transitioning from the idle state M102 to the stopped state M103 described above, and is the machine stop process (S108).

[0111] In addition to the configuration shown in Figure 9, the state transition model may also include working machine states such as "under maintenance" that indicate communication interruption or equipment malfunction. Multiple state transition condition confirmation processes (S205) and state transition processes (S207) may be managed and switched according to information regarding time, operator OP, working machine 101, and operating device 102.

[0112] To summarize the operations in the above state transition process, the state management unit 116 refers to the working state M101, idle state M102, stopped state M103, and manned operation state M104, which are included in the working machine state recorded in the working machine management database 113.

[0113] The state management unit 116 determines that the state transition conditions are met when the work machine 101 is stopped, and based on information regarding the connection and disconnection of communication, it performs state transition processing that involves the connection of communication between the work machine 101 and the operating device 102 via the network NW, and state transition processing that involves the disconnection of communication between the work machine 101 and the operating device 102 via the network NW.

[0114] On the other hand, the state management unit 116 determines that the state transition conditions are not met when the work machine 101 is in operation, and therefore does not perform either the state transition process that involves establishing communication between the work machine 101 and the operating device 102 via the network NW, or the state transition process that involves terminating communication between the work machine 101 and the operating device 102 via the network NW. In other words, in this embodiment, establishing and terminating communication between the work machine 101 and the operating device 102 is performed only when the work machine 101 is stopped.

[0115] The state management unit 116 performs state transition processing that includes a direct state transition between the working state M101 and the idle state M102, state transition processing that includes a direct state transition between the idle state M102 and the stopped state M103, state transition processing that includes a direct state transition between the idle state M102 and the manned operation state M104, and state transition processing that includes a direct state transition between the stopped state M103 and the manned operation state M104.

[0116] On the other hand, the state management unit 116 does not perform state transition processing that involves a direct state transition between the working state M101 and the stopped state M103, nor does it perform state transition processing that involves a direct state transition between the working state M101 and the manned operation state M104. In other words, in this embodiment, it is not possible to directly perform state transition processing to the working state M101 without first going through the idle state M102.

[0117] The state management unit 116 refers to information regarding the combination of work machine 101 and operating device 102 that are connected to the network NW, as recorded in the work machine management database 113. The state management unit 116 performs a state transition process that involves changing the combination of work machine 101 and operating device 102 that are connected to the network NW, through a direct state transition from the working state to the idle state, followed by a direct state transition from the idle state M102 to the working state M101. In other words, in this embodiment, the change in the combination of work machine 101 and operating device 102 that are connected to the network NW is only performed during the state transition process between the working state M101 and the idle state M102.

[0118] The status management unit 116 refers to information regarding whether the work machine 101, which is included in the work machine status recorded in the work machine management database 113, is stopped on a level ground G with its bucket 205 in contact with the ground G.

[0119] The state management unit 116 determines that the state transition conditions are met when the work machine 101 is stopped on a horizontal ground G with the bucket 205 in contact with the ground G, and performs state transition processing that involves a direct state transition from the working state M101 to the idle state M102, a state transition processing that involves a direct state transition from the stopped state M103 to the idle state M102, a state transition processing that involves a direct state transition from the manned operation state M104 to the idle state M102, and a state transition processing that involves a direct state transition from the manned operation state M104 to the stopped state M103.

[0120] On the other hand, the state management unit 116 determines that the state transition conditions are not met when the work machine 101 is not stopped with its bucket in contact with the ground G on a horizontal surface G, and does not perform state transition processing that involves a direct state transition from the working state M101 to the idle state M102, state transition processing that involves a direct state transition from the stopped state M103 to the idle state M102, state transition processing that involves a direct state transition from the manned operation state M104 to the idle state M102, and state transition processing that involves a direct state transition from the manned operation state M104 to the stopped state M103.

[0121] In other words, in this embodiment, unless the work machine 101 is in a parked position with its bucket in contact with the ground G on a horizontal surface, the state transition process involving the state transition to the idle state M102 and the stopped state M103 will not be performed.

[0122] The state management unit 116 refers to information regarding the received radio wave strength included in the work machine state recorded in the work machine management database 113. When the received radio wave strength exceeds the radio wave strength threshold, the state management unit 116 determines that the state transition condition is met and performs a state transition process that involves a direct state transition from the idle state M102 to the working state M101. On the other hand, when the received radio wave strength does not exceed the radio wave strength threshold, the state management unit 116 determines that the state transition condition is not met and does not perform a state transition process that involves a direct state transition from the idle state M102 to the working state M101. In other words, in this embodiment, when the received radio wave strength is low, the state transition process that involves a direct state transition from the idle state M102 to the working state M101 is not performed.

[0123] The state management unit 116 refers to information regarding communication delays included in the work machine status recorded in the work machine management database 113. When the communication delay is smaller than the communication delay threshold, the state management unit 116 determines that the state transition condition is met and performs a state transition process that involves a direct state transition from the idle state M102 to the working state M101. On the other hand, when the communication delay is not smaller than the communication delay threshold, the state management unit 116 determines that the state transition condition is not met and does not perform a state transition process that involves a direct state transition from the idle state M102 to the working state M101. In other words, in this embodiment, when the communication delay is large, the state transition process that involves a direct state transition from the idle state M102 to the working state M101 is not performed.

[0124] The state management unit 116 refers to information in the work machine management database 113 regarding whether or not the work machine 101, which is included in the work machine status, is located in the idle area. When the work machine 101 is located in the idle area, the state management unit 116 determines that the state transition conditions are met and performs state transition processing that involves a direct state transition from the working state M101 to the idle state M102, state transition processing that involves a direct state transition from the stopped state M103 to the idle state M102, and state transition processing that involves a direct state transition from the manned operation state M104 to the idle state M102.

[0125] On the other hand, the state management unit 116 determines that the state transition conditions are not met when the work machine is not located in the area where it can be paused, and does not perform state transition processing that involves a direct state transition from the working state to the paused state, state transition processing that involves a direct state transition from the stopped state to the paused state, and state transition processing that involves a direct state transition from the manned operation state to the paused state. In other words, in this embodiment, when the work machine 101 is not located in the area where it can be paused, state transition processing that involves a direct state transition to the paused state M102 is not performed.

[0126] The state management unit 116 refers to information recorded in the work machine management database 113 regarding whether or not the work machine 101, which is included in the work machine status, is located in the stoppable area. When the work machine 101 is located in the stoppable area, the state management unit 116 determines that the state transition conditions are met and performs state transition processing that involves a direct state transition from idle state M102 to stopped state M103 and state transition processing that involves a direct state transition from manned operation state M104 to stopped state M103.

[0127] On the other hand, when the work machine 101 is located in a stoppable area, the state management unit 116 determines that the state transition conditions are not met and does not perform state transition processing that involves a direct state transition from idle state M102 to stopped state M103, nor state transition processing that involves a direct state transition from manned operation state M104 to stopped state M103. In other words, in this embodiment, when the work machine 101 is not located in a stoppable area, state transition processing that involves a direct state transition to stopped state M103 is not performed.

[0128] The state management unit 116 refers to information in the work machine management database 113 regarding whether or not the stairs 230, which are included in the work machine state recorded, are stored. When the stairs 230 are stored, the state management unit 116 determines that the state transition conditions are met and performs state transition processing that involves a direct state transition from idle state M102 to stopped state M103 and state transition processing that involves a direct state transition from manned operation state M104 to stopped state M103.

[0129] On the other hand, the state management unit 116 determines that the state transition conditions are not met when the stairs 230 are not stored, and does not perform state transition processing that involves a direct state transition from the idle state M102 to the stopped state M103, nor does it perform state transition processing that involves a direct state transition from the manned operation state M104 to the stopped state M103. In other words, in this embodiment, when the stairs 230 are not stored, state transition processing that involves a direct state transition to the stopped state M103 is not performed.

[0130] In addition, in this embodiment, other information is also referenced in the state transition process described above. The state management unit 116 refers to the state transition conditions added for each operator OP recorded in the operator management database 115. The state management unit 116 performs the state transition process when it determines that the state transition conditions added for each operator OP are met, and does not perform the state transition process when it determines that the state transition conditions added for each operator OP are not met. In other words, in this embodiment, the state transition conditions are changed for each operator OP.

[0131] The state management unit 116 refers to the additional state transition conditions for the work machine 101 recorded in the work machine management database 113. The state management unit 116 performs a state transition process when it determines that the additional state transition conditions for each work machine 101 are met, and does not perform a state transition process when it determines that the additional state transition conditions for each work machine 101 are not met. In other words, in this embodiment, the state transition conditions are changed for each work machine 101.

[0132] Figure 10 is a flowchart showing the procedure for the state transition processing (S200) of the work machine 101 in this embodiment. The specific steps are described below. In S201, the state management unit 116 starts the state transition processing (S200). In S202, the state management unit 116 receives a state transition request transmitted from a mobile terminal 500 operated by an operator OP. The state transition request includes an identifier that uniquely identifies the target work machine 101 (work machine ID), the desired destination work machine state, and an identifier that uniquely identifies the operator OP who sent the state transition request (operator ID). Note that the transmission of the state transition request is not limited to the mobile terminal 500, but can also be transmitted from the operation unit 110, the machine control unit 106, another system, etc., and from any location.

[0133] In S203, the state management unit 116 determines whether a state transition is possible based on the state transition request. In this step, it checks whether a transition path exists and whether the operator's authority is appropriate. The transition path is the arrow in the state transition model shown in Figure 9. In the state transition model, state transitions that are not connected by arrows are determined to be impossible. For example, if the state management unit 116 receives a state transition request from the stopped state M103 of the work machine 101 to the working state M101, it determines that the state transition is impossible.

[0134] Operator authority is the authority information of the operator OP associated with the operator ID shown in Figure 8. Depending on the authority, the permission for state transitions is set. Specifically, "for operator OPs who manage the entire system, state transitions for all work machine states are permitted," and "for general operator OPs, only state transitions from idle state M102 to working state M101, and from working state M101 to idle state M102 are permitted," etc. This step makes it possible to prevent inappropriate system operation, such as work machine 101 being stopped by an operator OP without the necessary authority, which would reduce work efficiency.

[0135] In S204, if it is determined in S203 that a state transition is possible, the state management unit 116 acquires sensor information, machine control information, and operating device information from the work machine information collection unit 107 to be used in the state transition condition confirmation process S205. Note that only some of the information may be acquired based on the state of the work machine at the destination and source of the transition, the work machine ID, the operator ID, etc.

[0136] In S205, based on the state transition model shown in Figure 9, the state management unit 116 performs state transition condition confirmation processing using the information acquired in S204. The state transition processing confirmation processing, as described above with reference to Figure 9, includes checking the parking posture, checking the position of the work machine 101, checking the communication delay, checking the received radio wave strength, checking the state of the prime mover 220, and checking the stairs 230. Performing this step before the state change processing (S607) makes it possible to improve the stability of the work site before connecting and disconnecting the work machine 101 and the operating device 102. This step may be omitted if necessary.

[0137] In S206, the state management unit 116 checks whether it was determined in S205 that the state transition conditions were met. If S205 is omitted, it is assumed that the state transition conditions were met. In S207, the state management unit 116 performs state transition processing on the work machine 101 and the operating device 102 based on the state transition model. The state transition processing includes work machine connection processing (S101), work machine disconnection processing (S102), work machine startup processing (S103), and work machine stop processing (S104), as described above with reference to Figure 9. Details of each process will be described later in Figures 11 to 14. This step may be omitted if necessary.

[0138] In S208, the state management unit 116 determines whether the state transition process in S207 was successful. If step S207 is omitted, the state transition process is deemed to have been successful. In S209, the state management unit 116 transitions the state of the work machine in the work machine management DB 113. In S210, the state management unit 116 notifies the request sender of the result, including whether the state transition process (S200) was successful or not. If the state transition process failed, the state management unit 116 also notifies the cause of the failure. In S211, the state management unit 116 terminates the state transition process (S200).

[0139] Figures 11 to 14 will be used to explain the state transition process in more detail. Figure 11 is a sequence diagram showing the procedure for starting remote operation on the work machine 101 in the idle state M102 in this embodiment. The operator OP assigns the operating device 102 and connects the work machine 101 and the operating device 102 via the management server 103. Before connecting, the management server 103 checks the state transition conditions for the work machine 101, and if it is confirmed that the state transition conditions are met, it performs the work machine connection process (S101). The specific procedure will be explained below. Note that the contents described above in Figures 9 and 10 will not be explained again.

[0140] In S301, the operator OP accesses the management server 103 from the mobile terminal 500 and logs into the system. The operator OP sends an authentication request to the management server 103, including the operator ID and authentication password. The management server 103 authenticates the operator based on the authentication request and sends an authentication response to the operator, including the authentication result. Note that authentication may use methods such as TLS (Transport Layer Security).

[0141] In S302, the operator OP selects the control device 102 for which they request an assignment. The operator OP sends a control device assignment request to the management server 103, which includes their operator ID and an identifier (control device ID) that uniquely identifies the control device 102 for which they request an assignment. In S303, the management server 103 assigns the requested control device 102 to the operator OP if its status is "unused," and does not assign it if its status is "in use." If the management server 103 decides to assign the device, it changes the status of the control device it manages to "in use." After that, the management server 103 sends a control device assignment response to the operator OP, which includes information regarding the success or failure of the assignment.

[0142] In S304, the operator OP assigned to the operating device 102 connects the work machine 101 to the operating device 102 in order to start work. The operator OP sends a state transition request to the management server 103, which includes their operator ID, the work machine ID for which connection is requested, and the state of the work machine to which the request is made. Upon receiving the state transition request, the management server 103 performs a state transition feasibility determination (S203), acquires information on the work machine 101 (S204), and performs a state transition condition confirmation process (S205). If it is determined that a state transition is possible and the state transition conditions are met, the management server 103 performs a work machine connection process (S101).

[0143] In the machine connection process (S101), the video signal connection process is performed first (S401).

[0144] In the video signal connection process, the settings of the image transmission unit 108 of the work machine 101 are changed (S401α). The management server 103 sends a setting change request to the image transmission unit 108, including the setting items and setting content, and the work machine 101 changes the image transmission destination and whether or not image transmission is required. As a result, the image transmission unit 108 can transmit a video signal to the operating device 102.

[0145] In addition to the above, the resolution of the transmitted image, the coding rate of the transmitted image, the color information of the transmitted image, the refresh rate and error correction coding rate of the transmitted image may be further changed in the image transmission unit 108. Also, if it is not possible to set whether or not to transmit an image in the work machine 101, it is not necessary to set it. Furthermore, the work machine 101 may further change the image reception buffer amount of the image reception unit 111 of the operating device 102. The above setting changes may be made according to the operator OP, the work machine 101 and the operating device 102, and the setting values ​​may be changed accordingly.

[0146] If it is difficult to provide an API (Application Programming Interface) for receiving setting change requests in the image transmission unit 108, the settings of the work machine 101 may be changed by remote operation using SSH (Secure Shell) and a headless browser, etc.

[0147] In the work machine connection process (S101), the next step is the operation signal connection process (S402). In the operation signal connection process, the settings of the operation unit 110 of the operation device 102 are changed (S402α). The management server 103 sends a setting change request to the operation unit 110, including the setting items and setting content, and the operation device 102 changes the destination of the operation signal and whether or not to send the operation signal. As a result, the operation unit 110 can send an operation signal to the work machine 101.

[0148] If the operating device 102 cannot set whether or not to transmit an operation signal, it is not necessary to set it. If it is difficult to provide an API for receiving setting change requests in the operating unit 110, the settings of the operating unit 110 may be changed by remote operation using SSH and a headless browser, etc.

[0149] In other words, in this embodiment, when the state management unit 116 of the management server 103 performs state transition processing that involves communication between the work machine 101 and the operating device 102 via the network NW, it establishes a connection between the machine control unit 106 of the work machine 101 and the operating unit 110 of the operating device 102 after the connection between the image transmission unit 108 of the work machine 101 and the image receiving unit 111 of the operating device 102 has been established.

[0150] At least one of the work machine 101 and the operating device 102 is further equipped with a firewall to block unauthorized communications. When the state management unit 116 of the management server 103 performs state transition processing that involves establishing communication between the work machine 101 and the operating device 102 via the network NW, it establishes a connection between the image transmission unit 108 of the work machine 101 and the image receiving unit 111 of the operating device 102, and establishes a connection between the machine control unit 106 of the work machine 101 and the operation unit 110 of the operating device 102, and then configures the firewall to open communication between the work machine 101 and the operating device 102 via the network NW (S403).

[0151] The management server 103 sends a firewall configuration request to the work machine 101 and the operating device 102. After the work machine 101 and the operating device 102 configure the firewall, they return the firewall configuration request to the management server 103.

[0152] Furthermore, in the firewall settings, by configuring the firewall to block communication between the work machine 101 and the operating device 102 before the setting change, and then configuring it to open communication between the work machine 101 and the operating device 102 after the setting change, it is possible to connect the video signal and the operating signal simultaneously. In other words, regardless of the order of the video signal connection process (S401) and the operating signal connection process (S402), the stability of the connection between the work machine 101 and the operating device 102 can be improved.

[0153] After the firewall is configured, the management server 103 transitions the working machine status to the working state M101 (S209) and notifies the operator OP of the result of the status transition process. Since the connection between the working machine 101 and the operating device 102 is complete, the operator OP starts working (S305).

[0154] According to the procedure shown in Figure 11, in the state transition process (S200), the work machine connection process (S101) is performed after the state transition condition confirmation process (S205). This improves the stability before the connection between the work machine 101 and the operating device 102. Furthermore, in the work machine connection process (S101), the operation signal connection process (S402) is performed after the video signal connection process (S401). This prevents the operator from transmitting an operation signal before they can grasp the work environment via video, thereby improving the stability during the connection between the work machine 101 and the operating device 102.

[0155] Figure 12 is a sequence diagram showing the procedure for terminating remote operation on the work machine 101 in working state M101 in this embodiment. The operator OP disconnects communication between the work machine 101 and the control device 102 via the management server 103. Before disconnecting communication, the management server 103 performs a state transition condition confirmation process for the work machine 101, and if it is confirmed that the state transition condition is met, it performs the work machine disconnection process (S102). The specific procedure will be described below. Note that the contents described above in Figures 9 and 10 will not be explained again.

[0156] As shown in Figure 12, after operator OP authentication (S501), operator OP disconnects communication between the work machine 101 and the control device 102 in order to complete the work. Operator OP sends a state transition request to the management server 103, including their operator ID and the state of the work machine to which they wish to transition (S502). Upon receiving the state transition request, the management server 103 performs a state transition feasibility determination (S203), acquires information on the work machine 101 (S204), and performs a state transition condition confirmation process (S205). If the management server 103 determines that a state transition is possible and that the state transition conditions are met, the management server 103 performs a work machine disconnection process (S102).

[0157] When the state management unit 116 of the management server 103 performs a state transition process that involves disconnecting communication between the work machine 101 and the operating device 102 via the network NW, it sets up a firewall to block communication between the work machine 101 and the operating device 102 via the network NW before establishing a disconnection between the machine control unit 106 of the work machine 101 and the operating unit 110 of the operating device 102, and before establishing a disconnection between the image transmission unit 108 of the work machine 101 and the image receiving unit 111 of the operating device 102 (S601). The management server 103 sends a firewall setting request to the work machine 101 and the operating device 102. After setting up the firewall, the work machine 101 and the operating device 102 return the firewall setting request to the management server 103.

[0158] Furthermore, by configuring the firewall settings to block communication between the work machine 101 and the operating device 102, it is possible to disconnect both the video signal and the operating signal simultaneously. In other words, regardless of the order of the operating signal disconnection process (S602) and the video signal disconnection process (S603), it is possible to improve the stability during disconnection between the work machine 101 and the operating device 102.

[0159] In the machine cutting process (S102), the operation signal cutting process is performed first (S602). In the operation signal cutting process, the settings of the operation unit 110 of the operation device 102 are changed (S602α). The management server 103 sends a setting change request to the operation unit 110, including the setting item and setting content, and the operation device 102 changes whether or not operation signal transmission is required. As a result, the operation unit 110 can stop transmitting operation signals to the machine 101. When the state management unit 116 of the management server 103 performs a state transition process that involves disconnecting communication between the machine 101 and the operation device 102 via the network NW, it sets the destination address of the operation unit 110 of the operation device 102 to the address of the management server 103.

[0160] In addition to the above, if the operating device 102 cannot set whether or not to send an operation signal, the operating device 102 may set the destination of the operation signal to itself. If it is difficult to provide an API for receiving setting change requests in the operation unit 110, the operation unit 110 may change the settings by remote operation using SSH and a headless browser, etc.

[0161] In the machine cutting process (S102), the next step is to perform a video signal cutting process (S603). In the video signal cutting process, the settings of the image transmission unit 108 of the machine 101 are changed (S603α). The management server 103 sends a setting change request to the image transmission unit 108, including the setting items and setting content, and the machine 101 changes whether or not it needs to transmit images. As a result, the image transmission unit 108 can stop transmitting video signals to the operating device 102. When the state management unit 116 of the management server 103 performs a state transition process that involves disconnecting communication between the machine 101 and the operating device 102 via the network NW, it sets the destination address of the image transmission unit 108 of the machine 101 to the address of the management server 103.

[0162] In addition to the above, if the work machine 101 cannot set whether or not to transmit images, it may set itself as the destination for transmitting the image signal. If it is difficult to provide an API for receiving setting change requests in the image transmission unit 108, the image transmission unit 108 may change its settings by remote operation using SSH and a headless browser, etc.

[0163] In other words, in this embodiment, when performing a state transition process that involves disconnecting communication between the work machine 101 and the operating device 102 via the network NW, the disconnection between the machine control unit 106 of the work machine 101 and the operating unit 110 of the operating device 102 is established first, and then the disconnection between the image transmission unit 108 of the work machine 101 and the image receiving unit 111 of the operating device 102 is established.

[0164] After the video signal disconnection process is complete, the management server 103 transitions the work machine state to the idle state M102 (S209) and notifies the operator OP of the state transition process. The operator OP releases the operating device 102 that is currently in use (S503). The operator OP sends an operating device release request to the management server 103, which includes its operator ID. Based on the operator ID, the management server 103 identifies the operating device 102 that the operator OP is using and changes the operating device state to "unused" (S504). Subsequently, the management server 103 sends an operating device assignment response to the operator OP, which includes information regarding the success or failure of the release. Since the communication between the work machine 101 and the operating device 102 has been disconnected and the operating device 102 has been released, the operator OP finishes work.

[0165] According to the procedure shown in Figure 12, in the state transition process, the work machine disconnection process (S102) is performed after the state transition condition confirmation process (S205). This improves the stability of communication between the work machine 101 and the operating device 102 before disconnection. Furthermore, in the work machine disconnection process (S102), the video signal disconnection process (S603) is performed after the operation signal disconnection process (S602). This prevents the operator OP from transmitting an operation signal after they can no longer perceive the work environment via video, thereby improving the stability of communication between the work machine 101 and the operating device 102 during disconnection.

[0166] Figure 13 is a sequence diagram showing the procedure for starting up the work machine 101 in this embodiment. The operator OP starts up the work machine 101 via the management server 103. That is, this procedure makes it possible to establish a communication connection between the work machine 101 and the operating device 102. Before starting up, the management server 103 checks the state transition conditions related to the work machine 101, and if it is confirmed that the state transition conditions are met, it performs the work machine startup process (S103). The specific procedure will be described below. Note that the contents described above in Figures 9 and 10 will not be explained again.

[0167] After operator OP authentication (S701), operator OP starts up the work machine 101. The operator sends a state transition request to the management server 103, including their operator ID and the state of the work machine to which they wish to transition (S702). Upon receiving the state transition request, the management server 103 performs a state transition feasibility determination (S203), acquires information on the work machine 101 (S204), and performs a state transition condition confirmation process (S205). If it is determined that a state transition is possible and the state transition conditions are met, the work machine start-up process (S103) is performed.

[0168] In the machine startup process (S503), the prime mover start-up process is performed first (S801). In the prime mover start-up process, the machine control unit 106 of the machine 101 starts the prime mover 220. The management server 103 sends a prime mover start-up request to the machine control unit 106, and the machine control unit 106 starts the prime mover 220 (S801α). Subsequently, the machine control unit 106 sends a prime mover start-up response to the management server 103, including whether the prime mover 220 was started successfully. After the prime mover 220 has started up properly, power is supplied to all computers.

[0169] In the machine startup process (S103), the next step is to perform the computer startup process (S802). In the computer startup process, for example, Wake-on-LAN (WoL) is used to start each computer. WoL is a mechanism that allows computers to be started by sending a predetermined signal to a network interface device. The management server 103 sends a computer startup request to computers such as the communication device 207, the vehicle controller 209, and the information collection device 210, which are always running. Subsequently, the computers send a predetermined signal to the stopped computers. After that, the computers perform their own startup process (S802α) and send a computer startup response to the management server 103, including whether the computer startup was successful or not. This makes it possible to start any computer equipped on the machine 101.

[0170] In addition to the above, the power-saving mode of a computer operating in power-saving mode may be deactivated in response to a computer startup request. The computer may also be started when power is supplied after the prime mover 220 has started up normally. Instead of waiting for a computer startup response, the management server 103 may use a mechanism such as ICMP (Internet Control Message Protocol) echo to confirm the success or failure of the computer startup. After the computer startup process (S802α) is completed, the management server 103 transitions the machine status to the idle state M102 (S209) and notifies the operator OP of the result of the state transition process.

[0171] According to the procedure shown in Figure 13, in the state transition process (S200), the work machine startup process (S103) is performed after the state transition condition confirmation process (S205). This makes it possible to improve the stability of the work machine 101 before startup.

[0172] Figure 14 is a sequence diagram showing the procedure for stopping the idle work machine 101 in this embodiment. The operator stops the work machine 101 through the management server 103. That is, the communication connection between the work machine 101 and the operating device 102 is made impossible. Before stopping, the management server 103 checks the state transition conditions for the work machine 101, and if it is confirmed that the state transition conditions are met, it performs the work machine stop process (S104). The specific procedure will be described below. Note that the contents described above in Figures 9 and 10 will not be explained again.

[0173] After operator OP authentication (S901), operator OP starts up the work machine 101. Operator OP sends a state transition request to the management server 103, including their operator ID and the state of the work machine to which they wish to transition (S902). Upon receiving the state transition request, the management server 103 performs a state transition feasibility determination (S203), obtains information on the work machine 101 (S204), and performs a state transition condition confirmation process (S205). If it is determined that a state transition is possible and the state transition conditions are met, the management server performs a work machine stop process (S104).

[0174] In the machine shutdown process (S104), the computer shutdown process is performed first (S1001). In the computer shutdown process, the computer is shut down, for example, using a dedicated API provided by the computer. The management server 103 sends a computer shutdown request to computers such as the image transmission device 211 that do not need to operate in the shutdown state M103. A computer that receives a computer shutdown request performs its own shutdown process (S1001α) and sends a computer shutdown response to the management server 103, including whether the computer shutdown was successful or not. This makes it possible to shut down any computer of the machine 101.

[0175] In addition to the above, a computer that receives a computer shutdown request may switch to power-saving mode. Instead of waiting for a computer shutdown response, the management server 103 may use a mechanism such as ICMP echo to confirm the success or failure of the computer shutdown. If it is difficult for a computer to have an API to accept computer shutdown requests, the computer shutdown process may be performed remotely using SSH and a headless browser.

[0176] In the machine stop process (S104), the next step is to perform the prime mover stop process (S1002). In the prime mover stop process, the machine control unit 106 of the machine 101 stops the prime mover 220. The management server 103 sends a prime mover stop request to the machine control unit 106, and the machine control unit 106 stops the prime mover 220 (S1002α). Subsequently, the machine control unit 106 sends a prime mover stop response to the management server 103, including whether the prime mover stop was successful or not. After the prime mover 220 is stopped, the power supply to some computers and hydraulic electric auxiliary equipment is stopped. After the completion of the prime mover stop process (S1002), the management server 103 transitions the machine state to the stopped state M103 (S209) and notifies the operator OP of the result of the state transition process.

[0177] According to the procedure shown in Figure 14, in the state transition process (S200), the work machine stop process (S104) is performed after the state transition condition confirmation process (S205). This makes it possible to improve the stability of the work machine 101 before it is stopped.

[0178] An N-to-M remote control system is a system that features an interface that allows for setting changes and reloading without restarting the work machine and control device, and can switch connections with a simple mechanism in response to requests from the operator. In an N-to-M remote control system, efficient remote control requires switching connections between the work machine and the control device while considering the status of the operator and the work machine. If the status of the operator and the work machine is not sufficiently considered, the efficiency of the work site will decrease due to issues such as "switching connections to a work machine that is in operation" or "an operator who is not working smoothly continuing to work."

[0179] To address the above challenges, one possible approach is to detect instability in the operation of the work machine and the operator, and switch operators when instability is detected. Based on information from sensors and input devices such as buttons, instability such as "the work machine is not operating smoothly" or "the operator is unwell" can be detected. If instability is detected, the work will be handed over to a standby operator.

[0180] However, while such technology can improve the stability of a work machine during operation through remote control, it is difficult to improve the stability of communication between the work machine and the control device during connection and disconnection. For example, regarding the stability of communication during connection after a disconnection, there is the challenge of taking over a work machine in an unstable state. If the operator before the handover stops the work machine in an unstable state and disconnects the communication, the operator who reconnects the communication and takes over the work may not be able to determine that the situation is unstable.

[0181] Furthermore, regarding the stability of communication after a disconnection, there is a problem of erroneous operation due to delays in the connection of the video signal. In remote control systems, both video signals and control signals need to be transmitted, but in the connection process of communication between the work machine and the control device, the connection process of the video signal and the control signal is not necessarily completed simultaneously. Therefore, for example, if the connection of the video signal is delayed, a situation may occur where the video signal is not connected but the control signal is connected. That is, nothing is displayed on the display, but the operator is in a position to transmit control signals.

[0182] In this situation, the operator may rely on the display to determine the connection status between the work machine and the control device, potentially leading to incorrect operation. Specifically, the operator might assume that both the video signal and the control signal are disconnected, and then input commands into the control device to check its feel, potentially causing unintended operation.

[0183] On the other hand, in this embodiment, the connection and disconnection of communication between the work machine 101 and the operating device 102 is performed only when the work machine 101 is stopped. Therefore, since the connection and disconnection of communication are performed only when the work machine 101 is stopped and in a stable state, the state of the work machine 101 can be made more stable when switching between connecting and disconnecting communication between the work machine 101 and the operating device 102.

[0184] Furthermore, in this embodiment, it is not possible to directly transition to the working state M101 without first going through the idle state M102. The idle state M102 is when the work machine 101 is stopped and the prime mover 220 of the work machine 101 is running. Therefore, when switching from disconnection to connection of communication between the work machine and the operating device, the work machine 101 can be stabilized and the transition process to the working state M101 can be performed quickly.

[0185] Furthermore, in this embodiment, changes in the combination of the work machine 101 and the operating device 102 while communication is connected are only performed during the state transition process between the working state M101 and the idle state M102. As described above, the state transition process between the working state M101 and the idle state M102 is performed quickly and while the work machine 101 is in a stable state. Therefore, changes in the combination of the work machine 101 and the operating device 102 while communication is connected can also be performed quickly and while the work machine 101 is in a stable state. This improves work efficiency.

[0186] Furthermore, in this embodiment, the state transition process involving the transition to the idle state M102 and the stopped state M103 is not performed unless the work machine 101 is in a parked position with its bucket in contact with the ground G on a horizontal surface G. Therefore, the state transition process involving the transition to the idle state M102 and the stopped state M103 can be performed when the work machine 101 is in a stable parked position, thereby improving the stability of the work machine 101.

[0187] Furthermore, in this embodiment, when the received radio wave strength is low, a state transition process involving a direct state transition from the idle state M102 to the working state M101 is not performed. Therefore, when the received radio wave strength is low and the communication condition is poor, it is possible to prevent the working machine 101 from undergoing a state transition process to the working state M101.

[0188] Furthermore, in this embodiment, when the communication delay is large, the state transition process involving a direct state transition from the idle state M102 to the working state M101 is not performed. Therefore, when the communication delay is large and the communication condition is poor, it is possible to prevent the working machine 101 from undergoing the state transition process to the working state M101.

[0189] Furthermore, in this embodiment, when the work machine 101 is not located in the area where it can be paused, the state transition process that involves a direct state transition to the paused state M102 is not performed. Therefore, when the work machine 101 is located in a place unsuitable for transitioning to the paused state M102, it is possible to prevent the state transition process to the paused state M102 from being performed on the work machine 101.

[0190] Furthermore, in this embodiment, when the work machine 101 is not located in the stopping area, the state transition process that involves a direct state transition to the stopped state M103 is not performed. Therefore, when the work machine 101 is located in a place unsuitable for transitioning to the stopped state M103, it is possible to prevent the state transition process to the stopped state M103 from being performed on the work machine 101.

[0191] Furthermore, in this embodiment, when the stairs 230 are not stored, no state transition processing involving a direct state transition to the stopped state M103 is performed. Therefore, it is possible to prevent the state transition processing to the stopped state M103 from being performed on the work machine 101 when the stairs 230 are not stored and the work machine 101 is in an inappropriate state for transitioning to the stopped state M103.

[0192] Furthermore, in this embodiment, the state transition conditions are changed for each operator (OP). This allows the state transition conditions to be changed appropriately according to the operator's authority, experience, and skills, thereby enabling stable state transition processing.

[0193] Furthermore, in this embodiment, the state transition conditions are changed for each work machine 101. This allows the state transition conditions to be changed appropriately according to the performance, weight, and load capacity of the work machine 101, enabling stable state transition processing.

[0194] Furthermore, in this embodiment, the sensor 104 of the work machine 101 includes at least one of a positioning sensor 601, an acceleration sensor 602, a gyro sensor 603, a contact sensor 604, a three-dimensional sensor 605, a distance sensor 606, and a temperature sensor 607, so that various information about the work machine 101 can be obtained.

[0195] Furthermore, in this embodiment, when performing a state transition process that involves communication connection, the connection between the image transmission unit 108 of the work machine 101 and the image receiving unit 111 of the operating device 102 is established first, followed by the connection between the machine control unit 106 of the work machine 101 and the operating unit 110 of the operating device 102. When performing a state transition process that involves communication disconnection, the disconnection between the machine control unit 106 of the work machine 101 and the operating unit 110 of the operating device 102 is established first, followed by the disconnection between the image transmission unit 108 of the work machine 101 and the image receiving unit 111 of the operating device 102. As a result, the operator OP is always given image information when switching between a state where the work machine 101 can be operated and a state where the work machine 101 cannot be operated. Therefore, the remote operation of the work machine 101 by the operator OP can be made more stable.

[0196] Furthermore, in this embodiment, when performing a state transition process that involves disconnecting communication, the destination addresses of the operation unit 110 of the operating device 102 and the image transmission unit 108 of the work machine 101 are set to the address of the management server 103. This prevents inappropriate operations by the operator OP when performing a state transition process that involves disconnecting communication, and ensures the security of information.

[0197] Furthermore, in this embodiment, the firewall is configured to open communication after the communication connection is completed, and to block communication before the communication disconnection is completed, thereby ensuring the security of information.

[0198] Although several embodiments of the present invention have been described above, the invention is not limited to the embodiments described above and can be realized in various configurations without departing from the spirit of the invention. For example, configurations that arbitrarily combine or omit the configurations of the above embodiments can easily be conceivable. These variations are included in the scope of the invention described in the claims and its equivalents.

[0199] 100...Remote control system 101...Working machine 102...Operating device 103...Management server 104...Sensor 105...Camera 106...Machine control unit 107...Working machine information collection unit 108...Image transmission unit 109A...Working machine communication unit 109B...Operating device communication unit 109C...Management server communication unit 110...Operating unit 111...Image receiving unit 112...Display 113...Working machine management database 114...Operating device management database 115...Operator management database 116...Status management unit 119...Communication device 201...Crawler 202...Upper rotating body 203...Boom 204...Arm 205...Bucket 206...Manned operation room 207...Communication device 208...Antenna 209...Vehicle controller 210...Information collection device 211...Image transmission device 220...Motor 230...Stairs 301...Operator's seat 302...Operating lever 303...Control room controller 304...Image receiving device 307...Communication device 401...CPU 402...Memory 403...Auxiliary storage device 404...Communication interface 405...Input / output interface 500...Mobile terminal 601...Positioning sensor 602...Accelerometer 603...Gyroscope sensor 604...Contact sensor 605...Three-dimensional sensor 606...Distance sensor 1101...Work machine management table 1102...Work machine component device management table 1201...Operating device management table 1202...Operating device component device management table 1301...Operator management table M101...Working state M102...Stopped state M103...Stopped state M104...Manned operation state NW...Network OP...Operator G...Ground

Claims

1. A remote control system comprising: a work machine remotely controlled by communication over a network; an operating device for an operator to remotely control the work machine via the communication over the network; and a management server for managing the connection and disconnection of the communication between the work machine and the operating device over the network, wherein the work machine comprises: a work machine communication unit connected to the network; a sensor for obtaining information about the work machine; a camera for photographing the area around the work machine; a machine control unit that controls the operation of the work machine in response to operation commands received from the operating device via the work machine communication unit; an image transmission unit that processes and encodes images acquired from the camera and transmits them to the operating device via the work machine communication unit; and a work machine information collection unit that transmits information collected from the sensor and the machine control unit to the management server via the work machine communication unit, wherein the operating device comprises: an operating device communication unit connected to the network; an image receiving unit that decodes the images received from the work machine via the operating device communication unit; a display that displays the images decoded by the image receiving unit; and an operating unit that transmits the operation commands to the work machine, The management server comprises: a management server communication unit connected to the network; a work machine management database that records the work machine state relating to the state of the work machine based on information received from the work machine information collection unit of the work machine via the management server communication unit; and a state management unit that determines whether the work machine state satisfies predetermined state transition conditions, and when it determines that the state transition conditions are met, performs state transition processing of the work machine state in the work machine management database. The work machine state recorded in the work machine management database includes: information relating to the operation or stop of the work machine; and information relating to the connection or disconnection of the communication between the work machine and the operating device via the network. The state management unit refers to the work machine state recorded in the work machine management database.A remote control system characterized in that, when the work machine is stopped, it is determined that the state transition condition is met, and based on information regarding the connection or disconnection of the communication, the state transition process involving the connection of the communication between the work machine and the operating device via the network or the state transition process involving the disconnection of the communication is performed, and when the work machine is in operation, it is determined that the state transition condition is not met, and neither the state transition process involving the connection of the communication between the work machine and the operating device via the network nor the state transition process involving the disconnection of the communication between the work machine and the operating device via the network is performed.

2. The work machine comprises a prime mover, which is the power source for the operation of the work machine, and the work machine status recorded in the work machine management database includes: an operating state indicating that the prime mover of the work machine is operating, the work machine is in operation, and the communication between the work machine and the operating device via the network is established; a resting state indicating that the prime mover of the work machine is operating, the work machine is stopped, and the communication between the work machine and the operating device via the network is disconnected; a stopped state indicating that the prime mover of the work machine is stopped, the work machine is stopped, and the communication between the work machine and the operating device via the network is disconnected; and a manned operation state indicating that the operator is on board the work machine and operating the work machine, and the communication between the work machine and the operating device via the network is disconnected, and the status management unit is The remote control system according to claim 1, characterized in that it refers to the working state, the idle state, the stopped state, and the manned operation state included in the working machine state recorded in the working machine management database, and performs the state transition process including a direct state transition between the working state and the idle state, the state transition process including a direct state transition between the idle state and the stopped state, the state transition process including a direct state transition between the idle state and the manned operation state, and the state transition process including a direct state transition between the stopped state and the manned operation state, and does not perform the state transition process including a direct state transition between the working state and the stopped state and the state transition process including a direct state transition between the working state and the manned operation state.

3. The remote control system according to claim 2, comprising at least two of the aforementioned work machines and at least two of the aforementioned operating devices, wherein the work machine status recorded in the work machine management database includes information regarding combinations of the work machine and the operating device that are connected to the communication via the network, and the status management unit refers to the information regarding combinations of the work machine and the operating device that are connected to the communication via the network included in the work machine status recorded in the work machine management database, and performs the state transition processing which involves a change in the combination of the work machine and the operating device that are connected to the communication via the network, through a direct state transition from the working state to the idle state followed by a direct state transition from the idle state to the working state.

4. The work machine comprises a bucket for performing excavation work on the ground, and the work machine state recorded in the work machine management database includes information on whether the work machine is stopped with the bucket in contact with the ground on a level surface, and the state management unit refers to the information included in the work machine state recorded in the work machine management database regarding whether the work machine is stopped with the bucket in contact with the ground on a level surface, and determines that the state transition condition is met when the work machine is stopped with the bucket in contact with the ground on a level surface, and performs the state transition process which involves a direct state transition from the working state to the resting state, the state transition process which involves a direct state transition from the stopped state to the resting state, the state transition process which involves a direct state transition from the manned operation state to the resting state, and the state transition process which involves a direct state transition from the manned operation state to the stopped state. The remote control system according to claim 3, characterized in that when the work machine is not stopped with its bucket in contact with the ground on a level surface, it is determined that the state transition condition is not met, and the state transition process involving a direct state transition from the working state to the idle state, the state transition process involving a direct state transition from the stopped state to the idle state, the state transition process involving a direct state transition from the manned operation state to the idle state, and the state transition process involving a direct state transition from the manned operation state to the stopped state are not performed.

5. The remote control system according to claim 4, wherein the working machine status recorded in the working machine management database includes information regarding the received radio wave strength received by the working machine via the communication from the operating device through the network, and the status management unit refers to the information regarding the received radio wave strength included in the working machine status recorded in the working machine management database, determines that the state transition condition is met when the received radio wave strength exceeds a radio wave strength threshold, and performs the state transition process which involves a direct state transition from the idle state to the working state, and determines that the state transition condition is not met when the received radio wave strength does not exceed the radio wave strength threshold, and does not perform the state transition process which involves a direct state transition from the idle state to the working state.

6. The remote control system according to claim 5, wherein the working machine state recorded in the working machine management database includes information regarding the communication delay of the communication between the working machine and the operating device via the network, the state management unit refers to the information regarding the communication delay included in the working machine state recorded in the working machine management database, determines that the state transition condition is met when the communication delay is less than a communication delay threshold, and performs the state transition process which involves a direct state transition from the idle state to the working state, and determines that the state transition condition is not met when the communication delay is not less than a communication delay threshold, and does not perform the state transition process which involves a direct state transition from the idle state to the working state.

7. The remote control system according to 6, wherein the working machine status recorded in the working machine management database includes information on whether or not the working machine is located in a rest area, and the status management unit refers to the information on whether or not the working machine included in the working machine status recorded in the working machine management database is located in the rest area, and when the working machine is located in the rest area, it determines that the state transition condition is met and performs the state transition process involving a direct state transition from the working state to the rest state, the state transition process involving a direct state transition from the stopped state to the rest state, and the state transition process involving a direct state transition from the manned operation state to the rest state, and when the working machine is not located in the rest area, it determines that the state transition condition is not met and does not perform the state transition process involving a direct state transition from the working state to the rest state, the state transition process involving a direct state transition from the stopped state to the rest state, and the state transition process involving a direct state transition from the manned operation state to the rest state.

8. The remote control system according to claim 7, wherein the working machine status recorded in the working machine management database includes information on whether or not the working machine is located in a stoppable area, and the status management unit refers to the information on whether or not the working machine included in the working machine status recorded in the working machine management database is located in a stoppable area, determines that the state transition condition is met when the working machine is located in a stoppable area, and performs the state transition process involving a direct state transition from the idle state to the stopped state and the state transition process involving a direct state transition from the manned operation state to the stopped state, and determines that the state transition condition is not met when the working machine is not located in a stoppable area, and does not perform the state transition process involving a direct state transition from the idle state to the stopped state and the state transition process involving a direct state transition from the manned operation state to the stopped state.

9. The remote control system according to 8, wherein the work machine comprises a staircase that can be retracted and extended, the work machine state recorded in the work machine management database includes information on whether or not the staircase is retracted, the state management unit refers to the information on whether or not the staircase is retracted included in the work machine state recorded in the work machine management database, determines that the state transition condition is met when the staircase is retracted, and performs the state transition process involving a direct state transition from the idle state to the stopped state and the state transition process involving a direct state transition from the manned operation state to the stopped state, and determines that the state transition condition is not met when the staircase is not retracted, and does not perform the state transition process involving a direct state transition from the idle state to the stopped state and the state transition process involving a direct state transition from the manned operation state to the stopped state.

10. The remote control system according to claim 9, wherein the management server comprises an operator management database that records the state transition conditions added for each operator, and the state management unit refers to the state transition conditions added for each operator recorded in the operator management database, performs the state transition process when it determines that the state transition conditions added for each operator are satisfied, and does not perform the state transition process when it determines that the state transition conditions added for each operator are not satisfied.

11. The remote control system according to claim 10, wherein the work machine state recorded in the work machine management database includes the state transition conditions added for each work machine, the state management unit refers to the additional state transition conditions for the work machine recorded in the work machine management database, performs the state transition process when it determines that the state transition conditions added for each work machine are satisfied, and does not perform the state transition process when it determines that the state transition conditions added for each work machine are not satisfied.

12. The remote control system according to claim 11, characterized in that the sensors of the work machine include at least one of a positioning sensor, an acceleration sensor, a gyroscope, a contact sensor, a three-dimensional sensor, a distance sensor, and a temperature sensor.

13. A communication connection method in a remote control system according to claim 12, characterized in that the state management unit of the management server, when performing the state transition processing which involves connecting the communication between the work machine and the operating device via the network, establishes a connection between the machine control unit of the work machine and the operating device after the connection between the image transmission unit of the work machine and the image receiving unit of the operating device has been established, and when performing the state transition processing which involves disconnecting the communication between the work machine and the operating device via the network, establishes a disconnection between the image transmission unit of the work machine and the image receiving unit of the operating device after the disconnection between the machine control unit of the work machine and the operating device has been established.

14. The communication connection method according to 13, characterized in that when the state management unit of the management server performs the state transition processing which involves disconnecting the communication between the work machine and the operating device via the network, the destination addresses of the operating unit of the operating device and the image transmission unit of the work machine are set to the address of the management server.

15. The communication connection method according to 14, wherein at least one of the work machine and the operating device further comprises a firewall for blocking unauthorized communications, and the state management unit of the management server, when performing the state transition processing which involves establishing the communication between the work machine and the operating device via the network, establishes the connection between the image transmission unit of the work machine and the image receiving unit of the operating device, and establishes the connection between the machine control unit of the work machine and the operating unit of the operating device, and then sets up the firewall to open the communication between the work machine and the operating device via the network, when performing the state transition processing which involves disconnecting the communication between the work machine and the operating device via the network, establishes the disconnection between the machine control unit of the work machine and the operating unit of the operating device, and before establishing the disconnection between the image transmission unit of the work machine and the image receiving unit of the operating device, sets up the firewall to block the communication between the work machine and the operating device via the network.

Images