Remote operation system for construction machinery

Abstract

The present invention addresses the problem of improving safety. A remote operation system for a construction machine according to one embodiment of the present invention is provided with a construction machine, a remote operation room, and a control device, and the construction machine is provided with: an imaging device for acquiring an image of the surroundings; a detection device that detects additional information indicating information other than the video from among information indicating at least one of the situation of the construction machine and the situation around the construction machine; and a first communication device for transmitting the video and the additional information, the remote operation room comprising: a second communication device for transmitting the operation information from the operation device to the first communication device and receiving the video and the additional information; a display device for displaying an image; and an output device that outputs additional information, and a control device that controls transmission or reception of the additional information and necessary information necessary for operation of the construction machine between the first communication device and the second communication device, the necessary information including at least one of the video and the operation information. The control device performs different controls on the necessary information and the additional information in accordance with the communication status between the first communication device and the second communication device.

Description

Technical Field

[0001] This application claims priority based on Japanese Patent Application No. 2024-149990, filed on August 30, 2024. The entire contents of that Japanese application are incorporated herein by reference.

[0002] This invention relates to a remote operating system for construction machinery. Background Technology

[0003] Previously, a technology for remotely operating construction machinery has been proposed. In remote operation, the operator performs the operation while referring to an image displayed on a display device (see Patent Document 1).

[0004] Patent Document 1: Japanese Patent Application Publication No. 2023-102576

[0005] Communication between construction machinery and remote control devices may malfunction. Reference 1 describes how to send a stop command in the event of such communication failure.

[0006] Construction machinery and remote control devices exchange various types of information. This information includes both essential information for the machinery's operation and information relevant to the machinery's reference level. In other words, there is information that makes remote operation difficult if transmission or reception is impossible, and information that allows remote operation to continue even without transmission or reception. Therefore, when communication anomalies occur, controls corresponding to the information being transmitted or received are required. Summary of the Invention

[0007] In view of the above, safety can be improved by implementing different controls for each received or transmitted message based on the communication status between the construction machinery and the remote operating device.

[0008] One aspect of the present invention relates to a remote operating system for construction machinery comprising construction machinery, a remote control room, and a control device. The construction machinery includes: a camera device for acquiring images of the area surrounding the construction machinery; a detection device for detecting additional information other than the images from information indicating at least one of the conditions of the construction machinery and the conditions surrounding the construction machinery; and a first communication device for transmitting the images and the additional information. The remote control room includes: an operating device; a second communication device for transmitting operation information from the operating device to the first communication device and receiving the images and the additional information; a display device for displaying the images; and an output device for outputting the additional information. The control device controls the transmission or reception of necessary information and the additional information required for operating the construction machinery between the first and second communication devices. The necessary information includes at least one of the images and the operation information. The control device performs different controls on the necessary information and the additional information respectively based on the communication status between the first and second communication devices.

[0009] Invention Effects

[0010] According to one aspect of the present invention, security is enhanced by implementing different controls for each received or transmitted message based on the communication status. Attached Figure Description

[0011] Figure 1 This is a schematic diagram illustrating an example of the remote operating system according to the first embodiment.

[0012] Figure 2 This is a diagram illustrating a structural example of the drive control system of the excavator according to the first embodiment.

[0013] Figure 3 This is a functional block diagram illustrating a structural example of the remote operating system involved in the first embodiment.

[0014] Figure 4 This is a diagram illustrating an example of the configuration of a remote control room according to the first embodiment.

[0015] Figure 5 This is a conceptual diagram illustrating the information being sent and received between the construction machinery and the remote control room according to the first embodiment.

[0016] Figure 6 This is a diagram illustrating an example of volume output control in the output control unit according to the first embodiment.

[0017] Figure 7 This is a diagram showing an example of a screen displayed by the display control unit according to the first embodiment.

[0018] Figure 8 This is a flowchart showing the processing sequence of the remote controller according to the first embodiment, corresponding to the communication status of vibration information and tilt information.

[0019] Figure 9 This is a flowchart illustrating the processing sequence of the remote controller in the first embodiment corresponding to the communication status of the sound signal.

[0020] Figure 10 This is a flowchart illustrating the processing sequence corresponding to the communication status of the remote controller with necessary information according to the second embodiment.

[0021] In the diagram: 100-Construction machinery, 1-Lower traveling body, 2-Slewing mechanism, 3-Upper slewing body, 4-Boom, 5-Stick, 6-Bucket, S4-Machine body tilt sensor, S6-Camera device, S7-Vibration detection device, M1-Sound collection device, SP1-Sound output device, T1-Communication device, PS-Positioning device, 30-Controller, 301-Acquisition unit, 302-Transmitting control unit, 303-Receiving control unit, 304-Actuator drive unit, 31-Proportional valve, RC-Remote control room, T2-Communication device, D1E-Display device, R40-Remote controller, 401-Receiving control unit, 402-Display control unit, 403-Output control unit, 404-Decision unit, 405-Signal generation unit, 406-Transmitting control unit, R42-Operating device, R43-Operating sensor, R44-Vibration device, R45-Swing device. Detailed Implementation

[0022] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. The embodiments described below are illustrative and do not limit the invention. All features and combinations thereof in the embodiments of the present disclosure are not necessarily essential features of the invention. Furthermore, in the various drawings, the same or corresponding structures are sometimes labeled with the same or corresponding symbols, and repeated descriptions are omitted.

[0023] The construction machinery 100 disclosed herein is an excavator. However, the construction machinery 100 may also be machinery other than an excavator, such as a crane, asphalt roller, or forklift. Furthermore, in the illustrated example, the excavator 100 is an excavator equipped with a bucket 6 as an end attachment, but it may also be an application machine such as forestry machinery equipped with end attachments other than the bucket 6. Additionally, it may be a crawler crane equipped with a lower traveling body, an upper rotating body, and attachments provided on the upper rotating body.

[0024] (First Embodiment)

[0025] First, refer to Figure 1An overview of the remote operating system SYS according to the first embodiment will be described. Figure 1 This is a schematic diagram illustrating an example of the remote operating system SYS according to the first embodiment.

[0026] <Device constituting a remote operating system>

[0027] like Figure 1 As shown, the remote operating system SYS involved in the first embodiment includes construction machinery 100 and remote control room RC.

[0028] The construction machinery 100 and the remote control room RC are connected via the communication line NW to enable data transmission and reception.

[0029] The construction machinery 100 is capable of wireless communication. Furthermore, the construction machinery 100 is capable of transmitting and receiving data between itself and a device (e.g., a remote control room RC) connected to the communication line NW.

[0030] Construction machinery 100 is present at the work site where it operates. Furthermore, construction machinery 100 can transmit information related to the work site to a remote control room (RC). Thus, the remote control room (RC) can confirm the work site based on the information from construction machinery 100. Additionally, this embodiment does not limit the device for measuring the work site to construction machinery 100; other devices such as a fixed-point measuring device present at the work site, a drone flying over the work site, or a user-held camera device may also be used.

[0031] The remote operating system SYS can include one or more construction machines 100. Therefore, the remote operating system SYS can provide information related to the work site to the remote control room RC through one or more construction machines 100.

[0032] <Example of a remote control room structure>

[0033] The remote control room RC is equipped with a communication device T2, a remote controller R40, an operating device R42, an operating sensor R43, a display device D1E, a sound collection device M1E, a sound output device SP1E, a vibration device R44, and a swing device R45. Furthermore, the remote control room RC is equipped with an operating seat DS for the remote operator OP who remotely operates the construction machinery 100.

[0034] Communication device T2 (an example of the second communication device) is configured to control and be installed on the construction machinery 100 as a communication device T1 (see reference). Figure 2 Communication between them.

[0035] The remote controller (an example of a control device) R40 is an information processing device that performs various calculations. In this embodiment, the remote controller R40 is composed of a microcomputer including a CPU and memory. Furthermore, the various functions of the remote controller R40 are implemented by the CPU executing programs stored in the memory.

[0036] Display device D1E displays a screen based on information sent from construction machinery 100, enabling remote operator (OP) in remote control room (RC) to visually identify the surroundings of construction machinery 100. Display device D1E allows the operator to confirm the conditions of the work site, including the surroundings of construction machinery 100, even when in remote control room (RC).

[0037] The operating device R42 (an example of an operating unit) is equipped with an operation sensor R43 for detecting the operation content of the operating device R42. The operation sensor R43 may be, for example, a tilt sensor that detects the tilt angle of the operating lever or an angle sensor that detects the swing angle of the operating lever around its swing axis. The operation sensor R43 may also be composed of other sensors such as a pressure sensor, current sensor, voltage sensor, or distance sensor. The operation sensor R43 outputs information related to the detected operation content of the operating device R42 to the remote controller R40. The remote controller R40 generates an operation signal based on the received information and sends the generated operation signal to the construction machinery 100. The operation sensor R43 may also be configured to generate an operation signal. In this case, the operation sensor R43 may output the operation signal to the communication device T2 without going through the remote controller R40. Therefore, remote operation of the construction machinery 100 can be achieved from the remote control room RC.

[0038] The sound collection device M1E is a device for collecting sound generated within the remote control room (RC). In the example shown, the sound collection device M1E is an indoor microphone configured to capture the sound emitted by the remote operator (OP) within the remote control room (RC).

[0039] The sound output device SP1E is a device capable of outputting various sound information. The sound output device SP1E outputs sound based on information sent from the construction machinery 100, so that the remote operator OP located in the remote control room RC can hear the sounds emanating from the work site. The sound output device SP1E is configured, for example, to output sound from a sound collector M1 (see reference) installed on the outside of the construction machinery 100. Figure 2The captured sound. The sound output device SP1E can be a set-top device such as a speaker, or a wearable device such as earbuds or headphones. The speaker can be a mono speaker, a stereo speaker, or a surround sound speaker. Furthermore, the speaker can be an omnidirectional speaker or a directional speaker. The wearable device can have noise cancellation function, spatial audio function (stereo function), or bone conduction function.

[0040] The vibration device R44 causes the operating seat DS, where the remote operator OP sits, to vibrate with amplitude and frequency corresponding to the control from the remote controller R40. The vibration device R44 is configured, for example, to output to the vibration detection device S7 (see reference) installed on the construction machinery 100. Figure 2 The vibration detected. Furthermore, the object of vibration of the vibration device R44 according to this embodiment is not limited to the operating seat DS; for example, it could also be the operating device R42, etc.

[0041] The oscillation device R45, controlled by the remote controller R40, causes the operator's seat DS, in which the remote operator OP sits, to oscillate (tilt). The vibration device R44 is configured, for example, to reproduce the tilt of the machine body sensor S4 (see reference) mounted on the construction machinery 100 for the operator's seat DS. Figure 2 The tilt of the construction machinery 100 is detected. In this embodiment, an example is described where the swing device R45 causes the operating seat DS to swing (tilt) the construction machinery 100. However, in this embodiment, the method for enabling the remote operator OP to recognize the tilt of the construction machinery 100 is not limited to the method of swinging (tilting) the operating seat DS by the swing device R45. For example, the remote controller R40 may also display the current tilt angle of the construction machinery 100 on the display device D1E in a recognizable manner.

[0042] Construction machinery 100 includes: a lower traveling body 1; an upper slewing body 3, which is rotatably mounted on the lower traveling body 1 via a slewing mechanism 2; an accessory AT for performing various operations; and a driver's cab 10. The driver's cab 10 is also referred to as the driver's compartment or operator's cab. The front side of the construction machinery 100 (upper slewing body 3) corresponds to the side of the upper slewing body 3 when viewed from directly above along the axis of rotation of the construction machinery 100, where the accessory AT is installed. Furthermore, the left, right, and rear sides of the construction machinery 100 (upper slewing body 3) correspond to the left, right, and rear sides when viewed from the operator's seat inside the driver's cab 10, respectively.

[0043] The lower traveling body 1 includes, for example, a pair of tracks (not shown) on the left and right sides. Specifically, the tracks include a left track and a right track. The left track is powered by a left travel hydraulic motor 2ML (see reference). Figure 2Driven by the right track, the right track is powered by the right travel hydraulic motor 2MR (reference). Figure 2 The left travel hydraulic motor 2ML drives the left track, which is the driven part, and enables the left track to rotate. The right travel hydraulic motor 2MR drives the right track, which is the driven part, and enables the right track to rotate. Alternatively, the travel drive unit can also be an electric motor.

[0044] A boom 4 is rotatably mounted at the front center of the upper rotating body 3. A stick 5 is rotatably mounted at the end of the boom 4, and a bucket 6 is rotatably mounted at the end of the stick 5. In the example shown, the boom 4, stick 5, and bucket 6 constitute an example of an excavation attachment AT. The boom 4, stick 5, and bucket 6 are driven by the boom cylinder 7, stick cylinder 8, and bucket cylinder 9, respectively.

[0045] Bucket 6 is an example of a working tool (end attachment). Bucket 6 is used, for example, for excavation operations. Depending on the work being performed, other working tools may be installed at the end of the boom 5 instead of bucket 6. These other working tools may include, for example, large buckets, slope buckets, dredging buckets, and other types of buckets. Furthermore, these other working tools may be types of working tools other than buckets, such as mixers, crushers, grapples, or lifting magnets. The excavation attachment may also be equipped with a bucket tilting mechanism.

[0046] Alternatively, the construction machinery 100 can also electrically drive all driven parts, or parts thereof, including the lower traveling body 1, upper slewing body 3, boom 4, stick 5, and bucket 6. That is, the construction machinery 100 can also be a hybrid excavator or an electric excavator, etc., in which all driven parts or parts thereof are driven by electric actuators.

[0047] The construction machinery 100 described in this embodiment is configured to be remotely operated from outside the construction machinery 100. When the construction machinery 100 is remotely operated, the interior of the cab 10 can be unmanned.

[0048] Furthermore, the construction machinery 100 can also automatically operate the actuators regardless of the operator's actions. Thus, the construction machinery 100 achieves the function of automatically operating at least a portion of the driven parts, such as the lower traveling body 1, the upper slewing body 3, the boom 4, the stick 5, and the bucket 6, which is the so-called "equipment control function".

[0049] Figure 2 This is a diagram that roughly represents an example of the structure of construction machinery 100. Figure 2 In the diagram, the mechanical power transmission system, working oil pipeline, pilot line, and electrical control system are shown using double lines, thick solid lines, thick dashed lines, and dotted lines, respectively.

[0050] The drive system of the construction machinery 100 includes an engine 11, a regulator 13, a main pump 14, and a control valve unit 17. Furthermore, the hydraulic drive system of the construction machinery 100 includes hydraulic actuators such as a swing hydraulic motor 2A, a left travel hydraulic motor 2ML, a right travel hydraulic motor 2MR, a boom cylinder 7, a stick cylinder 8, and a bucket cylinder 9.

[0051] Engine 11 is one example of the power source for construction machinery 100, for example, it may be mounted at the rear of the upper rotating body 3. Alternatively, the power source for construction machinery 100 may be a combination of a battery or fuel cell power source and an electric motor. Specifically, under the direct or indirect control of controller 30, engine 11 drives the main pump 14 and pilot pump 15 to rotate at a constant, preset target speed. Engine 11 may be, for example, a diesel engine that uses diesel fuel. Alternatively, engine 11 may also be a gasoline engine or a hydrogen engine.

[0052] The regulator 13 controls the discharge volume of the main pump 14. For example, the regulator 13 adjusts the angle (deflection angle) of the swashplate of the main pump 14 according to the control command from the controller 30, thereby controlling the discharge volume of the main pump 14.

[0053] Similar to engine 11, main pump 14 is mounted, for example, at the rear of upper rotating body 3, and supplies working oil to control valve unit 17 via working oil lines. In the example shown, main pump 14 is a variable capacity hydraulic pump.

[0054] Control valve unit 17 is one of the hydraulic control devices for controlling the hydraulic system in construction machinery 100. In the example shown, control valve unit 17 includes control valves 171 to 176. Control valve unit 17 is configured to selectively supply working oil discharged from main pump 14 to one or more hydraulic actuators via control valves 171 to 176. Control valves 171 to 176 control the flow rate of working oil from main pump 14 to hydraulic actuators and the flow rate of working oil from hydraulic actuators to working oil reservoirs. Hydraulic actuators include boom cylinder 7, stick cylinder 8, bucket cylinder 9, left travel hydraulic motor 2ML, right travel hydraulic motor 2MR, and swing hydraulic motor 2A. Specifically, control valve 171 corresponds to left travel hydraulic motor 2ML, control valve 172 corresponds to right travel hydraulic motor 2MR, and control valve 173 corresponds to swing hydraulic motor 2A. Furthermore, control valve 174 corresponds to bucket cylinder 9, control valve 175 corresponds to boom cylinder 7, and control valve 176 corresponds to stick cylinder 8.

[0055] Pilot pump 15 is an example of a pilot pressure generating device, configured to supply working oil to the hydraulic control device via pilot lines. In the example shown, pilot pump 15 is a fixed-capacity hydraulic pump. However, the pilot pressure generating device can also be implemented using main pump 14. That is, in addition to supplying working oil to control valve unit 17 via working oil lines, main pump 14 can also supply working oil to various hydraulic control devices via pilot lines. In this case, pilot pump 15 can be omitted.

[0056] The discharge pressure sensor 28 is configured to detect the discharge pressure of the main pump 14. In the illustrated example, the discharge pressure sensor 28 outputs the detected value to the controller 30.

[0057] The operating device 26 is a device for an operator to operate the actuator. The operating device 26 includes, for example, an operating lever and an operating pedal. The actuator can be a hydraulic actuator or an electric actuator.

[0058] The operation sensor 29 is configured to detect the operation performed by the operator using the operation device 26. In this embodiment, the operation sensor 29 detects the operation direction and amount of the operation device 26 corresponding to each actuator and outputs the detected values ​​to the controller 30. In the example shown, the controller 30 can control the opening area of ​​the proportional valve 31 based on the output of the operation sensor 29. Furthermore, the controller 30 supplies the working oil discharged from the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17. The pressure of the working oil supplied to each pilot port (pilot pressure) is, in principle, the pressure corresponding to the operation direction and amount of the operation device 26 corresponding to each hydraulic actuator. Thus, the operation device 26 is configured to supply the working oil discharged from the pilot pump 15 to the pilot port of the corresponding control valve in the control valve unit 17.

[0059] The proportional valve 31, which functions as a control valve for equipment control, is configured as a conduit located at the pilot port of the control valve within the pilot pump 15 and the control valve unit 17, and the flow area of ​​this conduit can be varied. In the illustrated example, the proportional valve 31 operates according to control commands output by the controller 30. Therefore, the controller 30 can adjust the pilot pressure acting on the pilot port of the control valve via the proportional valve 31, independent of the operator's operation of the operating device 26.

[0060] With this structure, the controller 30 can activate the hydraulic actuator corresponding to the specific operating device 26 even without operation of the specific operating device 26.

[0061] And, as Figure 2As shown, the control system of the construction machinery 100 includes a controller 30, angle sensors S1, S2, S3, body tilt sensor S4, rotation sensor S5, camera device S6, positioning device PS, vibration detection device S7, sound collection device M1, sound output device SP1, display device D1, and communication device T1, etc.

[0062] The communication device T1 communicates with external devices through a communication network, including a mobile communication network, a satellite communication network, or the Internet. The communication device T1 may be, for example, a mobile communication module corresponding to mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation); a communication module corresponding to short-range wireless communication standards such as Wi-Fi (registered trademark) or Bluetooth (registered trademark); or a satellite communication module for connecting to a satellite communication network.

[0063] The boom angle sensor S1 detects the boom angle, which is the rotation angle of the boom 4 relative to the upper rotating body 3. The stick angle sensor S2 detects the stick angle, which is the rotation angle of the stick 5 relative to the boom 4. The bucket angle sensor S3 detects the bucket angle, which is the rotation angle of the bucket 6 relative to the stick 5.

[0064] The boom angle sensor S1, stick angle sensor S2, and bucket angle sensor S3 can be, for example, a rotary encoder, an accelerometer, a six-axis sensor, an IMU (Inertial Measurement Unit), etc., or they can be potentiometers using variable resistors, cylinder stroke sensors that detect the stroke of hydraulic cylinders, etc.

[0065] The detection signals corresponding to the boom angle detected by the boom angle sensor S1, the stick angle detected by the stick angle sensor S2, and the bucket angle detected by the bucket angle sensor S3 are input to the controller 30. In addition to angle, the detection signals may also include angular velocity.

[0066] In this embodiment, the boom angle sensor S1, the stick angle sensor S2, and the bucket angle sensor S3 are collectively referred to as angle sensors S1, S2, and S3.

[0067] A tilt sensor (an example of a detection device) S4 detects the tilt state of the machine body (lower traveling body 1 or upper rotating body 3) relative to the horizontal plane. The tilt sensor S4 is, for example, mounted on the upper rotating body 3, and detects the tilt angle of the construction machinery 100 (i.e., the upper rotating body 3) about two axes in the forward / backward and left / right directions. The tilt sensor S4 can be, for example, an accelerometer, a six-axis sensor, or an IMU. The detection signal corresponding to the tilt angle detected by the tilt sensor S4 is input to the controller 30.

[0068] The rotation sensor S5 outputs information related to the rotation of the upper rotating body 3. For example, the rotation sensor S5 detects the angular velocity of the upper rotating body 3 relative to the lower traveling body 1. The rotation sensor S5 can also detect the rotation angle. The rotation sensor S5 can be, for example, a gyroscope sensor, a rotary transformer, or a rotary encoder. The detection signal corresponding to the rotation angle or angular velocity of the upper rotating body 3 detected by the rotation sensor S5 is input to the controller 30.

[0069] The camera device S6 is mounted on the upper rotating body 3 to capture images of the periphery of the construction machinery 100, thereby obtaining images representing the periphery of the construction machinery 100. The camera device S6 may include, for example, a front camera, a left camera, a right camera, and a rear camera.

[0070] The front camera, which captures images of the front of the construction machinery 100, is mounted on the roof of the cab 10, the side of the boom 4, or other external parts of the cab 10. The left camera captures images of the left side of the construction machinery 100, the right camera captures images of the right side of the construction machinery 100, and the rear camera captures images of the rear of the construction machinery 100. Specifically, the front, left, right, and rear cameras are all single-lens wide-angle cameras equipped with imaging elements such as CCD or CMOS, and the image information captured is input to the controller 30. Furthermore, the images captured by the camera devices can be output to the display device D1.

[0071] The camera device S6 can constitute an object detection device for detecting objects present around the construction machinery 100. The object detection device can also be composed of devices other than a camera. For example, the object detection device can be a LiDAR (Light Detection and Ranging) system. A LiDAR, for example, is a device capable of measuring the distance between a group of more than one million points within the monitoring range and the LiDAR (laser source). Furthermore, the object detection device can also be other devices capable of measuring the distance to objects, such as a stereo camera, a distance imaging camera, or millimeter-wave radar. When using millimeter-wave radar or the like as an object detection device, the object detection device can derive the distance and direction of the object by emitting multiple signals (lasers, etc.) toward the object and receiving its reflected signals. Alternatively, the object detection device can be a combination of two or more devices. For example, the object detection device can be a combination of a camera device and a LiDAR, a combination of a camera device and a millimeter-wave radar, or a combination of a camera device and a stereo camera.

[0072] The positioning device PS determines the position of the upper rotating body 3. The positioning device PS, for example, is a GNSS (Global Navigation Satellite System) compass, which detects the position and orientation of the upper rotating body 3. The detection signal corresponding to the position and orientation of the upper rotating body 3 is input to the controller 30. The function of detecting the orientation of the upper rotating body 3 can also be achieved by an azimuth sensor installed on the upper rotating body 3. The positioning device PS according to this embodiment uses a reference coordinate system capable of determining the position globally to determine the current position of the construction machinery 100.

[0073] A reference coordinate system is, for example, the Global Geodetic System, which determines a location on Earth. The Global Geodetic System is a three-dimensional orthogonal XYZ coordinate system with the Earth's center of gravity as the origin, the direction of the intersection of the Greenwich Meridian and the equator as the X-axis, the direction of 90 degrees east longitude as the Y-axis, and the direction of the North Pole as the Z-axis.

[0074] Vibration detection device (an example of a detection device) S7 is configured to detect vibrations generated in the construction machinery 100. The vibration detection device S7 can be an IMU (Inertial Measurement Unit) or the like, and detects vibrations generated in the construction machinery 100 due to external causes.

[0075] The sound collecting device (an example of a detection device) M1 is a device for collecting external sound, also known as a microphone. For example, the sound collecting device M1 is installed on the upper rotating body 3 or the cab 10, and converts the sound (air vibration) generated around the construction machinery 100 into mechanical vibration, and then converts the mechanical vibration into an electrical signal. Specifically, the sound collecting device M1 may include a front microphone, a left microphone, a right microphone, and a rear microphone.

[0076] The front microphone collects sound generated in front of the construction machinery 100, the left microphone collects sound generated on the left side of the construction machinery 100, the right microphone collects sound generated on the right side of the construction machinery 100, and the rear microphone collects sound generated at the rear of the construction machinery 100. Furthermore, the electrical signals generated by the front, left, right, and rear microphones are input to the controller 30.

[0077] The sound output device SP1 is a device that outputs sound towards the surroundings of the construction machinery 100. The sound output device SP1 is an omnidirectional loudspeaker configured to output sound evenly in all directions. However, the sound output device SP1 can also be a directional loudspeaker that outputs sound in a specific direction, such as forward.

[0078] The controller 30 is an example of a control device, such as a computer comprising a CPU, volatile memory, non-volatile memory, and various input / output interfaces. The controller 30 implements various functions, for example, by reading programs from non-volatile memory and loading them into volatile memory for execution by the CPU. In the example shown, the controller 30 is configured to perform various functions to control the construction machinery 100. These functions include, for example, a device guidance function that guides the operator in manually operating the construction machinery 100. Other functions may include a contact avoidance function that automatically or autonomously causes the construction machinery 100 to move or stop in order to prevent contact between the construction machinery 100 and objects within its monitoring range.

[0079] The controller 30 is configured to output control commands to the regulator 13 as needed to change the discharge volume of the main pump 14.

[0080] Furthermore, the controller 30 may be configured, for example, to perform equipment guidance functions related to guiding the operator to manually operate the construction machinery 100 via the operating device 26. Also, the controller 30 may be configured, for example, to perform equipment control functions related to automatically supporting the operator to manually operate the construction machinery 100 via the operating device 26.

[0081] Furthermore, some of the functions of controller 30 can also be implemented by other controllers (control devices). That is, the functions of controller 30 can also be implemented in a distributed manner by multiple controllers. For example, equipment guidance functions and equipment control functions can be implemented by dedicated controllers (control devices).

[0082] <Framework Structure of Remote Operating Systems>

[0083] Figure 3This is a functional block diagram illustrating an example structure of the remote operating system SYS involved in this embodiment. Figure 3 The example shown illustrates the frame structure of the remote control room (RC) and the construction machinery 100 included in the remote operating system SYS. Further details regarding the hardware structure of the construction machinery 100 are omitted.

[0084] <Structure of the Remote Control Room (RC)>

[0085] The remote control room (RC) includes a remote controller R40, a communication device T2, an operation sensor R43, an operation device R42, a vibration device R44, a swing device R45, a sound collection device M1E, a sound output device SP1E, a display device D1E, and an input device D2E. The structures of the communication device T2, operation sensor R43, operation device R42, vibration device R44, swing device R45, sound collection device M1E, and sound output device SP1E have been described above and therefore omitted here.

[0086] The input device D2E is positioned within reach of the operator seated in the remote control room RC. It accepts various operational inputs from the operator and outputs signals corresponding to these inputs to the remote controller R40. The input device D2E includes a touch panel mounted on the display of various information images, a rotary switch located at the end of the lever of the operating device R42, and push-button switches, levers, toggle keys, and rotary dials located around the display device D1E. Signals corresponding to the operations performed on the input device D2E are input to the remote controller R40.

[0087] Next, the remote control room (RC) will be explained. Figure 4 This diagram illustrates a configuration example of a remote control room (RC). Multiple control devices R42 are installed within the remote control room (RC) with the operator's seat (DS) as the reference point.

[0088] In this embodiment, such as Figure 4 As shown, the display device D1E is a multi-screen display consisting of six monitors arranged in two rows and three columns. Specifically, the display device D1E includes a central monitor D1Ea, an upper monitor D1Eb, a left monitor D1Ec, a right monitor D1Ed, an upper left monitor D1Ee, and an upper right monitor D1Ef.

[0089] <<Description of Communication Status for Remote Operation>>

[0090] In this embodiment, the construction machinery 100 sends information to the remote operator OP, which is located in the remote control room RC, to identify the status of the construction machinery 100 or the status of the area surrounding the construction machinery 100.

[0091] Figure 5This is a conceptual diagram illustrating the information being sent and received between the construction machinery 100 and the remote control room RC involved in this embodiment.

[0092] like Figure 5 As shown, for example, images captured by camera device S6 are being transmitted from construction machinery 100 to remote control room RC.

[0093] In the remote control room RC, the remote controller R40 displays the received images on the display device D1E. The images contain information required by the remote operator OP to operate the construction machinery 100, and in this embodiment, they are considered essential information.

[0094] Operation information, generated by the operating device R42 in the remote control room RC, is being transmitted from the remote control room RC to the construction machinery 100. The construction machinery 100 is performing actions based on the received operation information. Thus, the operation information is the information required by the remote operator OP to operate the construction machinery 100, and in this embodiment, it is designated as one of the essential information.

[0095] In this embodiment, the information required for remotely operating the construction machinery 100 in the information sent or received between the remote control room RC and the construction machinery 100 is defined as necessary information.

[0096] On the other hand, additional information is defined as information other than images that the remote operator (OP) refers to in order to identify the condition of the construction machinery 100 or the conditions around the construction machinery 100 in the information sent or received between the remote control room (RC) and the construction machinery 100.

[0097] Furthermore, the sound signal collected by the sound collection device M1 is being transmitted from the construction machinery 100 to the remote control room RC. Additionally, the vibration information detected by the vibration detection device S7 is being transmitted from the construction machinery 100 to the remote control room RC. Furthermore, the tilt information detected by the machine tilt sensor S4 is being transmitted from the construction machinery 100 to the remote control room RC.

[0098] In the remote control room RC, the remote controller R40 is outputting received audio signals from the audio output device SP1E. Furthermore, the remote controller R40 is outputting received vibration information as vibration applied to the remote operator OP from the vibration device R44 via the operating seat DS. Additionally, the remote controller R40 is tilting the operating seat DS via the swing device R45 in a manner that reflects the current tilt state of the construction machinery 100 as indicated by the received tilt information.

[0099] In this embodiment, sound signals, vibration information, and tilt information are considered as supplementary information. However, this embodiment does not limit the supplementary information to sound signals, vibration information, and tilt information. Any information other than images representing at least one of the conditions of the construction machinery 100 and the conditions around the construction machinery 100 may be included. For example, force information for identifying the load generated in the accessory AT may also be included.

[0100] Furthermore, the additional information is not limited to information sent from the construction machinery 100 to the remote control room RC, but may also include information sent from the remote control room RC to the construction machinery 100. For example, audio signals sent from the remote control room RC to the construction machinery 100 may be included in the additional information.

[0101] In this embodiment, the communication device T1 of the construction machinery 100 uses multiple mobile communication lines to ensure communication bandwidth for real-time transmission and reception of information with the remote control room RC. Figure 5 In the example shown, a first mobile communication line and a second mobile communication line are used to enable the communication device T1 of the construction machinery 100 to communicate.

[0102] Furthermore, the construction machinery 100 may, for example, use the mobile communication line with higher communication quality among multiple mobile communication lines for sending or receiving necessary information, and use the mobile communication line with lower communication quality for sending or receiving additional information.

[0103] The communication device T1 of the construction machinery 100 is able to send and receive necessary information and additional information in real time by using multiple mobile communication lines.

[0104] However, the communication status between the construction machinery 100 and the remote control room RC often changes for various reasons. Therefore, the remote controller R40 and the controller 30 monitor the communication status to detect changes in the communication status.

[0105] The monitoring of communication status can be done using any method. For example, the remote controller R40 can detect anomalies such as packet loss or communication delay by executing a "PING" command with timestamp functionality enabled on the address assigned according to the information to be sent or received. For instance, if a delay is inferred based on the response speed detected by executing the "PING" command, and if the "PING" command does not respond, or if the timestamp deviates from the execution interval of the "PING" command (the time difference becomes larger), then packet loss or communication delay is identified. Furthermore, for each data packet sent by the controller 30 or the remote controller R40, a timestamp or data packet identification information can be embedded in the header, and the remote controller R40 or the controller 30 can confirm the information embedded in the received data packets to identify whether packet loss or communication delay has occurred.

[0106] Then, the remote controller R40 performs different controls on the necessary information and the additional information based on the communication status between the communication devices T1 and T2. Specifically, the remote controller R40 performs different actions depending on whether an anomaly has occurred in the transmission or reception of the necessary information or the transmission or reception of the additional information.

[0107] In other words, in the event of an anomaly while sending or receiving necessary information, security needs to be ensured. Therefore, there are methods for the remote controller R40 and controller 30 to stop actions based on operational information. The method for stopping the action could also be, for example, sending a stop signal from the remote controller R40 to the controller 30.

[0108] On the other hand, if an anomaly occurs when sending or receiving additional information, the remote operator (OP) can continue to operate the construction machinery 100 as long as the sending and receiving of necessary information does not fail.

[0109] However, in the event of an anomaly in the communication between communication devices T1 and T2 (e.g., packet loss or communication delay), when the received additional information is output to the remote control room RC, the remote operator (OP) may misidentify the construction machinery 100 or the conditions around it. For example, even if the construction machinery 100 has completed its movement from an inclined plane to a horizontal plane, if the remote controller R40 continues to tilt the operator seat DS based on delayed tilt information, the OP may mistakenly perceive that the construction machinery 100 is still moving on the inclined plane. Thus, in the event of a communication delay in the received additional information, the output based on that information may cause the OP to misidentify the condition of the construction machinery 100 or the conditions around it.

[0110] Furthermore, if the remote operator (OP) operates the construction machinery 100 based on an incorrect assessment of the condition of the construction machinery 100 or the conditions surrounding the construction machinery 100, the construction machinery 100 may perform actions inconsistent with the current situation.

[0111] Therefore, the remote controller R40 according to this embodiment, based on the communication status between the communication device T1 and the communication device T2, for example, if it is determined that the reception of vibration information is delayed or the data packet is lost, controls the output of vibration based on vibration information from the vibration device R44 to stop, while continuing the output of necessary information (e.g., images). Furthermore, if it is determined that the reception of sound signals and tilt information is delayed or the data packet is lost, the remote controller R40 similarly stops the output of sound signals from the sound output device SP1E or the tilt control of the tilt device R45 based on tilt information.

[0112] <<Excavator Function Box>>

[0113] Return to Figure 3 The functional blocks within the controller 30 of the construction machinery 100 will be described. The functional blocks within the controller 30 are conceptual and do not necessarily require the physical configuration shown in the figures. All or part of each functional block can be functionally or physically distributed / integrated in any unit. All or any part of the processing functions performed in each functional block are implemented by a program executed by the CPU. Alternatively, each functional block can be implemented in hardware based on wiring logic. The controller 30 includes an acquisition unit 301, a transmission control unit 302, a reception control unit 303, and an actuator drive unit 304 by implementing a program.

[0114] The acquisition unit 301 acquires signals from various detection devices installed on the construction machinery 100. For example, the acquisition unit 301 acquires tilt information from the body tilt sensor S4, indicating the tilt state of the machine body (lower traveling body 1 or upper rotating body 3) relative to the horizontal plane. Furthermore, the acquisition unit 301 acquires position information from the positioning device PS, indicating the measurement results of the position and orientation of the construction machinery 100. Additionally, the acquisition unit 301 acquires vibration information from the vibration detection device S7, indicating the detection results of vibrations generated in the construction machinery 100.

[0115] Furthermore, the acquisition unit 301 acquires images from the camera device S6. Additionally, the acquisition unit 301 acquires sound signals representing sounds collected from the vicinity of the construction machinery 100 from the sound collection device M1.

[0116] The transmission control unit 302 controls the transmission of various information based on the acquisition results of the acquisition unit 301 to the remote control room RC via the communication device (an example of the first communication device) T1. For example, the transmission control unit 302 controls the transmission of tilt information detected by the body tilt sensor S4, vibration information indicating vibrations generated in the construction machinery 100, images captured by the camera device S6, sound signals collected by the sound collection device M1, and position information indicating the position and orientation of the construction machinery 100 to the remote control room RC.

[0117] The receiving control unit 303 receives various information from the remote control room RC via the communication device T1. For example, the receiving control unit 303 receives operation signals from the remote control room RC for controlling the operation of the construction machinery 100.

[0118] The actuator drive unit 304 is configured to drive the actuator mounted on the construction machinery 100. In this embodiment, the actuator drive unit 304 generates and outputs operating signals for each of the plurality of solenoid valves included in the proportional valve 31 based on the operating signal sent from the remote control room RC.

[0119] Upon receiving the operation signal, each solenoid valve increases or decreases the pilot pressure at the pilot port of the corresponding control valve in control valve unit 17. As a result, the hydraulic actuator corresponding to each control valve operates at a speed corresponding to the stroke of the control valve.

[0120] <<Functional Framework of Remote Control Room>>

[0121] The functional blocks within the remote controller (an example of a control unit) R40 of the remote control room RC will be described. The functional blocks within the remote controller R40 are conceptual and do not necessarily require the physical configuration shown in the figures. All or part of each functional block can be functionally or physically distributed / integrated in any unit. All or part of the processing functions performed in each functional block are implemented by a program executed by the CPU. Alternatively, each functional block can be implemented in hardware based on wiring logic. The remote controller R40, due to the implementation of the program, includes a receiving control unit 401, a display control unit 402, an output control unit 403, a decision unit 404, a signal generation unit 405, and a transmission control unit 406.

[0122] The receiving control unit 401 performs control for receiving various information from the construction machinery 100 via the communication device T2.

[0123] For example, the receiving control unit 401 controls the receiving of necessary information from the construction machinery 100. Specifically, the receiving control unit 401 controls the receiving of images captured by the camera device S6.

[0124] As another example, the receiving control unit 401 controls the receiving of additional information from the construction machinery 100. Specifically, the receiving control unit 401 controls the receiving of tilt information detected by the body tilt sensor S4, vibration information detected by the vibration detection device S7, and sound signals collected by the sound collection device M1.

[0125] Then, the receiving control unit 401 controls the receiving of position information indicating the position and orientation of the construction machinery 100 from the construction machinery 100.

[0126] The display control unit 402 controls the display of various information on the display device D1E. For example, the display control unit 402 controls the display of images captured by the camera device S6 on the display device D1E as images showing the surroundings of the construction machinery 100. Furthermore, the display control unit 402 controls the display of information indicating the current status of the construction machinery 100 on the display device D1E based on the detection results of various detection devices of the construction machinery 100 received by the receiving control unit 401.

[0127] The output control unit 403 performs output control based on additional information. For example, the output control unit 403 controls the output of the received audio signal from the audio output device SP1E.

[0128] Furthermore, the output control unit 403 performs tilt control of the operating seat DS based on the received tilt information via the swing device R45. However, this embodiment does not limit the output based on tilt information to tilt control of the operating seat DS. For example, there is also a method where the display control unit 402 displays information (e.g., an icon of the construction machinery 100, etc.) that can identify the tilt angle of the construction machinery 100 on the display device D1E.

[0129] Furthermore, the output control unit 403 performs vibration control of the operating seat DS based on the received vibration information via the vibration device R44.

[0130] The determination unit 404 determines whether an abnormality has occurred in the communication status of the necessary information and supplementary information being received. For example, the determination unit 404 determines whether data packet loss or communication delay has occurred in the necessary information and supplementary information being received. The method for determining whether data packet loss or communication delay has occurred can be any method. For example, it can detect whether an abnormality such as data packet loss or communication delay has occurred by executing the "PING" command, or it can detect an abnormality such as data packet loss or communication delay by referring to information embedded in the header of each data packet.

[0131] Furthermore, the determination unit 404 can also determine whether any other abnormality has occurred, in addition to determining whether an abnormality has occurred in the communication status. For example, the determination unit 404 can determine whether a delay or abnormality has occurred in the encoding or decoding of the audio signal included in the additional information. The method for determining whether a delay or abnormality has occurred in the encoding of the audio signal can be any method. For example, the determination unit 404 can determine whether a delay or abnormality has occurred based on notifications output from the controller 30 of the construction machinery 100 at the start and end times of the audio signal encoding (e.g., a PING with timestamp function enabled).

[0132] The signal generation unit 405 generates an operation signal for controlling the operation of the construction machinery 100 based on the information received from the operation sensor R43.

[0133] Furthermore, if the determination unit 404 determines that the necessary information being received has become abnormal, the signal generation unit 405 generates an operation signal to stop the operation of the construction machinery 100.

[0134] The transmission control unit 406 controls the transmission of various information to the construction machinery 100. For example, the transmission control unit 406 controls the transmission of operation signals generated by the signal generation unit 405 to the construction machinery 100.

[0135] If the determination unit 404 determines that the communication status of the additional information being received has become abnormal, it stops outputting additional information from various output devices and continues to output necessary information (e.g., images) to the display device D1E.

[0136] For example, if the determination unit 404 determines that a data packet loss or communication delay has occurred in the tilt information being received, the output control unit 403 controls the tilt control of the operating seat DS by the swaying device R45. Furthermore, when controlling the sway to stop, the output control unit 403 gradually reduces the tilt amount based on the tilt information.

[0137] Furthermore, if the determination unit 404 determines that a data packet loss or communication delay has occurred in the received vibration information, the output control unit 403 controls the output of vibration from the vibration device R44 to stop. At this time, the output control unit 403 controls the amount of vibration output from the vibration device R44 to gradually decrease based on the vibration information.

[0138] Furthermore, if the determination unit 404 determines that a data packet loss or communication delay has occurred in the received audio signal, it determines whether there is anyone around the construction machinery 100. Only if it is determined that no one is present does the output control unit 403 control the output of audio from the audio output device SP1E to stop. At this time, the output control unit 403 controls the volume of the audio output from the audio output device SP1E to gradually decrease. That is, if someone is around the construction machinery 100, that person may make a sound. It is assumed that the sound made by a person should preferably be output to the remote operator OP even if it is interrupted or delayed. Therefore, in this embodiment, even if a data packet loss or delay occurs in the audio signal, audio output based on the audio signal will continue even if someone is around the construction machinery 100.

[0139] After stopping the output based on the additional information, the determination unit 404 determines whether the abnormality has been eliminated in the communication status of receiving the additional information.

[0140] For example, if the determination unit 404 determines that data packet loss and communication delay have been eliminated in the tilt information being received, the output control unit 403 initiates the tilt control of the operating seat DS by the swing device R45. Furthermore, when initiating the control, the output control unit 403 gradually increases the tilt amount based on the tilt information.

[0141] Similarly, when the determination unit 404 determines that data packet loss and communication delay have been eliminated in the received vibration information or sound signal, the output control unit 403 controls the output of vibration from the vibration device R44 or the output of sound from the sound output device SP1E. At this time, the output control unit 403 controls the gradual increase of the vibration amount output from the vibration device R44 or the volume output from the sound output device SP1E based on the vibration information.

[0142] Figure 6 This diagram illustrates an example of volume output control in the output control unit 403 according to this embodiment. Figure 6 In the example shown, the vertical axis represents volume, and the horizontal axis represents time. Additionally, in Figure 6 The example shown illustrates the case where the vertical axis represents volume, but this will be similarly converted depending on the amount of vibration or tilt, and the explanation is omitted. Figure 6 In the example shown, it is assumed that there is no one around the construction machinery 100.

[0143] exist Figure 6In the example shown, at time t1, the determination unit 404 determines that a data packet loss or communication delay has occurred in the received audio signal. Therefore, the output control unit 403 begins to gradually reduce the volume output from the audio output device SP1E. Then, at time t2, the output control unit 403 stops the audio output from the audio output device SP1E.

[0144] Then, at time t3, the determination unit 404 determines that data packet loss and communication delay have been eliminated in the received audio signal. Therefore, the output control unit 403 starts outputting audio from the audio output device SP1E. Furthermore, the output control unit 403 begins to gradually increase the volume output from the audio output device SP1E. Then, at time t4, the gradual increase in the volume output from the audio output device SP1E stops.

[0145] In this embodiment, control is achieved by reducing or increasing the amount of vibration, tilt, or volume based on additional information. This suppresses sudden stops or starts of the output, thereby preventing the remote operator from being startled. Therefore, the safety of operations performed by the remote operator can be ensured.

[0146] The display control unit 402 in this embodiment controls the display device D1E to display the content that has stopped output when the output related to additional information is stopped from various output devices.

[0147] Figure 7 This is a diagram showing an example of a screen displayed by the display control unit 402 according to this embodiment. Figure 7 The example shown illustrates the screen displayed by the control unit 402 when the determination unit 404 determines that data packet loss and communication delay have occurred in the vibration and tilt information being received.

[0148] exist Figure 7 In the example shown, the central display D1Ea of the display device D1E displays an image captured by the front camera of the imaging device S6. Furthermore, the display control unit 402 overlays a message stating "Communication error; stop vibration and tilt control" onto this image. The message in this embodiment is displayed, for example, in a semi-transparent form, allowing the image to be visually recognized through the message. Therefore, suppressing the display of the message hinders the operation of the construction machinery 100. After this display, the output control unit 403 begins to gradually reduce the control of vibration and tilt.

[0149] This display allows the remote operator to identify communication anomalies without taking their eyes off the device. Furthermore, the remote operator can detect when vibration and tilt controls gradually cease. The remote operator can use this display and control information to infer the current communication status. Therefore, the remote operator can perform operations that take the communication status into account, thus improving safety.

[0150] Furthermore, the display control unit 402 controls the display device D1E to display the resumed output content when the output related to additional information is restarted from various output devices. For example, if the determination unit 404 determines that the loss of vibration and tilt information data packets or communication delays have been eliminated, the display control unit 402 displays a message on the display device D1E stating "Communication restored; vibration and tilt control started." The remote operator OP can recognize that vibration and tilt control will restart through this display and control. Then, the remote operator OP can infer the condition of the construction machinery 100 or the condition around the construction machinery 100 based on the vibration and tilt control and operate accordingly, thus improving safety.

[0151] The following describes the control performed by the remote controller R40 according to this embodiment, corresponding to the communication status of the vibration information and tilt information in the additional information. Figure 8 This is a flowchart illustrating the processing sequence of the remote controller R40 involved in this embodiment, corresponding to the communication status of vibration information and tilt information.

[0152] The remote controller R40's receiving control unit 401 begins receiving vibration and tilt information (S1801).

[0153] The determination unit 404 determines whether a communication delay or data packet loss has occurred in the received vibration information and tilt information respectively (S1802). The determination criterion for whether a communication delay or data packet loss has occurred can be any determination criterion, such as a previously used determination criterion.

[0154] If the determination unit 404 determines that no communication delay or data packet loss has occurred (S1802: No), the output control unit 403 performs output control of the vibration of the operating seat DS based on the vibration information being received and tilt control of the operating seat DS based on the tilt information (S1803). Then, control is resumed from S1801.

[0155] On the other hand, if the determination unit 404 determines that a communication delay or data packet loss has occurred in at least one of the vibration information and tilt information being received (S1802: Yes), the display unit 402 displays the content of at least one of the output control to stop vibration and tilt control (S1804).

[0156] After performing at least one of the controls to gradually reduce the vibration output based on the vibration information and the tilt amount to perform tilt control based on the tilt information, the output control unit 403 stops at least one of the vibration output control and tilt control (S1805). Furthermore, the control to be stopped in the vibration output control and tilt control is the control corresponding to information indicating a communication delay or data packet loss (either the vibration information or the tilt information).

[0157] Then, the determination unit 404 determines whether communication delay and data packet loss have been eliminated in the vibration information and tilt information being received (S1806).

[0158] If the determination unit 404 determines that the communication delay or data packet loss has not been eliminated (S1806: No), the determination in S1806 will be repeated after a specified time has elapsed.

[0159] On the other hand, if the determination unit 404 determines that the communication delay or data packet loss has been eliminated (S1806: Yes), the display unit 402 displays the output of at least one of the output control for restarting vibration and tilt control (S1807).

[0160] After outputting at least one of the vibration output control and tilt control, the output control unit 403 performs at least one of the following controls: gradually increasing the vibration output based on the vibration information and adjusting the tilt amount based on the tilt information (S1808). Then, processing is repeated from step S1801. Furthermore, the control to be output in the vibration output control and tilt control is a control corresponding to information that has eliminated communication delay or data packet loss (either vibration information or tilt information).

[0161] As described above, the remote controller R40 of this embodiment, after ceasing output related to any one or more of the additional information, such as vibration information and tilt information, resumes output related to the additional information based on the communication status between communication device T1 and communication device T2. If communication delays or other issues are resolved, the output of vibration and any one or more of the tilt control are resumed. Therefore, the remote operator OP can identify the condition of the construction machinery 100 or the conditions around the construction machinery 100, thereby improving safety.

[0162] Next, the control performed by the remote controller R40 according to this embodiment in accordance with the communication status of the audio signal in the additional information will be described. Figure 9 This is a flowchart illustrating the processing sequence of the remote controller R40 according to this embodiment, corresponding to the communication status of the audio signal.

[0163] The remote controller R40's receiving control unit 401 begins receiving audio signals (S1901).

[0164] The determination unit 404 determines whether a communication delay or data packet loss has occurred in the received audio signal (S1902). The determination criterion for whether a communication delay or data packet loss has occurred can be any determination criterion, such as a previously used determination criterion.

[0165] If the determination unit 404 determines that no communication delay or data packet loss has occurred (S1902: No), the output control unit 403 controls the output of the sound signal being received from the sound output device SP1E (S1903). Then, control is resumed from S1901.

[0166] On the other hand, if the determination unit 404 determines that a communication delay or data packet loss has occurred in the received audio signal (S1902: Yes), the determination unit 404 determines whether there are people around in the received video (S1904). If it is determined that there are people around (S1904: Yes), the display control unit 402 displays a message indicating that there is an abnormality in the audio output (e.g., delay or interruption) (S1905). Then, the output control unit 403 controls the output of the audio signal represented by the received audio signal from the audio output device SP1E (S1903). Then, control is resumed from S1901.

[0167] On the other hand, if the determination unit 404 determines that there is no one around in the image being received (S1904: No), the display control unit 402 displays the content to stop the sound output (S1906).

[0168] Then, after gradually reducing the volume of the sound output based on the sound signal, the output control unit 403 stops the sound output control (S1907).

[0169] Then, the determination unit 404 determines whether communication delay and data packet loss have been eliminated in the received audio signal (S1908).

[0170] If the determination unit 404 determines that the communication delay or data packet loss has not been eliminated (S1908: No), the determination in S1908 will be repeated after a specified time has elapsed.

[0171] On the other hand, if the determination unit 404 determines that the communication delay or data packet loss has been eliminated (S1908: Yes), the display unit 402 displays the content of restarting the sound output (S1909).

[0172] After the sound output begins, the output control unit 403 gradually increases the volume of the sound signal output (S1910). Then, it restarts the process from step S1901.

[0173] The remote controller R40 of this embodiment continues to output sound signals regardless of the communication status when it determines that a person is present in the received image. Therefore, although there may be abnormalities such as delays or interruptions in the output sound, the sound emitted by a person present in the vicinity of the construction machinery 100 can be heard. Thus, the remote operator OP can identify the situation around the construction machinery 100, thereby improving safety.

[0174] The remote controller R40 involved in this embodiment is performing... Figure 8 and Figure 9 During the processing shown, the process of displaying the image included in the necessary information on the display device D1E continues. Therefore, in the event of an anomaly in the additional information, the remote operator (OP) can suppress erroneous judgments about the surrounding situation or the condition of the construction machinery 100 caused by the additional information by stopping the output based on the additional information. Furthermore, the remote operator (OP) can judge the surrounding situation based on the image, thereby continuing the operation of the construction machinery 100.

[0175] (Second Implementation)

[0176] In the first embodiment, a situation where communication issues occurred while receiving additional information was described. Therefore, in the second embodiment, a situation where communication issues occurred while receiving necessary information is described. Furthermore, the structure of the remote operating system SYS in the second embodiment is the same as that in the first embodiment, so its description is omitted.

[0177] Next, the control performed by the remote controller R40 of this embodiment in accordance with the communication status of necessary information will be described. Figure 10 This is a flowchart illustrating the processing sequence of the remote controller R40 involved in this embodiment, corresponding to the communication status of necessary information.

[0178] The receiving control unit 401 of the remote controller R40 begins receiving control (S2001) that includes necessary information about the image.

[0179] The determination unit 404 determines whether a communication delay or data packet loss has occurred in the necessary information being received (S2002).

[0180] If the determination unit 404 determines that no communication delay or data packet loss has occurred (S2002: No), for example, the display control unit 402 also controls the display device D1E to display an image containing the necessary information being received (S2003).

[0181] On the other hand, if the determination unit 404 determines that a communication delay or data packet loss has occurred in the necessary information being received (S2002: Yes), the display control unit 402 performs a communication status deterioration and stops displaying the content related to the additional information output (S2004).

[0182] Then, after controlling the output amount related to the additional information (e.g., vibration information, tilt information, and sound signal), the output control unit 403 stops the output related to the additional information (S2005).

[0183] Then, the transmission control unit 406 sends a request to the construction machinery 100 to stop sending additional information and a request to send the content of necessary information using the communication line originally used for communication of additional information (S2006).

[0184] Then, the receiving control unit 401 of the remote controller R40 begins to receive the necessary information containing the image from the switched communication line (S2007).

[0185] In this embodiment, the remote controller R40, upon determining that an anomaly has occurred in the communication status of essential information, stops sending additional information to the controller 30 and uses the communication bandwidth of the additional information for the communication of essential information, thereby continuing the communication of essential information. In this embodiment, as an example of a situation where the communication status of essential information does not meet the prescribed criteria, the situation where an anomaly has occurred in the communication status of essential information is described; in other words, the situation where a communication delay or data packet loss has occurred. However, this embodiment does not limit the situation where the communication status of essential information does not meet the prescribed criteria to the situation where a communication delay or data packet loss has occurred; any situation where an anomaly has occurred in the communication status of the received essential information to the point that the remote operator OP finds it difficult to operate the construction machinery 100 is acceptable. Furthermore, the remote controller R40 in this embodiment is not limited to the method of stopping the transmission of additional information; it may also reduce the amount of additional information communication compared to before the determination and use the communication bandwidth originally used for additional information for the communication of essential information, thereby continuing the communication of essential information. In this embodiment, by using the communication bandwidth of the additional information for the reception of essential information according to the communication status of essential information, the stability of the reception of essential information can be improved. Therefore, remote operators can easily grasp the status of construction machinery 100 or the situation around construction machinery 100 based on necessary information, thus improving safety.

[0186] (Third Implementation)

[0187] In the above embodiments, the control for receiving additional or necessary information has been described. However, the above embodiments are not limited to the control for receiving additional or necessary information. Therefore, in the third embodiment, the control for sending additional and necessary information will be described. Furthermore, the structure of the remote operating system SYS in the third embodiment is the same as that in the first embodiment, so its description is omitted.

[0188] The communication device T2 in this embodiment sends necessary information, including operation signals, to the communication device T1 of the construction machinery 100 under the control of the remote controller R40.

[0189] In this embodiment, the sound collection device M1E of the remote control room RC collects the sound emitted by the remote operator OP as an audio signal. The communication device T2 transmits the audio signal to the communication device T1 of the construction machinery 100 according to the control of the remote controller R40. In this embodiment, the audio signal transmitted to the communication device T1 of the construction machinery 100 is used as supplementary information.

[0190] Furthermore, the remote controller R40 performs the same control as the one described in the above embodiment when sending additional and necessary information.

[0191] For example, if the determination unit of the remote controller R40 determines that the communication status of the necessary information being transmitted has become abnormal, the transmission control unit 406 shall control the transmission as follows: stop sending additional information to the construction machinery 100, and send the necessary information using the communication line originally used for the additional information. The method for switching the communication line can be any method, whether or not it is a known method.

[0192] In this embodiment, the remote controller R40, upon determining that an anomaly has occurred in the communication status of essential information, stops sending additional information and uses the communication bandwidth originally allocated to additional information for sending essential information, thereby continuing the transmission of essential information. Furthermore, the remote controller R40 in this embodiment is not limited to the method of stopping the transmission of additional information; it can also reduce the amount of additional information communication compared to before the determination and use the communication bandwidth originally allocated to additional information for sending essential information, thereby continuing the transmission of essential information. In this embodiment, by using the communication bandwidth of additional information for sending essential information based on the communication status of essential information, the stability of essential information transmission can be improved. Therefore, in this embodiment, by stabilizing the transmission of essential information including operation signals, operability and the stability of the operation of the construction machinery 100 can be improved.

[0193] <Function>

[0194] In the above embodiment, safety is improved by being able to output or send information corresponding to the current communication status based on the communication status between the communication device T1 of the construction machinery 100 and the communication device T2 of the remote control room RC.

[0195] In the above embodiment, in the event of an anomaly in the communication status of additional information between the communication device T1 of the construction machinery 100 and the communication device T2 of the remote control room RC, control is performed to stop the output related to the additional information and continue the output related to the necessary information. This allows the construction machinery 100 to continue operating using the output of the necessary information, and it also suppresses the remote operator OP from making incorrect judgments due to the additional information, thereby improving safety.

[0196] The preferred embodiments and variations of this disclosure have been described above. However, the invention involved in this disclosure is not limited to the above embodiments. Various modifications and substitutions can be applied to the above embodiments without departing from the scope of the invention involved in this disclosure. Furthermore, the features described with reference to the above embodiments can be appropriately combined as long as there is no technical contradiction.

Claims

1. A remote operation system of a construction machine, comprising: a construction machine; a remote operation room; and a control device, wherein the construction machine includes an imaging device that acquires an image of a vicinity of the construction machine, a detection device that detects additional information indicating information other than the image among information indicating at least one or more of a state of the construction machine and a state of the vicinity of the construction machine, and a first communication device that transmits the image and the additional information, the remote operation room includes an operation device, a second communication device that transmits operation information from the operation device to the first communication device and receives the image and the additional information, a display device that displays the image, and an output device that outputs the additional information, and the control device controls transmission or reception of necessary information required for operation of the construction machine and the additional information between the first communication device and the second communication device, the necessary information including at least one or more of the image and the operation information, and the control device individually controls the necessary information and the additional information differently from each other in accordance with a communication state between the first communication device and the second communication device.

2. The remote operation system of the construction machine according to claim 1, wherein the control device controls to stop output of the additional information from the output device and continue output of the necessary information in accordance with a communication state of the additional information between the first communication device and the second communication device.

3. The remote operation system of the construction machine according to claim 2, wherein the additional information received by the second communication device from the first communication device includes a sound signal indicating a sound collected from the vicinity of the construction machine, and the control device continues to output the sound indicated by the sound signal included in the additional information regardless of the communication state in a case where it is determined that a person appears in the image received.

4. The remote operation system of the construction machine according to claim 2, wherein the control device starts to output the additional information from the output device in accordance with a communication state between the first communication device and the second communication device after stopping the output of the additional information.

5. The remote operation system of the construction machine according to claim 4, wherein the control device controls to gradually reduce an output amount of the additional information when stopping the output of the additional information, or controls to gradually increase the output amount of the additional information output from the output device when starting the output of the additional information.

6. The remote operation system of the construction machine according to claim 4, wherein the control device outputs a content of stopping the output of the additional information from the output device to the display device when stopping the output of the additional information from the output device, or outputs a content of starting the output of the additional information from the output device to the display device when starting the output of the additional information.

7. The remote operation system of the construction machine according to claim 1, wherein ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ The control device continues transmission or reception of the necessary information while reducing the amount of communication of the additional information compared to before the determination, in a case where it is determined that the communication condition of the necessary information between the first communication device and the second communication device does not satisfy a prescribed reference.

Images