Information processing system of work machine

The information processing system addresses the challenge of blind spots in working machine imaging by using separate imaging devices to capture and transmit comprehensive surroundings to a remote control room, improving safety through enhanced situational awareness.

JP2024159065APending Publication Date: 2024-11-08SUMITOMO HEAVY IND LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2023074813
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-04-28
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing imaging systems installed in working machines often struggle to capture objects in blind spots, making it difficult to grasp the complete surrounding situation effectively.

Method used

An information processing system that utilizes imaging devices separate from the working machine, capturing images from multiple angles and transmitting them to a remote control room for comprehensive situational awareness, including the use of fixed point measuring devices and LIDAR for spatial recognition.

Benefits of technology

Enhances safety by providing a comprehensive view of the working machine's surroundings, enabling operators to detect and avoid obstacles in blind spots through multiple perspectives.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024159065000001_ABST
    Figure 2024159065000001_ABST
Patent Text Reader

Abstract

To achieve improvement of safety.SOLUTION: An information processing system of a work machine includes: a work machine performing work at a work site; an imaging device which is provided separately from the work machine so as to image the work site; and a control device which receives image information obtained by imaging the periphery of the work machine including the work machine by the imaging device via network, and is configured to perform processing based on the received image information.SELECTED DRAWING: Figure 5
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an information processing system for a work machine. [Background technology]

[0002] 2. Description of the Related Art Conventionally, techniques have been proposed for grasping the situation around a work machine by using an imaging device provided on the work machine. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-183624 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, there are cases where it is difficult for an imaging device provided on a work machine to detect objects and the like that are present in the blind spot of the work machine.

[0005] In view of the above, by using image information captured around the work machine by an imaging device provided separately from the work machine, it becomes easier to grasp the situation around the work machine, thereby improving safety. [Means for solving the problem]

[0006] An information processing system for a work machine according to one aspect of the present invention includes a work machine performing work at a work site, an imaging device provided separately from the work machine for imaging the work site, and a control device configured to receive image information captured by the imaging device via a network, the image information including the work machine and the surroundings of the work machine, and to perform processing based on the received image information. Effect of the Invention

[0007] According to one aspect of the present invention, improved safety is achieved by using image information captured by an imaging device provided separately from the work machine. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of a remote control system according to the first embodiment. [Diagram 2] FIG. 2 is a side view showing the shovel (excavator) according to the first embodiment. [Diagram 3] FIG. 3 is a block diagram showing an example of the configuration of a drive system of the shovel according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of the configuration of an electrical system mounted on the shovel according to the first embodiment. [Diagram 5] FIG. 5 is a functional block diagram illustrating a configuration example of the remote control system according to the first embodiment. [Figure 6] FIG. 6 is a diagram showing an example of the layout of a remote control room according to the first embodiment. [Figure 7] FIG. 7 is a diagram showing an example of a screen displayed on the display device according to the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of the positional relationship between the fixed point measuring device and the shovel according to the first embodiment. [Figure 9] FIG. 9 is a diagram illustrating a method for calculating the distance between the shovel and an obstacle according to the first embodiment. [Figure 10] FIG. 10 is a diagram showing an example of switching in response to an operation on a screen displayed on the display device according to the first embodiment. [Figure 11] FIG. 11 is a sequence diagram showing a process for displaying information on the display device D1 in the remote control system for a shovel according to the first embodiment. [Figure 12] FIG. 12 is a diagram illustrating the movement of the shovel when image information displayed on the display device according to the first embodiment is switched. [Figure 13] FIG. 13 is a diagram showing an example of a screen displayed on the display device according to the first embodiment. [Figure 14] FIG. 14 is a side view showing an example of a crawler crane and a fixed point measuring device included in the processing system according to the second embodiment. [Figure 15] FIG. 15 is a side view showing an example of a continuous unloader and a fixed-point measuring device included in the processing system according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is an example, not a limitation of the invention, and all features and combinations described in the embodiment are not necessarily essential to the invention. In addition, the same or corresponding components in each drawing are denoted by the same or corresponding reference numerals, and the description may be omitted.

[0010] (First embodiment) First, an overview of a remote control system (an example of an information processing system for a work machine) SYS according to a first embodiment will be described with reference to Fig. 1. Fig. 1 is a schematic diagram showing an example of the remote control system SYS according to the first embodiment.

[0011] <Devices that make up the remote control system> As shown in FIG. 1, the remote operation system SYS according to the first embodiment includes a shovel 100, fixed point measurement devices (examples of imaging devices) C11 and C12, and a remote operation room RC.

[0012] The shovel 100 is an example of a work machine that performs work at a work site and is capable of wireless communication. The shovel 100 is capable of transmitting and receiving data to and from devices (e.g., a remote control room RC) connected to the communication network NT. The fixed point measurement devices C11, C12 are capable of transmitting and receiving data to and from devices (e.g., a remote control room RC) connected to the communication network NT.

[0013] The shovel 100 and the fixed point measurement devices C11, C12 are present at the work site where the shovel 100 performs work. The shovel 100 and the fixed point measurement devices (examples of imaging devices) C11, C12 transmit information about the work site to the remote control room RC. This allows the remote control room RC to check the work site from multiple angles according to the detection results from multiple types of devices.

[0014] The fixed point measurement devices C11 and C12 are an example of an imaging device provided at a work site separately from the shovel 100 in order to photograph the work site. In this embodiment, an example in which two fixed point measurement devices C11 and C12 are provided at the work site will be described, but the number of fixed point measurement devices provided at the work site is not limited, and may be one, or three or more.

[0015] In this embodiment, at least one of the fixed point measurement devices C11, C12 is set up to be able to photograph the shovel 100 present at the work site. The fixed point measurement devices C11, C12 transmit photographed image information to the remote control room RC. One of the image information photographed by the fixed point measurement devices C11, C12 shows the surrounding situation including the shovel 100. Therefore, the remote control room RC can recognize the surrounding situation including the shovel 100.

[0016] The fixed point measurement devices C11, C12 according to this embodiment are provided so as to capture images from a position higher than the upper rotating body 3 of the shovel 100. This enables the fixed point measurement devices C11, C12 to capture images of the situation on the opposite side of the shovel 100. Note that this embodiment does not limit the height of the fixed point measurement devices C11, C12, and they may be provided so as to be lower than the shovel 100, for example.

[0017] In this embodiment, an example is shown in which fixed-point measurement devices C11, C12 are used as an example of an imaging device provided separately from the shovel 100 to capture images of the work site. The imaging device provided separately from the shovel 100 may be other than the fixed-point measurement devices C11, C12, and may be, for example, a surrounding monitoring device already provided at the work site.

[0018] The fixed-point measuring devices C11 and C12 may be stereo cameras capable of detecting the distance to an object. Furthermore, a LIDAR (Light Detection and Ranging) may be installed near the fixed-point measuring devices C11 and C12 to detect the situation at the work site. For example, the LIDAR measures the distance between the LIDAR and one million or more points within the monitoring range. Note that this embodiment is not limited to a method using the LIDAR, and any spatial recognition device capable of measuring the distance to an object may be used. As another example, a distance measuring device such as a millimeter wave radar may be combined instead of the LIDAR. The shape of the object around the shovel 100 may be recognized as point cloud data by the LIDAR.

[0019] The remote operation system SYS may include one or more excavators 100.

[0020] <Example of remote control room configuration> The remote control room RC is equipped with a communication device T2, a remote controller R30, an operation device R26, an operation sensor R29, and a display device D1. Also, the remote control room RC is provided with an operation seat DS where an operator OP who remotely operates the excavator 100 sits.

[0021] The communication device T2 is configured to control communication between the communication device T1 (see FIG. 2) attached to the shovel 100 and the communication device T3 (see FIG. 5) attached to the fixed point measuring devices C11 and C12.

[0022] The remote controller R30 is a calculation device that executes various calculations. In this embodiment, the remote controller R30 is configured with a microcomputer including a CPU and a memory. The various functions of the remote controller R30 are realized by the CPU executing a program stored in the memory.

[0023] The display device D1 displays a screen based on information transmitted from each of the shovel 100 and the fixed point measurement devices C11 and C12 so that the operator OP in the remote control room RC can visually confirm the surroundings of the shovel 100. The display device D1 enables the operator to confirm the status of the work site including the surroundings of the shovel 100 even though he is in the remote control room RC.

[0024] The operation device R26 is provided with an operation sensor R29 for detecting the operation content of the operation device R26. The operation sensor R29 is, for example, an inclination sensor that detects the inclination angle of the operation lever, or an angle sensor that detects the swing angle of the operation lever around the swing axis. The operation sensor R29 may be composed of other sensors such as a pressure sensor, a current sensor, a voltage sensor, or a distance sensor. The operation sensor R29 outputs information on the detected operation content of the operation device R26 to the remote controller R30. The remote controller R30 generates an operation signal based on the received information and transmits the generated operation signal to the excavator 100. The operation sensor R29 may be configured to generate an operation signal. In this case, the operation sensor R29 may output the operation signal to the communication device T2 without passing through the remote controller R30. This allows remote operation of the excavator 100 to be realized from the remote operation room RC.

[0025] <Excavator configuration> Fig. 2 shows an excavator 100 as a work machine according to an embodiment. An upper rotating body 3 is rotatably mounted on a lower traveling body 1 of the excavator 100 via a rotating mechanism 2. A boom 4 is attached to the upper rotating body 3. An arm 5 is attached to the tip of the boom 4, and a bucket 6 is attached to the tip of the arm 5 as an end attachment.

[0026] The boom 4, the arm 5, and the bucket 6 constitute an excavation attachment, which is an example of an attachment. The boom 4 is driven by a boom cylinder 7, the arm 5 is driven by an arm cylinder 8, and the bucket 6 is driven by a bucket cylinder 9.

[0027] A boom angle sensor S1 is attached to the boom 4, an arm angle sensor S2 is attached to the arm 5, and a bucket angle sensor S3 is attached to the bucket link. A swing angular velocity sensor S5 is attached to the upper swing body 3.

[0028] The boom angle sensor S1 is one of the posture detection sensors, and is configured to detect the rotation angle of the boom 4. In this embodiment, the boom angle sensor S1 is a stroke sensor that detects the stroke amount of the boom cylinder 7, and derives the rotation angle of the boom 4 around a boom foot pin that connects the upper rotating body 3 and the boom 4 based on the stroke amount of the boom cylinder 7.

[0029] The arm angle sensor S2 is one of the posture detection sensors, and is configured to detect the rotation angle of the arm 5. In this embodiment, the arm angle sensor S2 is a stroke sensor that detects the stroke amount of the arm cylinder 8, and derives the rotation angle of the arm 5 around the connecting pin that connects the boom 4 and the arm 5 based on the stroke amount of the arm cylinder 8.

[0030] The bucket angle sensor S3 is one of the posture detection sensors, and is configured to detect the rotation angle of the bucket 6. In this embodiment, the bucket angle sensor S3 is a stroke sensor that detects the stroke amount of the bucket cylinder 9, and derives the rotation angle of the bucket 6 around a connecting pin that connects the arm 5 and the bucket 6 based on the stroke amount of the bucket cylinder 9.

[0031] Each of the boom angle sensor S1, the arm angle sensor S2, and the bucket angle sensor S3 may be a rotary encoder, an acceleration sensor, a potentiometer (variable resistor), an inclination sensor, an inertial measurement unit, etc. The inertial measurement unit may be configured, for example, by combining an acceleration sensor and a gyro sensor.

[0032] The machine body inclination sensor S4 detects the inclination state of the machine body (upper rotating body 3 or lower running body 1) with respect to a horizontal plane. The machine body inclination sensor S4 is attached to, for example, the upper rotating body 3, and detects the inclination angles around two axes in the forward / backward and left / right directions (hereinafter, "fore / aft inclination angle" and "left / right inclination angle") of the shovel 100 (i.e., upper rotating body 3). The machine body inclination sensor S4 may include, for example, a rotary encoder, an acceleration sensor, a six-axis sensor, an IMU, etc. A detection signal corresponding to the inclination angle (fore / aft inclination angle and left / right inclination angle) by the machine body inclination sensor S4 is input to the controller 30.

[0033] The rotation angular velocity sensor S5 is configured to detect the rotation angular velocity of the upper rotating body 3. In this embodiment, the rotation angular velocity sensor S5 is a gyro sensor. The rotation angular velocity sensor S5 may be configured to calculate a rotation angle based on the rotation angular velocity. The rotation angular velocity sensor S5 may be configured with another sensor such as a rotary encoder.

[0034] The upper rotating body 3 is equipped with a cabin 10 as a driver's room, an engine 11, a positioning device PS, an imaging device S6, a communication device T1, etc. Also, a controller 30 is installed inside the cabin 10. Also, a driver's seat, operating devices, etc. are installed inside the cabin 10.

[0035] The engine 11 is a drive source of the excavator 100. In this embodiment, the engine 11 is a diesel engine. An output shaft of the engine 11 is connected to input shafts of the main pump 14 and the pilot pump 15.

[0036] The positioning device PS is configured to measure the position of the excavator 100. In this embodiment, the positioning device PS is a Global Navigation Satellite System (GNSS) compass, and is configured to be able to measure the position and orientation of the upper rotating body 3.

[0037] The imaging device S6 is configured to capture an image of a working area (the surroundings of the shovel 100) where the shovel 100 (an example of a work machine) is working. In this embodiment, the imaging device S6 includes a rear camera S6B attached to the rear end of the upper surface of the upper rotating body 3, a front camera S6F attached to the front end of the upper surface of the cabin 10, a left camera S6L attached to the left end of the upper surface of the upper rotating body 3, and a right camera S6R attached to the right end of the upper surface of the upper rotating body 3. The imaging device S6 may be a celestial sphere camera installed at a predetermined position in the cabin 10. The predetermined position is, for example, a position corresponding to the eye position of an operator seated in a driver's seat installed in the cabin 10.

[0038] Moreover, the imaging device S6 may be a stereo camera capable of detecting the distance to an object. Furthermore, a LIDAR (Light Detection and Ranging) may be installed near the imaging device S6 to detect the situation at the work site. For example, the LIDAR measures the distance between the LIDAR and one million or more points within the monitoring range. Note that this embodiment is not limited to a method using the LIDAR, and any spatial recognition device capable of measuring the distance to an object may be used. As another example, a distance measuring device such as a millimeter wave radar may be combined instead of the LIDAR.

[0039] The communication device T1 is configured to control communication with equipment external to the shovel 100. In this embodiment, the communication device T1 is configured to control wireless communication between the communication device T1 and equipment external to the shovel 100 via a wireless communication network. The communication device T1 includes, for example, a mobile communication module compatible with mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), and 5G (5th Generation), a satellite communication module for connecting to a satellite communication network, and the like.

[0040] The controller 30 is a calculation device that executes various calculations. The controller 30 is provided, for example, in the cabin 10 and controls the drive of the excavator 100. The functions of the controller 30 may be realized by any hardware, software, or a combination thereof. For example, the controller 30 is mainly configured with a microcomputer including a central processing unit (CPU), a memory device such as a random access memory (RAM), a non-volatile auxiliary storage device such as a read only memory (ROM), and various interface devices for input and output. The controller 30 realizes various functions by, for example, executing various programs installed in the non-volatile auxiliary storage device on the CPU.

[0041] Fig. 3 is a block diagram showing a configuration example of the drive system of the shovel 100 in Fig. 2. In Fig. 3, mechanical power transmission lines are indicated by double lines, hydraulic oil lines by thick solid lines, pilot lines by dashed lines, and electrical control lines by dotted lines.

[0042] The drive system of the excavator 100 is made up of an engine 11, a regulator 13, a main pump 14, a pilot pump 15, a control valve unit 17, a controller 30, and a proportional valve 31. The engine 11 is controlled by an engine control unit 74.

[0043] The main pump 14 supplies hydraulic oil to a control valve unit 17 via a hydraulic oil line 16. In this embodiment, the main pump 14 is a swash plate type variable displacement hydraulic pump.

[0044] The regulator 13 is configured to control the discharge amount of the main pump 14. In this embodiment, the regulator 13 is configured to adjust the swash plate tilt angle of the main pump 14 in response to the discharge pressure of the main pump 14 or a control signal from the controller 30. The discharge amount (displacement volume) of the main pump 14 per rotation is controlled by the regulator 13.

[0045] The pilot pump 15 is an example of a pilot pressure generating device, and is configured to be able to supply hydraulic oil to hydraulic control devices via a pilot line. In this embodiment, the pilot pump 15 is a fixed displacement hydraulic pump. However, the pilot pressure generating device may be realized by the main pump 14. That is, the main pump 14 may have a function of supplying hydraulic oil to various hydraulic control devices via a pilot line, in addition to a function of supplying hydraulic oil to the control valve unit 17 via a hydraulic oil line. In this case, the pilot pump 15 may be omitted.

[0046] The control valve unit 17 is configured to selectively supply hydraulic oil received from the main pump 14 to one or more hydraulic actuators. In this embodiment, the control valve unit 17 includes a plurality of control valves corresponding to the plurality of hydraulic actuators. The control valve unit 17 is configured to selectively supply hydraulic oil discharged from the main pump 14 to one or more hydraulic actuators. The hydraulic actuators include, for example, the boom cylinder 7, the arm cylinder 8, the bucket cylinder 9, the left traveling hydraulic motor 1L, the right traveling hydraulic motor 1R, and the swing hydraulic motor 2A.

[0047] The controller 30 is configured to control the proportional valve 31 based on an operation signal received through the communication device T1. In this embodiment, the operation signal is transmitted from a remote control room RC. The operation signal may be generated by an operation device provided in the cabin 10.

[0048] The proportional valve 31, which functions as a control valve for machine control, is disposed in a pipe connecting the pilot pump 15 and the pilot port of the control valve in the control valve unit 17, and is configured so as to be able to change the flow passage area of ​​the pipe. In this embodiment, the proportional valve 31 operates in response to a control command output by the controller 30. Therefore, the controller 30 can supply the hydraulic oil discharged by the pilot pump 15 to the pilot port of the control valve in the control valve unit 17 via the proportional valve 31, regardless of the operation of the operating device 26 by the operator.

[0049] In this embodiment, the controller 30 can control the pilot pressure acting on the pilot port of each control valve by individually controlling the opening area of ​​each of the multiple solenoid valves included in the proportional valve 31. Therefore, the controller 30 can control the flow rate of hydraulic oil flowing into each hydraulic actuator and the flow rate of hydraulic oil flowing out of each hydraulic actuator, and therefore can control the movement of each hydraulic actuator.

[0050] In this way, the controller 30 can raise and lower the boom 4, open and close the arm 5, open and close the bucket 6, rotate the upper rotating body 3, and cause the lower traveling body 1 to travel, in response to an operation signal from outside, such as a remote operation room RC.

[0051] Fig. 4 is a diagram showing an example of the configuration of an electrical system mounted on the excavator of Fig. 1. As shown in Fig. 4, the engine 11 is connected to an engine control unit 74. Various data indicating the state of the engine 11 is transmitted from the engine control unit 74 to the controller 30. Note that the remote control room RC is the same as that in Fig. 1, and therefore a description thereof will be omitted.

[0052] The battery 70 is configured to supply electric power to various electric loads mounted on the excavator 100. The alternator 11a (generator), the starter 11b, the controller 30, the electric components 72, etc. are configured to operate with electric power stored in the battery 70. The starter 11b is configured to be driven by the electric power stored in the battery 70 and start the engine 11. The battery 70 is also configured to be charged with electric power generated by the alternator 11a.

[0053] The water temperature sensor 11c transmits data related to the temperature of the engine coolant to the controller 30. The regulator 13 transmits data related to the swash plate tilt angle to the controller 30. The discharge pressure sensor 14b transmits data related to the discharge pressure of the main pump 14 to the controller 30. The positioning device PS transmits data related to the position of the excavator 100 to the controller 30.

[0054] An oil temperature sensor 14c is provided in a pipe 14-1 between the main pump 14 and a hydraulic oil tank in which the hydraulic oil to be sucked by the main pump 14 is stored. The oil temperature sensor 14c transmits data relating to the temperature of the hydraulic oil flowing through the pipe 14-1 to the controller 30.

[0055] A urea water remaining amount sensor 21a provided in the urea water tank 21 transmits data relating to the remaining amount of urea water to the controller 30. A fuel remaining amount sensor 22a provided in the fuel tank 22 transmits data to the controller 30 relating to the remaining amount of fuel.

[0056] The communication device T1 is configured to transmit and receive information to and from a communication device T2 installed in a remote control room RC via wireless communication. In this embodiment, the communication device T1 and the communication device T2 are configured to transmit and receive information via a fifth generation mobile communication line (5G line), an LTE line, a satellite line, or the like.

[0057] <Block configuration of remote control system> Fig. 5 is a functional block diagram showing a configuration example of the remote control system SYS according to this embodiment. In the example shown in Fig. 5, block configurations of the remote control room RC, the fixed point measuring devices C11 and C12, and the excavator 100 included in the remote control system SYS are shown.

[0058] The excavator 100 includes a positioning device PS, a controller 30, a proportional valve 31, an imaging device S6, and a communication device T1. The remote control room RC includes an operation sensor R29, a remote controller R30, a display device D1, and a communication device T2. The fixed point measurement devices C11 and C12 include a controller C30, an imaging device S16, a rotation mechanism Z1, and a communication device T3.

[0059] The following describes functions of the controller 30 mounted on the shovel 100. As shown in Fig. 5, the controller 30 has, as functional blocks, a communication control unit 301, a shovel state identification unit 302, and an actuator driving unit 303.

[0060] The communication control unit 301 controls the transmission and reception of information between the communication device T1 and an external device. For example, the communication control unit 301 controls the reception of an operation signal from the remote controller R30 using the communication device T1. As another example, the communication control unit 301 controls the transmission of image information captured by the imaging device S6 to the remote controller R30 using the communication device T1.

[0061] The shovel state identifying unit 302 is configured to identify the state of the shovel 100. In this embodiment, the state of the shovel 100 includes the position and orientation of the shovel 100. The position of the shovel 100 is, for example, the position of the shovel 100 in the world coordinate system (the latitude, longitude, and altitude of the shovel 100 reference point). The shovel state identifying unit 302 identifies the position and orientation of the shovel 100 based on the output of the positioning device PS.

[0062] Then, the communication control unit 301 performs control to transmit information indicating the identified position and orientation of the shovel 100 to the remote control room RC using the communication device T1.

[0063] The actuator driving unit 303 is configured to drive the actuator mounted on the excavator 100. In this embodiment, the actuator driving unit 303 generates and outputs an actuation signal for each of the multiple solenoid valves included in the proportional valve 31, based on an operation signal from the remote controller R30 that is received and controlled by the communication control unit 301.

[0064] Each solenoid valve that receives the actuation signal increases or decreases the pilot pressure acting on the pilot port of the corresponding control valve in the control valve unit 17. As a result, the hydraulic actuator corresponding to each control valve operates at a speed according to the stroke amount of the control valve.

[0065] In this embodiment, the communication control unit 301 transmits image information captured by the imaging device S6 to the remote control room RC in order to communicate the situation around the shovel 100. This allows the operator OP in the remote control room RC to recognize the situation around the shovel 100.

[0066] However, there are cases where the image information captured by the imaging device S6 does not allow recognition of an object in a blind spot as viewed from the shovel 100. Furthermore, there are cases where the operator OP in the remote control room RC can more easily recognize the situation around the shovel 100 from image information overlooking the shovel 100 than from image information captured by the imaging device S6. Therefore, in this embodiment, fixed point measurement devices C11 and C12 are installed at the work site.

[0067] Next, the functions of the controller C30 mounted on the fixed point measuring devices C11 and C12 will be described. As shown in Fig. 5, the controller C30 has a communication control unit 2301 and a control unit 2302 as functional blocks, and performs processing for transmitting image information captured by the imaging device S16 to the remote operation room RC.

[0068] The communication control unit 2301 controls the transmission and reception of information between the communication device T3 and an external device. For example, the communication control unit 2301 controls the reception of a signal from the remote controller R30 using the communication device T3. As another example, the communication control unit 301 controls the transmission of image information captured by the imaging device S16 to the remote controller R30 using the communication device T3.

[0069] The control unit 2302 controls the zoom mechanism S16Z and the rotation mechanism Z1 of the imaging device S16. The control unit 2302 can adjust the imaging area of ​​the imaging device S16 by controlling the zoom mechanism S16Z. In addition, the control unit 2302 can adjust the shooting direction of the imaging device S16 (the optical axis direction of the imaging device S16) by controlling the rotation mechanism Z1.

[0070] The control unit 2302 according to this embodiment may control the zoom mechanism S16Z and the rotation mechanism Z1 based on a signal transmitted from the remote control room RC. Furthermore, the control unit 2302 may detect the position of the shovel 100 based on image information captured by the imaging device S16, and control the zoom mechanism S16Z and the rotation mechanism Z1 so as to follow the movement of the shovel 100. Note that the control method for following the movement of the shovel 100 may be any method, regardless of whether it is a well-known method, such as performing object tracking processing using continuously captured image information.

[0071] Next, functions of the remote controller R30 installed in the remote operation room RC will be described. The remote controller R30 is configured to remotely operate the excavator 100.

[0072] The remote controller R30 (an example of a control device) is configured to receive image information including the shovel 100 and capturing an image of the surroundings of the shovel 100 by the fixed point measurement devices C11, C12 via the communication network NT, and to perform processing based on the received image information. In this embodiment, as an example of processing based on the received image information, an example of displaying the image information and detecting an object based on the image information will be described. Note that in this embodiment, the processing based on the received image information is not limited to displaying the image information and detecting an object based on the image information, and other processing may be performed.

[0073] The remote controller R30 also has, as its functional blocks, a display control unit 1301, a determination unit 1302, a distance calculation unit 1303, an object detection unit 1304, a communication control unit 1305, and an operation signal generation unit 1306.

[0074] The communication control unit 1305 controls transmission and reception of information between the communication device T2 and an external device. For example, the communication control unit 1305 controls reception of image information from the shovel 100 using the communication device T2. As another example, the communication control unit 1305 controls reception of image information from the fixed point measuring devices C11 and C12 using the communication device T2.

[0075] The display control unit 1301 controls the display of information on the display device D1. For example, the display control unit 1301 controls the display of one or more pieces of image information received from the shovel 100 and image information received from the fixed point measuring devices C11 and C12 on the display device D1.

[0076] The object detection unit 1304 detects an object appearing in the image information received from the shovel 100 and the image information received from the fixed point measuring devices C11 and C12. For example, the object detection unit 1304 may detect an object appearing in the image information by extracting the contour of the object appearing in the image information. Furthermore, the object detection unit 1304 may identify the type of the detected object based on a database that associates the shape of the object with the type of the object.

[0077] The distance calculation unit 1303 calculates the distance from the shovel 100 to the object detected by the object detection unit 1304. Any method may be used to calculate the distance. When a LIDAR is provided near the imaging device, the distance calculation unit 1303 may acquire the distance to the object detected by the LIDAR. Furthermore, the distance to the object may be calculated using multiple pieces of image information. The method will be described later.

[0078] The determination unit 1302 determines whether or not to issue a warning based on the distance between the object and the shovel 100 calculated by the distance calculation unit 1303. The criterion for issuing a warning may be any criterion, and may be, for example, based on whether or not an object exists within a predetermined range based on the shovel 100, whether or not an object exists in the direction in which the shovel 100 moves, or whether or not an object exists within a range in which an attachment moves when the shovel 100 turns.

[0079] Then, the display control unit 1301 may display warning information on the display device D1 when the determination unit 1302 determines that a warning should be issued. At that time, the detected object may be surrounded or colored to make it easier for the operator OP to check.

[0080] The operation signal generating unit 1306 is configured to generate an operation signal. In this embodiment, the operation signal generating unit 1306 is configured to receive a signal corresponding to an operation accepted by the operating device R26 from the operation sensor R29, and to generate an operation signal for operating the shovel 100 based on the received signal.

[0081] The communication control unit 1305 performs control to transmit the operation signal generated by the operation signal generating unit 1306 to the shovel 100 using the communication device T3.

[0082] <Exterior of the remote control room> Next, the remote control room RC will be described. Fig. 6 is a diagram showing an example of the layout of the remote control room RC. In the remote control room RC, a plurality of operating devices R26 are provided with respect to an operating seat DS.

[0083] In this embodiment, the display device D1 is a multi-display consisting of three monitors arranged in three vertical rows and one horizontal row, as shown in Fig. 6. Specifically, the display device D1 includes a central monitor D1a, an upper monitor D1b, and a lower monitor D1c.

[0084] Then, the display device D1 displays the received image information by the display control unit 1301. The operator OP can perform an operation for switching the image information to be displayed (for example, moving the viewpoint) in addition to operating the shovel 100 via the operation device R26.

[0085] <Example of display screen in remote control room> Next, an example of a screen displayed on the display device D1 will be described. Fig. 7 is a diagram showing an example of a screen displayed on the display device D1 according to this embodiment. In the example shown in Fig. 7, an example of a screen displayed on the upper monitor D1b and the central monitor D1a is shown, and the screen displayed on the lower monitor D1c is omitted.

[0086] The central monitor D1a displays image information received from the shovel 100. The captured image is captured by the imaging device S6. Specifically, the display area 701 of the central monitor D1a displays image information captured by the front camera S6F, the display area 702 displays image information captured by the left camera S6L, the display area 703 displays image information captured by the rear camera S6B, and the display area 704 displays image information captured by the right camera S6R. This allows the operator OP to check the situation around the shovel 100 from the viewpoint of the shovel 100, thereby improving safety.

[0087] Image information captured by any one of the fixed point measuring devices C11, C12 is displayed on the upper monitor D1b. That is, image information including the shovel 100 and capturing an image of the periphery of the shovel 100 is displayed on the upper monitor D1b. The image information is captured from a position higher than the shovel 100. Therefore, when the operator OP visually recognizes the image information, the operator OP can easily grasp the situation around the shovel 100 from a perspective looking down on the shovel 100.

[0088] That is, when operating the shovel 100, the operator OP in the remote control room RC according to this embodiment can check the surrounding situation from a viewpoint looking down from above the shovel 100, in addition to the viewpoint from the shovel 100. Depending on the situation of the shovel 100, it may be easier to grasp the situation of the shovel 100 from a viewpoint looking down from above the shovel 100 than from the viewpoint from the shovel 100.

[0089] For example, when referring to the display area 703 in FIG. 7, the operator can recognize that an obstacle 711 exists behind the shovel 100, but it may be difficult for the operator to grasp the distance to the obstacle 711.

[0090] In contrast to this, in this embodiment, image information showing an overview of the surrounding situation including the shovel 100, captured by any one of the fixed point measuring devices C11, C12, is displayed on the upper monitor D1b. When the operator OP refers to the screen, the operator OP can intuitively grasp the distance between the shovel 100 and the obstacle 711. In other words, since it becomes easier to grasp the surrounding situation of the shovel 100, it is possible to realize improved safety. In this way, the operator OP can grasp the surrounding situation including the obstacle 711, and improve the visibility of the surrounding situation of the shovel 100.

[0091] <Location of fixed point measurement devices> Next, an example of the arrangement of the fixed point measuring devices C11 and C12 will be described. Fig. 8 is a diagram illustrating an example of the arrangement relationship between the fixed point measuring device C11 and the shovel 100. As shown in Fig. 8, the fixed point measuring device C11 is provided at a higher position than the shovel 100 so as to be able to overlook the shovel 100. The angle of view of the fixed point measuring device C11 only needs to be able to image the surroundings including the shovel 100, and may be between 60° and 120°, for example. This allows the fixed point measuring device C11 to image not only the shovel 100, but also an obstacle 801 and the situation on the opposite side of the shovel.

[0092] The fixed point measuring device C11 may also include a rotation mechanism Z1 and rotate horizontally based on a support pillar. For example, the fixed point measuring device C11 may rotate using the rotation mechanism Z1 so that the shovel 100 is located near the center. The controller C30 of the fixed point measuring device C11 may control the rotation of the rotation mechanism Z1 in accordance with the position of the shovel 100 detected from image information (to track the shovel 100), or may control the rotation of the rotation mechanism Z1 based on a signal received from the shovel 100.

[0093] Furthermore, a zoom mechanism S16Z is provided in the imaging device S16 of the fixed point measurement device C11. This allows the controller C30 of the fixed point measurement device C11 to adjust the imaging range. For example, the controller C30 may control the zoom mechanism S16Z based on a signal received from the remote control room RC.

[0094] 8, the fixed point measurement device C11 has been described, but the fixed point measurement device C12 and the like have a similar configuration and description thereof will be omitted. In other words, by making it possible to capture images of the work site with the fixed point measurement devices C11 and C12, the operator OP can check the situation around the shovel 100 from a bird's-eye view by referring to the upper monitor D1b, no matter where the shovel 100 is located in the work site. Therefore, the shovel 100 can check the surroundings from an objective perspective from a high position, and the distance between the shovel 100 and surrounding objects can be intuitively grasped, thereby improving safety.

[0095] <Detecting object position> The remote controller R30 according to this embodiment detects whether or not an object exists around the shovel 100 based on image information captured by the fixed point measurement devices C11 and C12. Furthermore, it also calculates the distance between the shovel 100 and the object. Therefore, an example of a distance calculation method according to this embodiment will be described.

[0096] Fig. 9 is a diagram illustrating a method for calculating the distance between the shovel 100 and an obstacle 801. In the example shown in Fig. 9, the remote controller R30 calculates the distance between the shovel 100 and the obstacle 801 based on image information captured by the imaging device S16 of the fixed point measurement device C11 and image information captured by the rear camera S6B of the shovel 100.

[0097] The object detection unit 1304 of the remote controller R30 according to this embodiment detects an obstacle 801 from the image information received from the fixed point measurement device C11, and also detects the obstacle 801 from the image information captured by the rear camera S6B. The object detection unit 1304 also detects the shovel 100 from the image information received from the fixed point measurement device C11.

[0098] The distance calculation unit 1303 acquires information indicating the position and orientation of the fixed point measurement device C11 in the world coordinate system from the fixed point measurement device C11 via the communication control unit 1305. In addition, the distance calculation unit 1303 acquires information indicating the position and orientation of the shovel 100 in the world coordinate system from the shovel 100 via the communication control unit 1305. In addition, the distance calculation unit 1303 recognizes in advance the position and height at which the rear camera S6B of the shovel 100 is attached.

[0099] This allows the distance calculation unit 1303 to recognize the distance d1 between the imaging device S16 of the fixed point measurement device C11 and the rear camera S6B. The distance calculation unit 1303 can then determine the angle θ1 between the direction in which the shovel 100 exists and the direction in which the obstacle 801 exists, based on the positional relationship between the shovel 100 and the obstacle 801, relative to the fixed point measurement device C11, based on the position of the obstacle 801 shown in the image information of the rear camera S6B received from the shovel 100, and the positional relationship between the shovel 100 and the fixed point measurement device C11 in the world coordinate system, based on the angle θ2 between the direction in which the imaging device S16 of the fixed point measurement device C11 exists and the direction in which the obstacle 801 exists, relative to the rear camera S6B.

[0100] Therefore, the distance calculation unit 1303 can calculate the distance d2 between the shovel 100 and the obstacle 801 by the triangulation method. Furthermore, the distance calculation unit 1303 may recognize the shape, height, and the like of the obstacle 801 by calculating the distance for each feature point on the surface of the obstacle 801 by the above method.

[0101] This enables the determination unit 1302 to determine whether or not the obstacle 801 is close to the shovel 100, whether or not the attachment will come into contact with the obstacle 801 when the shovel 100 turns, and so on.

[0102] Then, the display control unit 1301 displays warning information based on the determination result of the determination unit 1302 on the display device D1.

[0103] In this way, the remote controller R30 according to this embodiment detects whether or not an object is present around the shovel 100 based on image information captured by the imaging devices S16 of the fixed point measurement devices C11, C12, and outputs warning information indicating the detection result. This allows the operator OP to recognize whether or not the shovel 100 will come into contact with an obstacle or the like when operating the shovel 100, thereby improving safety.

[0104] Note that this embodiment is not limited to outputting warning information, and the remote controller R30 may transmit an operation signal based on the detection result to the shovel 100 so that the shovel 100 does not come into contact with the detected object. The operation signal may be, for example, an emergency stop signal. This makes it possible to prevent the shovel 100 from coming into contact with the object, thereby improving safety.

[0105] Furthermore, the remote controller R30 may control whether or not the communication control unit 1305 permits transmission of an operation signal, depending on the result of the determination by the determination unit 1302. For example, when an operation is performed to move the shovel 100, if the determination unit 1302 determines that no object exists in the direction of movement, the communication control unit 1305 may permit transmission of an operation signal. In this manner, in this embodiment, if it is confirmed that remote operation of the shovel 100 is safe based on image information from the fixed point measuring devices C11, C12, transmission of an operation signal indicating the remote operation may be enabled.

[0106] In this manner, in this embodiment, by using the detection results of the imaging device S6 and the fixed point measuring devices C11 and C12 provided on the shovel 100, the situation around the shovel 100 can be confirmed from multiple viewpoints. Therefore, for example, even if an object is in a blind spot from one viewpoint, the object can be detected from the other viewpoint, thereby improving safety.

[0107] <Enlarged view> Next, operations on the screen displayed on the display device D1 will be described. Fig. 10 is a diagram showing an example of switching in response to operations on the screen displayed on the display device D1 according to this embodiment. The example shown in Fig. 10(a) shows an example of an initial screen displayed on the upper monitor D1b.

[0108] In the remote control room RC, the operator OP can perform an enlargement or reduction operation on the screen displayed on the upper monitor D1b. The communication control unit 1305 of the remote controller R30 uses the communication device T2 to transmit an enlargement or reduction operation signal to the fixed point measuring devices C11 and C12 in accordance with an operation received from an input device (not shown) provided in the remote control room RC.

[0109] The control unit 2302 of the controller C30 of the fixed point measurement devices C11 and C12 controls the zoom mechanism S16Z of the imaging device S16 based on the received enlargement or reduction operation signal. In this way, the fixed point measurement devices C11 and C12 are configured to be able to change the imaging area by including the zoom mechanism S16Z.

[0110] Thereafter, the communication control unit 2301 of the controller C30 transmits image information captured by the imaging device S16 after the control of the zoom mechanism S16Z has been performed to the remote controller R30 using the communication device T3.

[0111] FIG. 10(b) shows an example of a screen displayed on the upper monitor D1b based on image information transmitted from the fixed point measurement device C11 after an enlargement operation has been performed.

[0112] 10(b), an operation for adjusting the imaging direction of the fixed point measurement device C11 may be performed. In this way, when the controller C30 receives an operation signal indicating the operation, the controller C30 controls the rotation operation of the imaging device S16 by the rotation mechanism Z1 or the swiveling operation of the imaging device S16 (tilt adjustment operation of the imaging device S16 in the vertical direction).

[0113] This allows an area of ​​interest to the operator OP to be enlarged and displayed. Fig. 10(b) shows a screen on which the periphery of an obstacle 711 present in the vicinity of the shovel 100 is displayed by a zoom-in operation by the operator OP. In other words, the operator OP can enlarge or change the area displayed on the display device D1 by operating the operating device, making it easier for the operator OP to grasp the situation around the shovel 100. This makes it possible to improve safety.

[0114] Furthermore, it may be combined with other detection devices. For example, it is possible to provide a detection device at the entrance / exit of the work site. Then, when the control unit 2302 of the controller C30 of the fixed point measurement devices C11 and C12 receives a signal indicating that the detection device detects a person, a truck, or the like, the control unit 2302 controls so that the area showing the entrance / exit is enlarged. Then, the control unit 2302 of the controller C30 of the fixed point measurement devices C11 and C12 transmits image information in which the area showing the entrance / exit is enlarged to the remote controller R30. This allows the operator OP of the remote controller R30 to perform work while taking into account the surrounding conditions such as the entrance / exit, thereby improving work efficiency.

[0115] In this embodiment, image information captured by a fixed-point measuring device C11 provided separately from the shovel 100 is displayed on the display device D1. Therefore, the display device D1 can display an area that cannot be captured by the imaging device S6 of the shovel 100. This makes it possible to improve safety.

[0116] In this embodiment, the enlargement and reduction of the image information is not limited to a method using the zoom mechanism S16Z, etc. For example, the controller C30 may use a so-called digital zoom, etc., in which a part of the image information captured by the imaging device S16 is cut out and enlarged.

[0117] <Processing Procedure> Next, a process for displaying information on the display device D1 in the remote control room RC will be described. Fig. 11 is a sequence diagram showing a process for displaying information on the display device D1 in the remote control system for a shovel according to this embodiment.

[0118] First, the imaging device S6 of the shovel 100 captures an image of the surroundings of the shovel 100 and generates image information (S1101).

[0119] Then, the communication control unit 301 of the excavator 100 transmits image information captured by the imaging device S6 to the remote control room RC (S1102).

[0120] Meanwhile, the imaging devices S16 of the fixed point measuring devices C11 and C12 capture images of the surroundings of the shovel 100 and generate image information (S1111).

[0121] Then, the communication control unit 2301 of the fixed point measuring devices C11 and C12 transmits image information captured by the imaging device S16 to the remote control room RC (S1112).

[0122] The display control unit 1301 of the remote controller R30 in the remote operation room RC displays the received image information (of the shovel 100, and the fixed point measuring device C11 or C12) on the display device D1 (S1121).

[0123] The object detection unit 1304 of the remote controller R30 in the remote operation room RC detects objects present around the shovel 100 based on the image information received from the shovel 100 and the fixed point measurement devices C11 and C12 (S1122).

[0124] The distance calculation unit 1303 of the remote controller R30 in the remote operation room RC calculates the distance between the detected object and the shovel 100 based on the image information received from the shovel 100 and the fixed point measurement devices C11 and C12 (S1123).

[0125] Then, the determination unit 1302 determines whether or not it is necessary to issue warning information based on the distance between the detected object and the shovel 100. The criteria for issuing warning information include a case where the distance between the object and the shovel 100 is shorter than a predetermined reference distance, or a case where there is a possibility that the attachment of the shovel will come into contact with the object when the shovel 100 starts to swing.

[0126] When the determining unit 1302 determines that it is necessary to issue warning information, the display control unit 1301 of the remote controller R30 displays warning information for calling the operator OP's attention on the display device D1 (S1124).

[0127] In this embodiment, by performing the above-mentioned processing, the display device D1 displays the situation around the shovel 100 and warning information. This makes it easier for the operator OP to grasp the surrounding situation, thereby improving safety.

[0128] In this embodiment, image information of either one of the fixed point measuring devices C11, C12 is displayed on the display device D1 depending on the position of the shovel 100. Switching of image information displayed on the display device D1 will be described.

[0129] Fig. 12 is a diagram illustrating the movement of the shovel 100 when image information displayed on the display device D1 according to this embodiment is switched. In the example shown in Fig. 12, an imaging range 1211 of the fixed point measurement device C11 and an imaging range 1212 of the fixed point measurement device C12 are shown.

[0130] In this embodiment, the remote controller R30 stores an imaging range 1211 of the fixed point measuring device C11 and an imaging range 1212 of the fixed point measuring device C12 in the world coordinate system.

[0131] As a result, when the shovel 100 is present within the imaging range 1211, image information captured by the fixed point measuring device C11 is displayed on the display device D1.

[0132] When the shovel 100 moves in the direction of the arrow 1201, the remote controller R30 can recognize that the shovel 100 is moving from an imaging range 1211 to an imaging range 1212 from information transmitted from the shovel 100 (the position and orientation of the shovel 100).

[0133] Then, at the timing when the shovel 100 moves from the imaging range 1211 to the imaging range 1212, the display control unit 1301 switches the image information displayed on the display device D1 from the image information received from the fixed point measurement device C11 to the image information received from the fixed point measurement device C11.

[0134] As a result, regardless of the position of the shovel 100, image information showing an overview of the situation around the shovel 100, captured by the fixed point measuring devices C11 and C12, is displayed on the display device D1. This makes it easy for the operator OP to check the situation around the shovel 100.

[0135] In the above-described embodiment, an example has been described in which the fixed point measurement devices C11 and C12 transmit image information to the remote control room RC. However, in this embodiment, the transmission destination of the image information captured by the fixed point measurement devices C11 and C12 is not limited to the remote control room RC. For example, the transmission destination of the image information captured by the fixed point measurement devices C11 and C12 may be the excavator 100. Furthermore, when the work site is managed by a management device in a management center, the fixed point measurement devices C11 and C12 may transmit image information to the management device. This allows the management center to grasp the situation around the excavator at the work site.

[0136] In the present embodiment, an example has been described in which either one of the image information captured by the fixed point measuring devices C11 and C12 is displayed. However, the present embodiment is not limited to a method of displaying either one of the image information captured by the fixed point measuring devices C11 and C12. For example, the remote controller R30 may combine the image information captured by the fixed point measuring devices C11 and C12 and display the combined image information on the display device D1.

[0137] Also, the positional relationship of the surroundings including the shovel 100 and the fixed point measurement devices C11 and C12 may be displayed on the display device D1. Fig. 13 is a diagram showing an example of a screen displayed on the upper monitor D1b. In the screen shown in Fig. 13, a display area 1311 showing the positional relationship of the surroundings including the shovel 100 and the fixed point measurement devices C11 and C12 is displayed on the upper monitor D1b together with image information captured by the fixed point measurement device C11.

[0138] In the display area 1311, the shovel 100 and the fixed-point measurement devices C11 and C12 are displayed as images when viewed from above. An icon 100I indicates the shovel 100, an icon 711I indicates an icon of an obstacle 711, an icon C11I indicates the fixed-point measurement device C11, and an icon C12I indicates the fixed-point measurement device C12. In addition, the display area 1311 may display the imaging ranges 1211 and 1212 of the fixed-point measurement devices C11 and C12, respectively. For example, a dotted line 1211I extending from the icon C11I indicates the imaging range 1211, and a dotted line 1212I extending from the icon C12I indicates the imaging range 1212.

[0139] The upward direction of the display area 1311 is the imaging direction of the front camera S6F. That is, the imaging direction of the front camera S6F (the forward direction of the upper rotating body 3 of the shovel 100) is displayed as the upward direction of the display area 1311. As a result, when the upper rotating body 3 of the shovel 100 rotates, the display control unit 1301 performs display such that the display area 1311 rotates in accordance with the rotation.

[0140] Furthermore, the display control unit 1301 may perform enlargement or reduction of the display area 1311 in response to an enlargement operation or reduction operation received from the operator OP. Furthermore, the display control unit 1301 may set the range displayed as the display area 1311 (in other words, the size of the periphery of the displayed excavator 100) in response to a range setting received from the operator OP.

[0141] (Variation 1) In the above-described embodiment, the case where the shovel 100 is operated from the remote control room RC has been described. However, the above-described embodiment is not limited to the method of operating the shovel 100 from the remote control room RC.

[0142] For example, the fixed point measurement devices C11, C12 may transmit image information to the shovel 100, and the received image information may be displayed on a display device in the cabin 10 of the shovel 100. Also, the controller 30 of the shovel 100 may display warning information indicating the result of the detection of an obstacle based on the image information received from the fixed point measurement devices C11, C12. When operating the shovel 100, the situation around the shovel 100 can be recognized from a bird's-eye view, thereby improving safety.

[0143] Second Embodiment In the above embodiment, the case where the work machine is a shovel has been described. However, in this embodiment, a crane may be used as an example of the work machine. Therefore, in this embodiment, a case where a crawler crane is used as the work machine will be described.

[0144] FIG. 14 is a side view showing an example of a crawler crane and a fixed point measuring device included in the processing system according to this embodiment.

[0145] The crawler crane 100B comprises a lower running body 1B, an upper rotating body 3B mounted on the lower running body 1 so as to be freely rotatable via a rotating mechanism 2B, an attachment AT, a mast 5B, a backstop 6B, a hook HK, a counterweight 9B, and a cabin 10B.

[0146] The lower traveling body 1B includes, for example, a pair of left and right crawlers (not shown). The lower traveling body 1B causes the crawler crane 100B to travel by hydraulically driving each crawler by a left traveling hydraulic motor (not shown) and a right traveling hydraulic motor (not shown).

[0147] The upper rotating body 3B rotates relative to the lower traveling body 1B as a result of the rotating mechanism 2B being hydraulically driven.

[0148] The attachment AT (an example of a working device) includes a boom 4B and a main hoisting rope 7B.

[0149] The boom 4B is attached to the front center of the upper rotating body 3B so as to be able to rise and fall. A main hoisting rope 7B hangs down from the tip of the boom 4B, and a hook HK is attached to the tip of the main hoisting rope 7B. That is, the hook HK is attached to the tip of the boom 4B via the main hoisting rope 7B.

[0150] The main hoisting rope 7B has its base end attached to a main hoisting winch 7Ba that is attached to the rear portion between the base end and tip of the boom 4B, and its tip is attached to the hook HK. The main hoisting rope 7B can be wound up and unwound by the main hoisting winch 7Ba that is hydraulically driven by a main hoisting hydraulic motor (not shown), thereby raising and lowering the hook HK.

[0151] The mast 5B is attached to the upper rotating body 3B slightly rearward of the base end of the boom 4B so as to be rotatable about a rotation axis parallel to the rotation axis of the boom 4B. The tip of the mast 5B is connected to the tip of the boom 4B via a pendant rope 5Ba, and the boom 4B is raised and lowered via the mast 5B by winding in and out the boom hoist rope 5Bb using the hydraulically driven boom hoist winch 5Bc.

[0152] The backstop 6B has a base end attached to a portion of the upper rotating body 3B rearward of the base end of the boom 4B so as to be rotatable about a rotation axis parallel to the rotation axis of the boom 4B, and a tip end attached to a rear portion between the base end and tip end of the boom 4B so as to be rotatable about a rotation axis parallel to the rotation axis of the boom 4B. The backstop 6B expands and contracts in response to the raising and lowering operation of the boom 4B, and has a function of supporting the boom 4B from the rear when the boom 4B is in a substantially upright state, for example.

[0153] The hook HK is attached to the end of the main hoisting rope 7B and is used to suspend a load.

[0154] The counterweight 9B is provided at the rear end of the upper rotating body 3B and has the function of balancing the weight of the boom 4B and the suspended load SL.

[0155] The cabin 10B is attached, for example, to the right front end portion of the upper rotating body 3B. A cockpit and operating devices for operating various actuators are provided inside the cabin 10B.

[0156] The imaging device S26 is provided on the upper surface of the counterweight 9B and monitors the surroundings of the crawler crane 100B. In this case, the crawler crane 100B transmits image information of the surroundings of the crawler crane 100B based on the output of the imaging device S26 described below to an external device, and the image information may be displayed on a display device provided in the external device (for example, a remote control room or a management device).

[0157] In this embodiment, the position of the imaging device S26 arranged on the crawler crane 100B is not limited to the example shown in Fig. 14. The imaging devices S26L and S26R may be provided somewhere other than on the counterweight 9B, for example, on the upper rotating body 3B. The imaging device S26B may also be provided somewhere other than on the counterweight 9B, for example, at the bottom of the counterweight 9B. This reduces the burden of installing the imaging device S26 in response to changes in the number of counterweights 9B.

[0158] The imaging device S26 is provided on the upper surface of the counterweight 9B, and captures images of the surroundings of the crawler crane 100B from areas relatively close to the crawler crane 100B to areas relatively far from the crawler crane 100B to obtain image information. The imaging device S26 includes a rear camera S26B, a left camera S26L, a right camera S26R, and a front camera S26F. The imaging device S26 mounted on the crawler crane 100B is similar to the imaging device S6 mounted on the above-mentioned excavator 100, and a description thereof will be omitted.

[0159] The rear camera S26B, left camera S26L, and right camera S26R are attached to the upper rear end, upper left end, and upper right end of the counterweight 9B, respectively, and capture images of the rear, left side, and right side of the crawler crane 100B.

[0160] The crawler crane 100B is also provided with a front camera S26F to detect objects present in front of it. The front camera S26F may be attached at any position, for example, on the top surface of the counterweight 9B or near the front side of the upper rotating body 3B of the crawler crane 100B. The front camera S26F captures an image of the area in front of the crawler crane 100B.

[0161] The rear camera S26B captures an image of a detection range behind the counterweight 9B, for example, a detection range in the horizontal direction from the left rear to the right rear (i.e., a circumferential direction as seen from the crawler crane 100B). The left camera S26L captures an image of a detection range on the left side of the counterweight 9B, for example, a detection range in the horizontal direction from the left front to the left rear of the counterweight 9B (i.e., a circumferential direction as seen from the crawler crane 100B). The right camera S26R captures an image of a detection range on the right side of the counterweight 9B, for example, a detection range in the horizontal direction from the right front to the right rear of the counterweight 9B (i.e., a circumferential direction as seen from the crawler crane 100B). The front camera S26F captures an image of a detection range in front of the upper rotating body 3B of the crawler crane 100B, for example, a detection range in the horizontal direction from the left front to the right front (i.e., a circumferential direction as seen from the crawler crane 100B). Furthermore, the imaging device S26 is attached to the upper part of the upper rotating body 3B with its optical axis facing diagonally downward, and captures images of a detection range in the vertical direction including the ground near the crawler crane 100B to the far side of the crawler crane 100B.

[0162] The imaging device S26 generates image information at a predetermined period (e.g., 1 / 30 seconds) from when the crawler crane 100B is started (i.e., key switch ON) to when it is stopped (i.e., key switch OFF). The image information generated by the imaging device S26 is taken into the controller 30. In addition, the three-dimensional image information output from the imaging device S26 is transmitted from the controller 30 to an external device (e.g., remote control room RC) via the communication device T1.

[0163] As shown in FIG. 14, in this embodiment, a fixed point measuring device C21 is disposed to capture images of the periphery of a work area where a crawler crane 100B hoists a load.

[0164] 14, the fixed point measuring device C21 takes images from a higher position compared to the state in which the crawler crane 100B is hoisting a load. The angle of view of the fixed point measuring device C21 may be between 60° and 120°, for example, as long as it can capture an image of the surroundings including the crawler crane 100B.

[0165] The fixed point measurement device C21 may also include a rotation mechanism Z21 and rotate horizontally based on the support. For example, the fixed point measurement device C21 may rotate using the rotation mechanism Z1 so that an image can be taken with the hook HK or the load suspended by the hook HK at the center. A controller (not shown) of the fixed point measurement device C21 may control the rotation of the rotation mechanism Z21 in accordance with the position of the hook HK or the load detected from the image information (to follow the load), or may control the rotation of the rotation mechanism Z21 based on a received signal.

[0166] Furthermore, a zoom mechanism may be provided within the imaging device of the fixed point measurement device C21 to make the imaging range adjustable.

[0167] The crawler crane 100B may be operated in an operation room within the cabin 10B, or may be operated in a remote operation room RC as in the above-described embodiment.

[0168] In this embodiment, the image information captured by the fixed point measuring device C21 is displayed on a display device in the control room in the cabin 10B or in the remote control room RC together with the image information captured by the imaging device S26.

[0169] On the display device, the image information captured by the fixed point measuring device C21 shows the situation around the crawler crane 100B along with a part of the crawler crane 100B (for example, the hook HK). Therefore, when the image information captured by the fixed point measuring device C21 is displayed, the operator can recognize the situation around the crawler crane 100B.

[0170] In this embodiment, a processing system including a crawler crane 100B, which is one aspect of a crane, has been described as an example of a work machine. However, in this embodiment, the crane included in the processing system is not limited to a crawler crane. For example, the present invention may be applied to one or more of a jib crane and a retractable crane. In other words, when one or more of a jib crane or a retractable crane are installed at a port wharf, a shipyard, etc., a fixed-point measuring device may be separately installed so that the surrounding situation including the jib crane or the retractable crane can be confirmed. When operating the jib crane or the retractable crane, image information captured by the fixed-point measuring device may be referred to, or the operation may be performed while paying attention to a warning based on image information captured by the fixed-point measuring device.

[0171] Furthermore, the present invention may be applied to one or more of a bridge crane and a gantry crane. When a bridge crane or a gantry crane is installed, a fixed point measurement device may be provided separately so that the surrounding situation including the bridge crane or the gantry crane can be confirmed. The fixed point measurement device may be equipped with a rotation mechanism or a zoom mechanism so that it can capture an image of the area including the bridge crane or the gantry crane even when the bridge crane or the gantry crane moves on a traveling rail. When operating the bridge crane or the gantry crane, the image information captured by the fixed point measurement device may be referred to, or the operation may be performed while heeding a warning based on the image information captured by the fixed point measurement device.

[0172] Furthermore, it may be applied to an overhead crane. When the overhead crane is installed indoors, the fixed point measurement device may be installed indoors. The fixed point measurement device may be equipped with a rotation mechanism or a zoom mechanism so that it can capture an image of an area including the overhead crane even when the overhead crane moves. When operating the overhead crane, the image information captured by the fixed point measurement device may be referred to, or the operation may be performed while heeding a warning based on the image information captured by the fixed point measurement device.

[0173] In this way, it is sufficient that at least a part of the work machine is included in the imaging range of the fixed point measurement device. By providing the above-mentioned configuration in this embodiment, even when the work machine is a crane, it is possible to obtain the same effects as in the first embodiment.

[0174] (Third embodiment) In the above-described embodiment, the working machine is a shovel or a crane. However, the above-described embodiment does not limit the working machine to a working machine such as a shovel or a crane. Therefore, in this embodiment, a case where an unloader is used as an example of a working machine will be described.

[0175] <Configuration of continuous unloader> FIG. 15 is a side view showing an example of a continuous unloader and a fixed-point measuring device included in the processing system according to this embodiment.

[0176] As shown in Fig. 15, a continuous unloader 100C (an example of a work machine) is a so-called bucket elevator type that is installed at a quay QY and continuously carries out (unloads) onto land bulk cargo M in a hold HD of a ship SP that is berthed at the quay QY. The bulk cargo is, for example, coal, coke, iron ore, etc.

[0177] The quay QY is constructed of, for example, reinforced concrete, and two rails 53 are installed on the quay QY in a direction parallel to the extension of the quay QY, i.e., the longitudinal direction of the ship SP to be docked. The continuous unloader 100C is configured to be movable on the two rails 53, and unloads from the ship SP while stopped at a predetermined position.

[0178] The continuous unloader 100C includes a traveling section 51, a rotating frame 55, a boom 57, a bucket elevator 59, and a cab 66.

[0179] The traveling section 51 is configured to be movable on two rails 53 of the quay QY.

[0180] The rotating frame 55 is mounted on the running part 51 so as to be capable of rotating.

[0181] The boom 57 is provided to extend forward from the revolving frame 55 to the portion on the sea where the ship SP is to be docked from the quay QY, and is configured to be capable of being raised and lowered relative to the revolving frame 55. Specifically, the boom 57 can be raised and lowered in response to the extension and contraction of a cylinder 65 attached between the boom 57 and the revolving frame 55.

[0182] The bucket elevator 59 is provided at the tip of the boom 57 so as to extend downward, i.e., toward the ship SP (hold HD). The bucket elevator 59 has a scraping part 61 at its tip, and bulk cargo scraped by the scraping part 61 is transported upward by a bucket (not shown) and landed on land.

[0183] The scraping unit 61 can change its angle of inclination in the fore-aft direction with respect to the elevator body 64. For example, when scraping off loose cargo directly below the opening OPE of the hold HD, the scraping unit 61 is used in a state where it is not inclined in the fore-aft direction with respect to the elevator body 64. On the other hand, when scraping off loose cargo that is present on the inner wall side of the opening OPE of the hold HD, the scraping unit 61 is used in a state where it is inclined forward with respect to the elevator body 64.

[0184] A parallel link 58 is provided between the revolving frame 55 and the bucket elevator 59, and the action of the parallel link 58 allows the bucket elevator 59 to maintain a vertical state regardless of the hoisting angle of the boom 57. The bucket elevator 59 can also move up and down in response to the up and down hoisting of the boom 57. A counterweight 63 is supported on the revolving frame 55 via a link extending rearward on the opposite side to the boom 57, and a balancing lever 62 is provided to connect the counterweight 63 and the boom 57. This allows the load to be balanced between the bucket elevator 59 and the counterweight 63.

[0185] The operator's cab 66 is provided at the front of the revolving frame 55 (that is, at the position in the direction in which the boom 57 extends), and is used for an operator to sit in and operate the continuous unloader 100C.

[0186] The cab 66 is equipped with, for example, a control device 110. The cab 66 is also provided with a display device 140 and an input device 150. The control device 110 may receive image information from the fixed point measurement device C31. The control device 110 may use the received image information to perform the same processing as the remote controller R30 described in the above embodiment.

[0187] When the continuous unloader 100C is used to unload bulk cargo, an operator is placed in the cab 66, and a worker is placed inside the hold HD to inform the operator of the status of the bulk cargo. The operator in the cab 66 and the worker in the hold HD work together to unload bulk cargo using the continuous unloader 100C.

[0188] The operator in the driver's cab 66 can operate the continuous unloader 100C while checking the image information displayed on the display device 140, which shows the opening OPE of the hold HD and the condition inside it, as well as the contents of communications from workers about the condition inside the hold HD.

[0189] The bucket elevator 59 includes an elevator body 64 that is provided to extend in the vertical direction, and a chain bucket (not shown) that revolves between an upper portion 59a and a lower portion (scraping portion 61) of the bucket elevator 59. The chain bucket includes a pair of roller chains and a plurality of buckets supported so as to be suspended from the pair of roller chains. The roller chain revolves in the direction of arrow W relative to the elevator body 64. As a result, the chain bucket circulates while moving in a circle between the upper portion 59a of the bucket elevator 59 and the scraping portion 61.

[0190] The bucket scrapes the bulk goods into its interior through the opening at the scraping section 61. The bucket, having scraped and stored the bulk goods, rises with its opening facing upward as the roller chain rises. The bulk goods inside the bucket that arrive at the upper part 59a of the bucket elevator 59 are sent from the discharge chute to a rotary feeder 87 provided on the outer periphery of the bucket elevator 59.

[0191] The rotary feeder 87 transports the bulk material sent from the bucket through a discharge chute to a boom conveyor 89 provided on the boom 57 .

[0192] The boom conveyor 89 is disposed inside the boom 57. Bulk goods on the rotary feeder 87 of the bucket elevator 59 are transferred to the boom conveyor 89 and transported toward the revolving frame 55. A hopper is provided at the end of the boom conveyor 89 on the revolving frame 55 side, and the bulk goods transported by the boom conveyor 89 are supplied to a belt conveyor 93 through the hopper.

[0193] The belt conveyor 93 is provided on the traveling section 51. The belt conveyor 93 transports the bulk goods to the ground belt conveyor 95. As a result, the bulk goods are carried out to the ground equipment 99 via the ground belt conveyor 95.

[0194] The continuous unloader 100C is also provided with imaging devices S36A, S36B, S36C, and S36D. The imaging direction of the imaging devices S36A, S36B, S36C, and S36D is downward. This allows the status of the bulk cargo directly below the opening OPE of the hold HD to be confirmed. However, since the continuous unloader 100C is large, it is preferable to be able to confirm the surrounding status other than the bulk cargo. Image information captured by the imaging devices S36A, S36B, S36C, and S36D may be transmitted to the control device 110. This allows the control device 110 to display the received image information on the display device 120.

[0195] In the processing system according to this embodiment, in addition to the continuous unloader 100C, a fixed point measuring device C31 is provided.

[0196] As shown in FIG. 15, in this embodiment, a fixed-point measuring device C31 is disposed to capture an image of the periphery of the continuous unloader 100C.

[0197] 15, the fixed-point measurement device C31 takes images from a position higher than the opening OPE of the hold HD. The angle of view of the fixed-point measurement device C31 may be between 60° and 120°, for example, as long as it can capture an image of the surroundings including the bulk cargo transported by the continuous unloader 100C.

[0198] A zoom mechanism may be provided in the imaging device of the fixed point measurement device C31 to adjust the imaging range, so that the fixed point measurement device C31 may capture an image of the interior of the hold HD.

[0199] The continuous unloader 100C may be operated from the operator's cab 66 or from the remote control room RC shown in the above-described embodiment.

[0200] In this embodiment, image information captured by the imaging devices S36A, S36B, S36C, and S36D, and image information captured by the fixed point measurement device C31 are displayed on the display device 140 in the driver's cab 66 or the display device D1 in the remote control room RC.

[0201] The image information captured by the fixed-point measurement device C31 shows the situation inside the opening OPE of the hold HD, the transport situation of the bulk cargo, or the situation around the continuous unloader 100C. Therefore, when the image information captured by the fixed-point measurement device C31 is displayed, the operator can recognize various situations related to the continuous unloader 100C.

[0202] That is, in this embodiment, when an operator operates the continuous unloader 100C, the operator can check the working status of the continuous unloader 100C and the status around the continuous unloader 100C by referring to the image information captured by the imaging devices S36A, S36B, S36C, and S36D and the image information captured by the fixed point measuring device C31. This can improve safety.

[0203] In this embodiment, the case where the work machine is the continuous unloader 100C has been described. In other words, a structure used for large-scale transportation such as the continuous unloader shown in this embodiment may be used as the work machine. As other examples of work machines, the present invention may be applied to a stacker or a reclaimer, or to a belt conveyor device. Furthermore, the present invention may be applied to a work ship for performing work underwater, etc.

[0204] Furthermore, the above-mentioned embodiments are not limited to a method using a fixed-point measuring device, but may be realized by reusing a monitoring device or the like already installed at a work site such as a port or a factory as an imaging device, and transmitting image information captured by the monitoring device or the like to the controller of the above-mentioned work machine, a controller in a remote control room, or a management device, etc.

[0205] <effect> In the above-described embodiment and modified examples, a fixed-point measuring device (an example of an imaging device) is configured to receive image information including the work machine and capturing an image of the surroundings of the work machine via a network and to perform processing based on the received image information, so that the situation around the work machine can be confirmed based on image information capturing an image of the surroundings of the work machine from a viewpoint other than the work machine, thereby achieving improved safety.

[0206] For example, the operator OP in the remote control room RC can recognize the situation around the work machine from a viewpoint different from that of the work machine, and can therefore operate the work machine while taking into account objects that are in the blind spot of the work machine, thereby improving safety.

[0207] Although the above describes embodiments using a shovel, a crawler crane, a continuous unloader, etc. as examples of a work machine, the present invention is not limited to the above-mentioned embodiments. Various changes, modifications, substitutions, additions, deletions, and combinations are possible within the scope of the claims. Naturally, these also fall within the technical scope of the present invention.

[0208] That is, in the above-described embodiment, a case has been described in which the information processing system is applied to a system that remotely controls or operates a work machine as an example of the information processing system for a work machine, but the information processing system may also be applied to systems of other types. [Explanation of symbols]

[0209] 100 Shovel 100B Crawler Crane 100C Continuous Unloader C11, C12 fixed point measuring device T1, T2, T3 Communications Equipment S6, S26, S36 imaging device 1 Undercarriage 2 Swivel mechanism 3. Upper rotating body 4. Boom 5 Arm 6 Bucket 7 Boom cylinder 8 Arm Cylinder 9 Bucket Cylinder 11 Engine 30 Controller RC Remote Control Room R26 operating device R29 Operation Sensor R30 Remote Controller 1301 Display control unit 1302 Judgment section 1303 Distance calculation unit 1304 Object detection unit 1305 Communication Control Unit 1306 Operation signal generation section D1 display device

Claims

1. A work machine that performs work at a work site; an imaging device provided separately from the work machine for imaging the work site; a control device configured to receive image information captured by the imaging device, including the work machine, and the surroundings of the work machine, via a network, and to perform processing based on the received image information; An information processing system for a work machine comprising:

2. The control device is configured to receive a signal from an operation device, and to transmit to the work machine an operation signal for operating the work machine based on the received signal.

2. An information processing system for a work machine according to claim 1.

3. The control device is configured to display the image information on a display device.

3. An information processing system for a work machine according to claim 1 or 2.

4. the control device is configured to transmit a control signal for changing an imaging range to the imaging device in accordance with an operation received from an operation device; The imaging device is configured to be able to change an imaging area based on the control signal.

4. An information processing system for a work machine according to claim 3.

5. The control device is configured to detect whether or not an object is present around the work machine based on the image information, and output a detection result, or transmit an operation signal based on the detection result to the work machine.

2. An information processing system for a work machine according to claim 1.

6. The imaging device is provided so as to be able to overlook the work machine from a position higher than the work machine.

2. An information processing system for a work machine according to claim 1.

Citation Information

Patent Citations

  • Remote control assistance device and remote control assistance system

    JP2022055922A

  • Remote operation support system and remote operation support device

    JP2022179081A

  • Shovel

    JP2020183624A