Remote information acquisition system for industrial machinery

Abstract

To reduce a load on a network.SOLUTION: A remote information acquisition system of a work machine, comprises: a first device that specifies a position and a type of an article existed in a work area from measurement information generated by a space recognition device that measures the work area where the work machine operates, and transmits the position and the type; and a second device constructed so as to perform a remote operation of the work machine. The second device is constructed so that a model corresponded to the type is arranged at coordinates corresponded to the position in a vertical three-dimensional space for displaying the position and the type on a screen when the position and the type are received.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to a remote information acquisition system for a working machine. Mu Related.

Background Art

[0002] Conventionally, a remote operation system for assisting remote operation of a working machine such as a hydraulic excavator has been known (see Patent Document 1). This remote operation system is configured to display an image captured by a camera attached to the working machine at the work site on a display device installed in the remote operation room.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the remote operation system described in Patent Document 1, in order to display the image captured by the camera on the display device, it is necessary to transmit the image from the working machine to the remote operation room via a communication network. Therefore, when the data amount of the image is large compared to the communication speed of the communication network, it may be difficult to display the image captured by the camera in the remote operation room.

[0005] In view of the above, the burden on the network is reduced by reducing the communication amount between the camera to be imaged and the device for confirming the imaged area.

Means for Solving the Problems

[0006] A remote information acquisition system for a working machine according to an aspect of the present invention includes the position and type of an object existing in the work area from measurement information generated by a space recognition device that measures the work area where the working machine is working, and 3D size andIdentify the location and type 3D size and The system comprises a first device that transmits and a second device configured to remotely control the work machine, wherein the second device is The system further includes a memory unit that stores a three-dimensional shape model corresponding to the aforementioned type. The aforementioned position and the aforementioned type The aforementioned three-dimensional size and When a signal is received, a virtual 3D space for displaying it on the screen will be created corresponding to the aforementioned type. The three-dimensional shape Model Based on the received three-dimensional size, It is configured to be positioned at coordinates corresponding to the aforementioned position. [Effects of the Invention]

[0007] According to one aspect of the present invention, the burden on the network is reduced by reducing the amount of data. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a schematic diagram showing an example of a remote control system according to the first embodiment. [Figure 2] Figure 2 is a side view showing a shovel (excavator) according to the first embodiment. [Figure 3] Figure 3 is a block diagram showing an example of the configuration of the drive system of an excavator according to the first embodiment. [Figure 4] Figure 4 shows an example of the electrical system configuration installed in the excavator according to the first embodiment. [Figure 5] Figure 5 is a functional block diagram showing an example configuration of a remote control system according to the first embodiment. [Figure 6] Figure 6 shows an example of the layout of a remote control room according to the first embodiment. [Figure 7] Figure 7 shows the space to be detected by the spatial recognition device installed on the shovel according to the first embodiment. [Figure 8] Figure 8 shows an object detected by the object detection controller according to the first embodiment. [Figure 9] Figure 9 is a diagram showing the size, inclination, and position of an object identified by the arrangement unit according to the first embodiment, represented as a rectangular parallelepiped. [Figure 10] FIG. 10 is a diagram illustrating a display screen of a display device displayed by a display control unit according to the first embodiment. [Figure 11] FIG. 11 is a diagram showing a work site of a crawler crane according to a modified example. [Figure 12] FIG. 12 is a diagram showing an example of a screen displayed on a display device in a remote operation room according to a modified example.

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiments described below are illustrative and do not limit the invention, and not all features and combinations thereof described in the embodiments are necessarily essential to the invention. In each drawing, the same or corresponding components are denoted by the same or corresponding reference numerals, and the description thereof may be omitted.

[0010] (First Embodiment) First, referring to FIG. 1, an overview of a remote operation system SYS (an example of a remote information acquisition system) according to the first embodiment will be described. FIG. 1 is a schematic diagram showing an example of the remote operation system SYS according to the first embodiment.

[0011] <Devices Constituting the Remote Operation System> As shown in FIG. 1, the remote operation system SYS according to the first embodiment includes an excavator 100, a drone DR, a fixed-point camera (an example of a fixed-point measurement device) C12, and a remote operation room RC.

[0012] Wireless communication between the excavator 100 and the drone DR is enabled. Then, the excavator 100 and the drone DR can transmit and receive data to and from a device (e.g., the remote operation room RC) connected to the public network NT via the wireless communication device NT1. The fixed-point camera C12 can transmit and receive data to and from a device (e.g., the remote operation room RC) connected to the public network NT.

[0013] The excavator 100, the drone DR, and the fixed-point camera C12 are present at the work site where the excavator 100 performs work. Thus, in this embodiment, a plurality of types of devices are provided at the work site. And the excavator 100, the drone DR, and the fixed-point camera (an example of a fixed-point measuring device) C12 can transmit information regarding the work site to the remote operation room RC. Thereby, the remote operation room RC can comprehensively confirm the work site according to the detection results from the plural types of devices. Note that this embodiment does not limit the device (the first device) for transmitting the measurement information to the excavator 100, the drone DR, and the fixed-point camera C12, and other types of devices such as a spatial recognition device that can be carried by the user may also be used.

[0014] For example, the excavator 100 is provided with a spatial recognition device C1 (see FIG. 2), and the drone DR is provided with a spatial recognition device C11. The fixed-point camera C12 has a function as a spatial recognition device. Therefore, the excavator 100, the drone DR, and the fixed-point camera C12 can generate measurement information indicating the measurement results of the work site by the spatial recognition device. Transmitting the measurement information, which is the measurement result of each of the excavator 100, the drone DR, and the fixed-point camera C12, to the remote operation room RC will increase the load on the public network NT. Therefore, the remote operation system SYS according to this embodiment transmits information regarding the object detected based on the measurement information instead of the measurement information.

[0015] The spatial recognition devices C1, C11, and the fixed-point camera C12 may use LIDAR (Light Detection and Ranging) to image the work site. LIDAR measures, for example, the distance between more than one million points within the monitoring range and the LIDAR. Note that this embodiment does not limit the method using LIDAR, and any spatial recognition device capable of measuring the distance to an object may be used. Further, this embodiment may be combined with an imaging device capable of detecting color or the like. As another example, a stereo camera may be used, or a distance measuring device such as a millimeter-wave radar may be combined.

[0016] The drone DR included in the remote control system SYS may be one unit or multiple units. This allows the remote control system SYS to provide information about the work site to the remote control room RC through multiple drone DRs.

[0017] The fixed-point camera C12 included in the remote control system SYS may be one unit or multiple units. This allows the remote control system SYS to provide information about the work site to the remote control room RC through multiple C12 units.

[0018] The scouring 100 included in the remote control system SYS may be one unit or multiple units. This allows the remote control system SYS to provide information about the work site to the remote control room RC through multiple scouring 100 units.

[0019] <Example of remote control room configuration> The remote control room RC is equipped with a communication device T2, a remote controller 40, an operating device 26, an operating sensor 29, and a display device D1. The remote control room RC also has an operating seat DS where the operator OP sits to remotely control the shovel 100.

[0020] The communication device T2 is configured to control communication between the communication device T1 attached to the shovel 100 (see Figure 2), the communication device T3 attached to the drone DR, and the communication device (not shown) attached to the fixed-point camera C12.

[0021] The remote controller (an example of a second device) 40 is an arithmetic unit that performs various calculations. In this embodiment, the remote controller 40 is composed of a microcomputer including a CPU and memory. The various functions of the remote controller 40 are realized by the CPU executing a program stored in memory.

[0022] Display device D1 displays a screen based on information transmitted from the shovel 100, drone DR, and fixed-point camera C12, allowing the operator OP in the remote control room RC to visually check the area around the shovel 100. Display device D1 allows the operator to check the conditions of the work site, including the area around the shovel 100, even though the operator is in the remote control room RC.

[0023] An operating device 26 (an example of an operating unit) is equipped with an operating sensor 29 for detecting the operation of the operating device 26. The operating sensor 29 is, for example, a tilt sensor that detects the tilt angle of the operating lever, or an angle sensor that detects the oscillation angle of the operating lever around its pivot axis. The operating sensor 29 may also consist of other sensors such as a pressure sensor, a current sensor, a voltage sensor, or a distance sensor. The operating sensor 29 outputs information regarding the detected operation of the operating device 26 to the remote controller 40. The remote controller 40 generates an operation signal based on the received information and transmits the generated operation signal to the shovel 100. The operating sensor 29 may also be configured to generate an operation signal. In this case, the operating sensor 29 may output the operation signal to the communication device T2 without going through the remote controller 40. This enables remote control of the shovel 100 from the remote control room RC.

[0024] <Shovel configuration> Figure 2 shows a construction machine, an excavator 100, according to an embodiment. An upper slewing body 3 is rotatably mounted on the lower traveling body 1 of the excavator 100 via a slewing mechanism 2. A boom 4 is attached to the upper slewing body 3. An arm 5 is attached to the tip of the boom 4, and a bucket 6 is attached to the tip of the arm 5 as an end attachment.

[0025] The boom 4, arm 5, and bucket 6 constitute an excavation attachment, which is an example of an attachment. The boom 4 is driven by the boom cylinder 7, the arm 5 is driven by the arm cylinder 8, and the bucket 6 is driven by the bucket cylinder 9.

[0026] A boom angle sensor S1 is attached to boom 4, an arm angle sensor S2 is attached to arm 5, and a bucket angle sensor S3 is attached to the bucket link. A slewing angular velocity sensor S4 is attached to the upper slewing body 3.

[0027] The boom angle sensor S1 is one of the attitude detection sensors and is configured to detect the rotation angle of the boom 4. In this embodiment, the boom angle sensor S1 is a stroke sensor that detects the stroke amount of the boom cylinder 7 and derives the rotation angle of the boom 4 around the boom foot pin that connects the upper slewing body 3 and the boom 4 based on the stroke amount of the boom cylinder 7.

[0028] The arm angle sensor S2 is one of the attitude detection sensors and is configured to detect the rotation angle of the arm 5. In this embodiment, the arm angle sensor S2 is a stroke sensor that detects the stroke amount of the arm cylinder 8 and derives the rotation angle of the arm 5 around the connecting pin that connects the boom 4 and the arm 5 based on the stroke amount of the arm cylinder 8.

[0029] The bucket angle sensor S3 is one of the attitude detection sensors and is configured to detect the rotation angle of the bucket 6. In this embodiment, the bucket angle sensor S3 is a stroke sensor that detects the stroke amount of the bucket cylinder 9 and derives the rotation angle of the bucket 6 around the connecting pin that connects the arm 5 and the bucket 6 based on the stroke amount of the bucket cylinder 9.

[0030] The boom angle sensor S1, arm angle sensor S2, and bucket angle sensor S3 may each be a rotary encoder, acceleration sensor, potentiometer (variable resistor), tilt sensor, or inertial measuring device, etc. The inertial measuring device may be, for example, a combination of an acceleration sensor and a gyro sensor.

[0031] The rotational angular velocity sensor S4 is configured to detect the rotational angular velocity of the upper rotating body 3. In this embodiment, the rotational angular velocity sensor S4 is a gyro sensor. The rotational angular velocity sensor S4 may also be configured to calculate the rotation angle based on the rotational angular velocity. The rotational angular velocity sensor S4 may also be composed of other sensors such as a rotary encoder.

[0032] The upper rotating body 3 is equipped with a cabin 10 serving as the driver's cab, an engine 11, a positioning device 18, a sound collection device A1, a spatial recognition device C1, and a communication device T1, among other things. A controller 30 is also installed inside the cabin 10. The driver's seat and control devices are also installed inside the cabin 10.

[0033] Engine 11 is the power source for the shovel 100. In this embodiment, engine 11 is a diesel engine. The output shaft of engine 11 is connected to the input shafts of the main pump 14 and the pilot pump 15, respectively.

[0034] The positioning device 18 is configured to measure the position of the shovel 100. In this embodiment, the positioning device 18 is a GNSS (Global Navigation Satellite System) compass and is configured to measure the position and orientation of the upper rotating body 3.

[0035] The sound collection device A1 is configured to collect sounds generated around the shovel 100. In this embodiment, the sound collection device A1 is a microphone attached to the upper rotating body 3.

[0036] The spatial recognition device C1 is configured to measure the work area in which the shovel 100 (an example of a work machine) is working. In this embodiment, the spatial recognition device C1 includes a rear camera C1B mounted on the rear end of the upper surface of the upper rotating body 3, a front camera C1F mounted on the front end of the upper surface of the cabin 10, a left camera C1L mounted on the left end of the upper surface of the upper rotating body 3, and a right camera C1R mounted on the right end of the upper surface of the upper rotating body 3. The spatial recognition device C1 may also be a 360-degree camera installed at a predetermined position inside the cabin 10. The predetermined position is, for example, a position corresponding to the eye position of an operator seated in the driver's seat installed inside the cabin 10.

[0037] The communication device T1 is configured to control communication with equipment located outside the shovel 100. In this embodiment, the communication device T1 is configured to control wireless communication between the communication device T1 and equipment located outside the shovel 100 via a wireless communication network. The communication device T1 includes, for example, a mobile communication module that supports mobile communication standards such as LTE (Long Term Evolution), 4G (4th Generation), and 5G (5th Generation), and a satellite communication module for connecting to a satellite communication network.

[0038] The controller 30 is an arithmetic unit that performs various calculations. The controller 30 is installed, for example, in the cabin 10 and controls the drive of the shovel 100. The functions of the controller 30 may be realized by any hardware, software, or a combination thereof. For example, the controller 30 is mainly composed of a microcomputer including a CPU (Central Processing Unit), memory devices such as RAM (Random Access Memory), non-volatile auxiliary storage devices such as ROM (Read Only Memory), and various input / output interface devices. The controller 30 realizes various functions by executing various programs installed in the non-volatile auxiliary storage device on the CPU.

[0039] Figure 3 is a block diagram showing an example of the drive system configuration for the excavator 100 shown in Figure 2. In Figure 3, mechanical power transmission lines are shown as double lines, hydraulic fluid lines as thick solid lines, pilot lines as dashed lines, and electrical control lines as dotted lines.

[0040] The drive system of the Shovel 100 consists of an engine 11, a regulator 13, a main pump 14, a pilot pump 15, a control valve unit 17, a controller 30, and a solenoid valve unit 45, etc. The engine 11 is driven and controlled by an engine control unit 74.

[0041] The main pump 14 supplies hydraulic fluid to the control valve unit 17 via the hydraulic fluid line 16. In this embodiment, the main pump 14 is a swashplate type variable displacement hydraulic pump.

[0042] The regulator 13 is configured to control the discharge rate of the main pump 14. In this embodiment, the regulator 13 is configured to adjust the swash plate tilt angle of the main pump 14 according to the discharge pressure of the main pump 14 or a control signal from the controller 30. The discharge rate (displaced volume) per revolution of the main pump 14 is controlled by the regulator 13.

[0043] The pilot pump 15 is configured to supply hydraulic fluid to various hydraulic control devices via the pilot line 25. In this embodiment, the pilot pump 15 is a fixed-displacement hydraulic pump. However, the pilot pump 15 may be omitted. In this case, the function that the pilot pump 15 performed may be realized by the main pump 14. That is, the main pump 14 may have a function to supply hydraulic fluid to the solenoid valve unit 45, etc., via a throttle, in addition to its function of supplying hydraulic fluid to the control valve unit 17.

[0044] The control valve unit 17 is configured to selectively supply hydraulic fluid received from the main pump 14 to one or more hydraulic actuators. In this embodiment, the control valve unit 17 includes a plurality of control valves corresponding to a plurality of hydraulic actuators. The control valve unit 17 is configured to selectively supply hydraulic fluid discharged from the main pump 14 to one or more hydraulic actuators. The hydraulic actuators include, for example, a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, a hydraulic motor 1L for left-side travel, a hydraulic motor 1R for right-side travel, and a hydraulic motor 2A for slewing.

[0045] The controller 30 is configured to control the solenoid valve unit 45 based on operation signals received through the communication device T1. In this embodiment, the operation signals are transmitted from the remote control room RC. The operation signals may also be generated by an operating device located inside the cabin 10.

[0046] The solenoid valve unit 45 includes a plurality of solenoid valves arranged in each pilot line 25 that connects the pilot pump 15 to the pilot port of each control valve in the control valve unit 17.

[0047] In this embodiment, the controller 30 can control the pilot pressure acting on the pilot port of each control valve by individually controlling the opening area of ​​each of the multiple solenoid valves. Therefore, the controller 30 can control the flow rate of hydraulic fluid flowing into each hydraulic actuator and the flow rate of hydraulic fluid flowing out of each hydraulic actuator, and consequently, the movement of each hydraulic actuator can be controlled.

[0048] In this way, the controller 30 can perform actions such as raising and lowering the boom 4, opening and closing the arm 5, opening and closing the bucket 6, rotating the upper slewing body 3, and driving the lower traveling body 1 in response to operation signals from an external location such as a remote control room.

[0049] Figure 4 shows an example of the electrical system configuration installed in the excavator shown in Figure 1. As shown in Figure 4, the engine 11 is connected to the engine control unit 74. The engine control unit 74 transmits various data indicating the status of the engine 11 to the controller 30. The controller 30 is configured to store various data indicating the status of the engine 11 in the memory 30a. The remote control room RC is the same as in Figure 1, so its explanation is omitted.

[0050] Battery 70 is configured to supply power to various electrical loads mounted on the shovel 100. The alternator 11a (generator), starter 11b, controller 30, and electrical components 72 are all configured to operate using the power stored in battery 70. The starter 11b is powered by the power stored in battery 70 and is configured to start the engine 11. Battery 70 is also configured to be charged by the power generated by the alternator 11a.

[0051] The water temperature sensor 11c transmits data regarding the engine coolant temperature to the controller 30. The regulator 13 transmits data regarding the swashplate tilt angle to the controller 30. The discharge pressure sensor 14b transmits data regarding the discharge pressure of the main pump 14 to the controller 30. The positioning device 18 transmits data regarding the position of the shovel 100 to the controller 30.

[0052] An oil temperature sensor 14c is provided in the pipeline 14-1 between the main pump 14 and the hydraulic oil tank, which stores the hydraulic oil that the main pump 14 draws in. The oil temperature sensor 14c transmits data regarding the temperature of the hydraulic oil flowing through the pipeline 14-1 to the controller 30.

[0053] A urea solution level sensor 21a installed in the urea solution tank 21 transmits data regarding the remaining amount of urea solution to the controller 30. A fuel level sensor 22a installed in the fuel tank 22 transmits data regarding the remaining amount of fuel to the controller 30.

[0054] Communication device T1 is configured to send and receive information with communication device T2 installed in the remote control room RC via wireless communication. In this embodiment, communication device T1 and communication device T2 are configured to send and receive information via a fifth-generation mobile communication line (5G line), LTE line, or satellite line, etc.

[0055] <Block configuration of the remote control system> Figure 5 is a functional block diagram showing an example configuration of the remote control system SYS according to this embodiment. In the example shown in Figure 5, the block configurations of the remote control room RC, the drone DR, and the excavator 100, which are included in the remote control system SYS, are shown.

[0056] The shovel 100 is equipped with a positioning device 18, a controller 30, a solenoid valve unit 45, an auxiliary storage device 48, a spatial recognition device C1, and a communication device T1. The remote control room RC is equipped with an operation sensor 29, a remote controller 40, a display device D1, and a communication device T2. The drone DR is equipped with a positioning device D18, an object detection controller D30, a spatial recognition device C11, an auxiliary storage device D48, and a communication device T3.

[0057] The functions of the controller 30 mounted on the shovel 100 will now be described. As shown in Figure 5, the controller 30 has, as a functional block, an object detection controller 31, a shovel state identification unit 32, and an actuator drive unit 33.

[0058] The object detection controller 31 includes a detection unit 31a, a position calculation unit 31b, a tag information assignment unit 31c, and a transmission information generation unit 31d. Based on the measurement information detected by the spatial recognition device C1, it recognizes objects present around the shovel 100 and performs processing to transmit information about those objects to the remote control room RC.

[0059] In this embodiment, instead of the shovel 100 transmitting measurement information detected by the spatial recognition device C1 to the remote control room RC to inform the remote control room RC of the situation around the shovel 100, it transmits information about objects present around the shovel 100, extracted from the measurement information.

[0060] In other words, the measurement information generated by the spatial recognition device C1 is large in size. When transmitting such measurement information to the remote control room RC, it places a heavy burden on the public network NT used for communication. In particular, depending on the environment of the public network NT, such as when the shovel 100 is working in the suburbs of a city, there is a possibility that the transmission of measurement information may be delayed.

[0061] Therefore, the object detection controller 31 according to this embodiment detects objects present around the shovel 100 from the measurement information generated by the spatial recognition device C1, and generates information (hereinafter referred to as "transmission information") for transmitting information about the objects to the remote control room RC.

[0062] The transmitted information will include, for example, information containing "{tag, x, y, z, roll, pitch, yaw, x_len, y_len, z_len, α}" (hereinafter referred to as object detection information) for each detected object.

[0063] "Tag" (hereinafter referred to as "tag information") is information indicating the type of object detected. For example, "tag" stores information indicating a person (object to avoid), a building (object to avoid), the ground (work target), soil (work target), or a suspended load (work target).

[0064] "x, y, z" represent information indicating the position of the detected object in three-dimensional coordinates. In this embodiment, the information indicating the object's position may be, for example, position information in a world coordinate system based on GNSS (Global Navigation Satellite System). In this embodiment, latitude and longitude are shown on the x and y axes, and altitude is shown on the z axis.

[0065] "roll, pitch, yaw" are information indicating the inclination of the detected object in three-dimensional coordinates. In this embodiment, the information indicating the inclination of the object represents the roll angle, pitch angle, and yaw angle, respectively.

[0066] "x_len, y_len, z_len" represent information indicating the three-dimensional size of the detected object. That is, "x_len" is the length in the x-axis direction, "y_len" is the length in the y-axis direction, and "z_len" is the length in the z-axis direction. In this embodiment, the shape of the detected object is maintained as a rectangular parallelepiped. Furthermore, any method can be used to set the length in the axis direction that cannot be detected by the spatial recognition device C1. For example, information indicating that the object was not detected may be set, or a standard length corresponding to the detected object may be assigned. When setting a standard length, a margin for safety may be included.

[0067] "α" represents attribute information of the detected object. For example, if "tag" indicates a person, "α" may include the person's state (standing, crouching, lying down) or the color of the clothes they are wearing. Also, if "α" indicates soil, "α" may include information indicating the shape of the soil (for example, polygon information modeling the surface of the soil) or a texture image representing the pattern of the soil's surface.

[0068] Next, we will explain the processes that take place before the information is sent.

[0069] The detection unit 31a detects objects within the detection range from the measurement information detected by the spatial recognition device C1. The information of the object detected by the detection unit 31a includes the shape, size, and color of the object. Any method can be used for detecting the object and its shape, etc., regardless of whether it is a well-known method. The information of the detected object is not limited to the shape, size, and color of the object; for example, it may not include color, or it may include other information.

[0070] The position calculation unit 31b determines the position, inclination, and size of the object detected by the detection unit 31a. In this embodiment, the position calculation unit 31b determines the position, inclination, and size of the object when it is considered to be a rectangular parallelepiped. The position of the object can be any position that is sufficient to identify the object. For example, if the object is considered to be a rectangular parallelepiped, the center position of the bottom surface of the rectangular parallelepiped may be used as the position of the object.

[0071] As a method for determining the specific position of an object, the position calculation unit 31b can recognize the direction and distance from the spatial recognition device C1 to the object from the measurement information of the spatial recognition device C1. In addition, the position calculation unit 31b can determine the position information of the shovel 100 in the world coordinate system from the signal from the positioning device 18. Therefore, the position calculation unit 31b can acquire the position information of the object in the world coordinate system based on the position information of the shovel 100 in the world coordinate system and the direction and distance from the spatial recognition device C1 installed on the shovel 100 to the object.

[0072] In this way, the position calculation unit 31b identifies the position information, inclination, and size of the object (rectangular prism) in the world coordinate system for each object detected from the measurement information.

[0073] The tag information assignment unit 31c assigns tag information and attribute information corresponding to the object detected by the detection unit 31a. The tag information assignment unit 31c refers to the tag information storage unit 48a stored in the auxiliary storage device 48.

[0074] In this embodiment, the tag information storage unit 48a stores the correspondence between information representing an object detected from measurement information (for example, a combination of the shape, size, and color of the detected object), tag information to be assigned to that object, and attribute information. In other words, the tag information assignment unit 31c can identify the tag information (an example of the type of object) and attribute information (an example of the state of the object) corresponding to the object detected by the detection unit 31a by referring to the tag information storage unit 48a.

[0075] Furthermore, if the detected object is the ground (work target) or soil (work target), the tag information assignment unit 31c may generate polygon information that models the surface of the object based on the information indicating the object.

[0076] The transmission information generation unit 31d generates transmission information that includes object detection information "{tag, x, y, z, roll, pitch, yaw, x_len, y_len, z_len, α}" for each detected object, based on the position, inclination, and size of the object (rectangular prism) calculated by the position calculation unit 31b, and the tag information (an example of the type of object) and attribute information identified by the tag information assignment unit 31c. The transmission information generation unit 31d then transmits the generated transmission information to the remote control room RC.

[0077] In this embodiment, transmission information containing object detection information "{tag, x, y, z, roll, pitch, yaw, x_len, y_len, z_len, α}" for each object is transmitted. Each of x, y, z, roll, pitch, yaw, x_len, y_len, and z_len is numerical data of a predetermined size (e.g., 8 to 16 bits), and each of tag and α is string data of a predetermined size (e.g., several characters). Therefore, the amount of data transmitted as transmission information can be reduced compared to when transmitting measurement information from the spatial recognition device C1.

[0078] The shovel status identification unit 32 is configured to identify the status of the shovel 100. In this embodiment, the status of the shovel 100 includes the position and orientation of the shovel 100. The position of the shovel 100 is, for example, the position of the shovel 100 in the world coordinate system (latitude, longitude, and altitude of the shovel 100 reference point). The shovel status identification unit 32 identifies the position and orientation of the shovel 100 based on the output of the positioning device 18. The shovel status identification unit 32 then transmits information indicating the identified position and orientation of the shovel 100 to the remote control room RC.

[0079] The actuator drive unit 33 is configured to drive the actuator mounted on the shovel 100. In this embodiment, the actuator drive unit 33 generates and outputs an operating signal for each of the multiple solenoid valves included in the solenoid valve unit 45 based on the operation signal transmitted from the remote controller 40.

[0080] Upon receiving an activation signal, each solenoid valve increases or decreases the pilot pressure acting on the pilot port of the corresponding control valve in the control valve unit 17. As a result, the hydraulic actuator corresponding to each control valve operates at a speed corresponding to the stroke amount of the control valve.

[0081] Next, the functions of the object detection controller D30 mounted on the drone DR will be described. As shown in Figure 5, the object detection controller D30 has a detection unit D30a, a position calculation unit D30b, a tag information assignment unit D30c, and a transmission information generation unit D30d as functional blocks. Based on the measurement information detected by the spatial recognition device C11, it recognizes objects present in the work site and processes information about those objects to transmit to the remote control room RC. Operation of the drone DR is possible as long as the work site is included in the detection target of the spatial recognition device C11, and may be automatic control or manual control by any operator.

[0082] The detection unit D30a detects objects within the detection range from the measurement information detected by the spatial recognition device C11. The information of the object detected by the detection unit D30a includes the shape, size, and color of the object. However, the information of the detected object is not limited to the shape, size, and color of the object; for example, it may not include color, or it may include other information.

[0083] Similar to the position calculation unit 31b described above, the position calculation unit D30b calculates the position information of the object in the world coordinate system, the tilt of the object (rectangular prism), and the size of the object (rectangular prism) for each detected object, based on the measurement information from the spatial recognition device C11 and the position information of the drone DR in the world coordinate system from the positioning device D18.

[0084] The tag information assignment unit D30c refers to the tag information storage unit D48a of the auxiliary storage device D48 and assigns tag information and attribute information corresponding to the object detected by the detection unit D30a. The method for assigning tag information and attribute information is the same as that of the tag information assignment unit 31c of the shovel 100, so the explanation is omitted.

[0085] The transmission information generation unit D30d generates transmission information that includes object detection information "{tag, x, y, z, roll, pitch, yaw, x_len, y_len, z_len, α}" for each detected object, based on the position, inclination, and size of the object (rectangular prism) calculated by the position calculation unit D30b, and the tag information and attribute information identified by the tag information assignment unit D30c. The transmission information generation unit D30d then transmits the generated transmission information to the remote control room RC.

[0086] In this embodiment, the block configuration for the shovel 100 and drone DR to transmit transmission information has been described. However, this embodiment does not limit the devices that transmit transmission information to the shovel 100 and drone DR; the fixed-point camera C12 also transmits transmission information. Furthermore, a measuring device including a spatial recognition device held by the worker may also transmit transmission information. In this way, any device present at the work site may transmit transmission information. The configurations of these devices are the same as those of the shovel 100 and drone DR, so their description will be omitted.

[0087] Next, the functions of the remote controller 40 installed in the remote control room RC will be described. The remote controller 40 is configured to remotely control the excavator 100. The functional blocks include an object restoration controller 41 and an operation signal generation unit 42.

[0088] The object restoration controller 41 includes a virtual 3D space generation unit 41a, a placement unit 41b, and a display control unit 41c. When it receives transmitted information from the shovel 100 and the drone DR, it places a 3D model corresponding to the tag information included in the transmitted information in the virtual 3D space at coordinates corresponding to the position and inclination included in the transmitted information, and displays the virtual 3D space on the display device D1.

[0089] The virtual 3D space generation unit 41a generates a virtual 3D space (hereinafter referred to as the virtual 3D space). The shape of the virtual 3D space can be any shape; for example, it may be generated as a virtual cylindrical shape. The center coordinates of the virtual 3D space can be any position; for example, they may be the coordinates that indicate the center of the work site.

[0090] The virtual 3D space generation unit 41a then projects a predetermined image onto the inner surface of the virtual 3D space. The predetermined image can be any image, and an image corresponding to the work site is preferred. For example, if the work site is a forest, an image representing a forest may be projected. The projected image may also be switched according to the current work time. This allows the operator OP working in the remote control room RC to recognize the current time of day.

[0091] The placement unit 41b places a 3D model representing an object in a virtual 3D space based on the object detection information for each object included in the transmitted information, "{tag, x, y, z, roll, pitch, yaw, x_len, y_len, z_len, α}". In this embodiment, the placement unit 41b identifies the size of the object (rectangular prism) using "x_len, y_len, z_len", the inclination of the object (rectangular prism) using "roll, pitch, yaw", and the position in the virtual 3D space using "x, y, z". In this embodiment, the coordinates of the virtual 3D space and the coordinates of the world coordinate system can be converted according to a predetermined conversion table (not shown). This makes it possible to convert the position, size, and inclination of the world coordinate system included in the object detection information to the corresponding position, size, and inclination in the virtual 3D space.

[0092] Furthermore, the placement unit 41b identifies the type of object using "tag" and identifies the state or shape of the object using "α". In doing so, the placement unit 41b refers to the 3D model storage unit 49a.

[0093] The auxiliary storage device 49 stores the 3D model storage unit 49a. The 3D model storage unit 49a stores a 3D model (hereinafter referred to as the 3D model) according to the type of object indicated by "tag". Furthermore, the 3D model storage unit 49a stores the shape of the 3D model according to the state of the object indicated by "α". As a result, in this embodiment, a 3D display of the detected object becomes possible. Therefore, in this embodiment, the operator OP can easily grasp the work site by referring to the 3D display. Consequently, the workload of the operator OP can be reduced.

[0094] Therefore, the placement unit 41b can identify a 3D model corresponding to the type of object indicated by "tag" with a shape corresponding to the state of the object indicated by "α".

[0095] The placement unit 41b then applies the shape of the specified 3D model to a rectangular parallelepiped in the virtual 3D space, which is represented by the position, inclination, and size of the object. In this way, the placement unit 41b places the 3D model in the virtual 3D space.

[0096] The placement unit 41b places 3D models corresponding to all objects included in the transmitted information. The placement unit 41b places 3D models based on the transmitted information of all devices (e.g., the shovel 100, the drone DR, and the fixed-point camera C12) transmitted to the remote control room RC. This allows the placement unit 41b to place 3D models corresponding to objects even if they exist in areas that are blind spots for, for example, the shovel 100.

[0097] The display control unit 41c displays the virtual 3D space in which the 3D model is placed on the display device D1. The viewpoint and point of focus when the display control unit 41c displays the virtual 3D space can be any position. For example, the display control unit 41c may set the position where the cabin of the shovel 100 is located as the initial starting point, and a position a predetermined distance in front of the cabin as the initial point of focus. The point of focus and viewpoint for displaying the virtual 3D space can be moved within the virtual 3D space in response to operations from the control device 26 located in the remote control room RC.

[0098] The operation signal generation unit 42 is configured to generate operation signals. In this embodiment, the operation signal generation unit 42 is configured to generate operation signals based on the output of the operation sensor 29.

[0099] Next, the remote control room RC will be described. Figure 6 shows an example of the layout of the remote control room RC. The remote control room RC is equipped with multiple operating devices 26, with the operator's seat DS as the reference point.

[0100] In this embodiment, the display device D1 is a multi-display consisting of nine monitors arranged in three vertical rows and three horizontal columns, as shown in Figure 6. Specifically, the display device D1 includes the central monitor D1a, the upper monitor D1b, the lower monitor D1c, the left monitor D1d, the right monitor D1e, the upper left monitor D1f, the upper right monitor D1g, the lower left monitor D1h, and the lower right monitor D1i.

[0101] The display device D1 then displays a virtual 3D space on which the 3D model is placed, controlled by the display control unit 41c. The operator OP can, via the operating device 26, operate the shovel 100 and also switch the display of the virtual 3D space (for example, move the viewpoint).

[0102] <Processing performed on the work site side> Next, we will explain the processing performed on the shovel 100. Figure 7 is a diagram showing the space to be detected by the spatial recognition device C1 installed on the shovel 100. As shown in Figure 7, the spatial recognition device C1 acquires information about the space 6000 around the shovel 100 and generates measurement information.

[0103] As shown in Figure 7, the measurement information includes information about the space 6000 that is the target of detection, specifically about person 6001, person 6002, and building 6003.

[0104] The object detection controller 31 generates transmission information based on the measurement information. Figure 8 shows an object detected by the object detection controller 31.

[0105] Based on the measurement information, the detection unit 31a detects objects present in the space 6000 shown in Figure 7. As shown in Figure 8, the detection unit 31a detects object 7001, object 7002, and object 7003.

[0106] The position calculation unit 31b then calculates the position information in the world coordinate system for each of the reference points 7001A for object 7001, 7002A for object 7002, and 7003A for object 7003. Furthermore, the position calculation unit 31b calculates the size and inclination of objects 7001, 7002, and 7003. Lengths in the depth direction that cannot be detected by the spatial recognition device C1 do not need to be set. For example, the drone DR may acquire and set the depth directions of each of objects 7001, 7002, and 7003. In this case, the object restoration controller 41 can determine the size of the object by combining the recognition results of multiple spatial recognition devices.

[0107] The tag information assignment unit 31c assigns tag information and attribute information to each of the objects 7001, 7002, and 7003. For example, the tag information assignment unit 31c assigns the tag information "person" and attribute information "standing" to object 7001, the tag information "person" and attribute information "standing" to object 7002, and the tag information "building" to object 7003. For example, the tag information storage unit 48a associates the shape and size of a person with the tag information "person". Furthermore, the tag information storage unit 48a associates the vertical size (size) of a person when standing or the shape of a person when standing with the attribute information "standing". As a result, the tag information assignment unit 31c can assign the tag information "person" and attribute information "standing" by referring to the tag information storage unit 48a.

[0108] The transmission information generation unit 31d then generates transmission information based on the position, inclination, and size of the object calculated by the position calculation unit 31b, and the tag information and attribute information of the object identified by the tag information assignment unit 31c, and transmits the transmission information.

[0109] In this embodiment, the transmission information generation unit 31d transmits the transmission information, which reduces the amount of data compared to when measurement information is transmitted by the spatial recognition device C1. In this embodiment, the interval at which the transmission information is transmitted can be any interval, for example, the same as the interval at which image information is transmitted in normal remote monitoring.

[0110] <Processing performed in the remote control room> Next, the processing performed on the remote control room RC will be described. In this embodiment, the remote control room RC displays the status of the work site on the display device D1 based on the information transmitted from the shovel 100, the drone DR, and the fixed-point camera C12.

[0111] First, the virtual 3D space generation unit 41a generates a virtual 3D space. In this embodiment, the placement unit 41b determines the size of the object (rectangular prism) using "x_len, y_len, z_len", determines the inclination of the object (rectangular prism) using "roll, pitch, yaw", and determines the position in the virtual 3D space using "x, y, z".

[0112] Figure 9 is a diagram showing the size, inclination, and position of objects identified by the placement unit 41b according to this embodiment, represented as a rectangular parallelepiped. In the example shown in Figure 9, objects 9001, 9002, and 9003 are placed at their identified sizes and positions. Furthermore, in the virtual three-dimensional space, the viewpoint 9011 is set based on information received from the shovel 100 indicating the position and orientation of the shovel 100. Then, the point of focus should be set at a position corresponding to the orientation of the shovel 100.

[0113] The placement unit 41b then places the 3D models indicated in the tag information so that they fit within the rectangular parallelepipeds indicated for each of the objects 9001, 9002, and 9003. The 3D models are placed in the state indicated in the attribute information. In this embodiment, a 3D model representing the current state of the object can be displayed. Therefore, the operator OP can recognize the current state of the object even when viewing a 3D model, thus reducing the workload.

[0114] In other words, object 9001 and object 9002 each contain a 3D model representing a "person" in a "standing" state. Object 9003 contains a 3D model representing a "building".

[0115] Furthermore, objects placed in the virtual 3D space may be positioned using the transmission information from the drone DR and the fixed-point camera C12, in addition to the transmission information from the shovel 100. This allows objects that are in blind spots for the shovel 100's spatial recognition device C1 to be positioned.

[0116] The display control unit 41c displays a virtual 3D space on which the 3D model is placed on the display device D1. Figure 10 is an example of the display screen of the display device D1 as displayed by the display control unit 41c. In this embodiment, the display screen shown in Figure 10 may be displayed on any one of the nine monitors of the display device D1, or it may be displayed on multiple monitors (for example, 4x4 or 3x3).

[0117] The display screen shown in Figure 10 shows the view of the virtual 3D space from viewpoint 9011 shown in Figure 9. As a result, a 3D model 1001 representing a person, a 3D model 1002 representing a person, and a 3D model 1003 representing a building are displayed according to the orientation of the shovel 100.

[0118] The operator OP can use the control device 26 to move their viewpoint within the virtual three-dimensional space to display the work site on the display device D1. For example, by moving their viewpoint, the operator OP can check the situation in areas that are not visible from the cabin of the shovel 100. In this way, by moving the viewpoint, it becomes possible to display information within the virtual three-dimensional space. Therefore, it becomes easier for the operator OP in the operator's seat DS to grasp the situation at the work site.

[0119] Thus, in this embodiment, the operator OP at the control desk DS can check the situation at the work site by referring to a virtual 3D space based on the transmitted information.

[0120] (modified version) The embodiments described above described the case of remotely controlling an excavator. However, the embodiments described above are not limited to the remote control of an excavator. Therefore, the modified examples will describe the case of remotely controlling a crawler crane.

[0121] Figure 11 shows a working site of a modified crawler crane. In the example shown in Figure 11, the crawler crane 1100 is operated by an operator OP located in a remote control room RC. The remote control room RC is the same as in the embodiment described above, and its description is omitted. Components similar to those in the embodiment described above are assigned the same reference numerals and their descriptions are omitted.

[0122] In the example shown in Figure 11, the crawler crane 1100 comprises a lower traveling body 1102, an upper slewing body 1103, a boom 1104, and a counterweight 1105. The crawler crane 1100 is equipped with a positioning device 18, a communication device T1, and a controller 1130.

[0123] A spatial recognition device C1101 is provided near the tip of the boom 1104. The spatial recognition device C1101 is positioned so that its optical axis 1151 points downward. This allows the spatial recognition device C1101 to detect the conditions at the crawler crane 1100's work site.

[0124] Furthermore, the controller 1130 of the crawler crane 1100 is equipped with an object detection controller 31, similar to the embodiment described above. The object detection controller 31 generates transmission information based on the measurement information detected by the spatial recognition device C1101 and transmits the transmission information to the remote control room RC using the communication device T1.

[0125] In the example shown in Figure 11, the crawler crane 1100 is handing over the load 1112 to a worker 1113 located on top of the building 1114.

[0126] In the situation shown in Figure 11, when operating the crawler crane 1100 from the remote control room RC from the viewpoint of the cabin, the building 1114 becomes an obstacle, making it difficult to check the condition of the load 1112.

[0127] Therefore, in this modified example, the means for displaying the virtual three-dimensional space described in the above embodiment is used. In this modified example, the viewpoint can be moved as long as it is within the range 1114 detected by the spatial recognition device C1101. For example, the work site can be displayed from viewpoint 1111.

[0128] Figure 12 shows an example of a screen displayed on the display device D1 in the remote control room RC. The example shown in Figure 12 is a virtual three-dimensional space representing the work site from viewpoint 1111.

[0129] As shown in Figure 12, the display screen shows a 3D model 1201 of worker 1113 and a 3D model 1202 of cargo 1112. In this way, the operator OP in the remote control room RC can confirm the location of cargo 1112 from the location where worker 1113 is located.

[0130] In other words, the operator OP in the remote control room RC can control the rotation of the upper rotating body 1103 of the crawler crane 1100 and the lowering of the boom 1104 by checking the display screen shown in Figure 12, so that the load 1112 can be handed over to the worker 1113.

[0131] In this modified example, the operator (OP) of the remote control room (RC) can improve work efficiency by moving their viewpoint when viewing the virtual 3D space according to the task.

[0132] Incidentally, when constructing building 1114, it is easier for the operator OP of the remote control room RC to check the current status of building 1114. For this reason, a 3D model of building 1114 with a more detailed structure is preferable.

[0133] Therefore, as the 3D model of the building 1114 shown in this modified example, 3D CAD data of building 1114 may be used. For example, the 3D CAD data of building 1114 is stored in the auxiliary storage device 49 of the remote control room RC. The information transmitted from the crawler crane 1100 only needs to include information that identifies building 1114 as tag information. This allows the object restoration controller 41 to display a 3D model of building 1114 based on the 3D CAD data in a virtual 3D space.

[0134] Furthermore, when the placement unit 41b of the object restoration controller 41 places a 3D model based on CAD data, it may place a 3D model based on the current building status. For example, the attribute information may include the current number of floors completed, or the difference in CAD data to represent the current building status. This allows the object restoration controller 41 to display a 3D model that takes the current building status into account.

[0135] (Modification 2) In this embodiment and its modifications, the case of transmitting transmission information has been described. However, this embodiment is not limited to such a transmission method, and for example, the data to be transmitted may be switched between transmission information and measurement information in response to operations from the remote control room (RC).

[0136] For example, if the remote control room RC wants to display a screen based on specific measurement information rather than a virtual 3D space, the remote controller 40 sends a data switching request to the work machine (e.g., shovel 100 or crawler crane 1100) according to the operation from the control device 26, etc. The work machine then transmits the measurement information identified by the spatial recognition devices C1, C1101, etc. The display screen D1 in the remote control room RC then displays a screen based on the measurement information. This allows the operator to recognize the current situation based on the measurement information. For example, if a worker at the work site is performing a predetermined task, and the operator wants to recognize the progress of that task, switching to a display screen based on the measurement information allows the operator OP in the remote control room RC to recognize the work status. By switching the display screen in this way, the operator OP can display information according to the situation at the work site, thus reducing their workload.

[0137] <effect> In the embodiments and modifications described above, the working site can be monitored from the remote control room RC, enabling remote operation of the work machinery (e.g., shovel 100 or crawler crane 1100). At that time, the work machinery (e.g., shovel 100 or crawler crane 1100), a drone, or a fixed-point camera generates transmission information with a smaller data volume than the measurement information based on the measurement information generated by the spatial recognition device, and transmits this transmission information to the remote control room RC. Since the transmitted transmission information has a smaller data volume than the measurement information, the burden on the network can be reduced. Furthermore, because the data volume of the transmitted information is small, the time it takes to reach the remote control room RC can be shortened. Alternatively, delays in the time it takes to reach the remote control room RC can be suppressed.

[0138] Therefore, in a remote control room (RC), the operator (OP) can immediately recognize the current situation at the work site and take appropriate action right away. This improves safety.

[0139] The above describes an embodiment using a shovel as an example of a work machine, but the present invention is not limited to the above embodiments. Various changes, modifications, substitutions, additions, deletions, and combinations are possible within the scope described in the claims. These also naturally fall within the technical scope of the present invention.

[0140] In other words, the remote control system described above was explained as an example of a remote information acquisition system applied to a system for operating work machinery, but it may also be applied to other types of systems. For example, it may be applied to a system for remotely monitoring or remotely controlling a vehicle traveling on a road. Furthermore, it may be applied to a security surveillance system in an area where many people are present (displaying 3D models of people and their belongings). Thus, it can be applied to any system that monitors or manipulates objects from a remote location. [Explanation of Symbols]

[0141] 100 Shovel 1100 Crawler Crane C1, C11 Spatial recognition device T1, T2, T3 communication devices 1. 1102 Lower running body 2. Swivel mechanism 3. 1103 Upper rotating body 4, 1104 Boom 5 Arms 6 buckets 4 Boom 5 Arms 6 buckets 7 Boom Cylinder 8 Arm Cylinder 9 Bucket Cylinder 11 Engine 1105 Counterweight 18. D18 Positioning device 26 Operating device 29 Operation Sensors 30, 1130 controller 31. D30 Object Detection Controller 31a, D30a Detection unit 31b, D30b Position calculation section 31c, D30c Tag information assignment unit 31d, D30d Transmission Information Generation Unit 32 Shovel condition identification unit 33 Actuator drive unit 45 Solenoid valve unit 48, D48 Auxiliary storage device 48a, D48a Tag information storage unit DR Drone C12 Fixed-point camera (an example of a fixed-point measurement device) RC Remote Control Room 40 Remote Controllers 26 Operating device 29 Operation Sensors D1 display device 49 Auxiliary storage device 49a 3D Model Memory Unit 40 Remote Controllers 41 Object Restoration Controller 41a Virtual 3D Space Generation Unit 41b Placement part 41c Display Control Unit 42 Operation signal generation section

Claims

1. A first device that identifies the position, type, and three-dimensional size of an object present in a work area from measurement information generated by a spatial recognition device that measures the work area in which a work machine is working, and transmits the position, type, and three-dimensional size. The device comprises a second device configured to remotely control the aforementioned work machine, The second apparatus described above is The system further includes a memory unit that stores a three-dimensional shape model corresponding to the aforementioned type. When the position, type, and three-dimensional size are received, the system is configured to place a model of the three-dimensional shape corresponding to the type in a virtual three-dimensional space for display on the screen, with a size based on the received three-dimensional size, at coordinates corresponding to the position. A remote information acquisition system for industrial machinery.

2. The viewpoint for displaying the aforementioned virtual three-dimensional space is configured to be movable within the virtual three-dimensional space in response to operations from the control unit. The remote information acquisition system according to claim 1.

3. The first device is at least one of the work machine, the drone, and the fixed-point measurement device provided to image the work area. The remote information acquisition system according to claim 1.

4. The first device described above is provided in multiple configurations of different types. The remote information acquisition system according to claim 3.

5. The type includes information indicating whether or not the object is the work object of the work machine, The first device, if the information includes that the object is the work object of the work machine, identifies the state of the object and transmits the identified state of the object. The second apparatus described above is Furthermore, upon receiving the aforementioned state, the system is configured to place a model of the three-dimensional shape in the virtual three-dimensional space in a shape corresponding to the aforementioned state, at the aforementioned coordinates. The remote information acquisition system according to claim 1.

Images